LVSM: A Large View Synthesis Model with Minimal 3D Inductive Bias
Authors: Haian Jin
Hanwen Jiang
Hao Tan
Kai Zhang
Sai Bi
Tianyuan Zhang
Fujun Luan
Noah Snavely
Zexiang Xu
Venue: ICLR 2025 Oral (2025)
Keywords: novel view synthesis
transformer
large model
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We put forward a purely transformer-based large view synthesis model, which achieves impressive novel view synthesis results on both object-level and scene-level with minimal 3D inductive bias.

Abstract: We propose the Large View Synthesis Model (LVSM), a novel transformer-based approach for scalable and generalizable novel view synthesis from sparse-view inputs. We introduce two architectures: (1) an encoder-decoder LVSM, which encodes input image tokens into a fixed number of 1D latent tokens, functioning as a fully learned scene representation, and decodes novel-view images from them; and (2) a decoder-only LVSM, which directly maps input images to novel-view outputs, completely eliminating intermediate scene representations. Both models bypass the 3D inductive biases used in previous methods---from 3D representations (e.g., NeRF, 3DGS) to network designs (e.g., epipolar projections, plane sweeps)---addressing novel view synthesis with a fully data-driven approach. While the encoder-decoder model offers faster inference due to its independent latent representation, the decoder-only LVSM achieves superior quality, scalability, and zero-shot generalization, outperforming previous state-of-the-art methods by 1.5 to 3.5 dB PSNR. Comprehensive evaluations across multiple datasets demonstrate that both LVSM variants achieve state-of-the-art novel view synthesis quality, delivering superior performance even with reduced computational resources (1-2 GPUs). Please see our anonymous website for more details: https://haian-jin.github.io/projects/LVSM/

Published as a conference paper at ICLR 2025
LVSM: A LARGE VIEW SYNTHESIS MODEL WITH MINIMAL 3D INDUCTIVE BIAS
Haian Jin' Hanwen Jiang Hao Tan Kai Zhang?
Sai Bi
Tianyuan Zhang*
Fujun Luan Noah Snavely Zexiang Xu
'Cornell University The University of Texas at Austin 3 "Adobe Research "Massachusetts Institute of Technology
ABSTRACT
We propose the Large View Synthesis Model (LVSM), a novel transformer-based approach for scalable and generalizable novel view synthesis from sparse-view inputs. We introduce two architectures: (I) an encoder-decoder LVSM, which encodes input image tokens into a fixed number of 1D latent tokens, function- ing as a fully learned scene representation, and decodes novel-view images from them; and (2) a decoder-only LVSM, which directly maps input images to novel- view outputs, completely eliminating intermediate scene representations. Both models bypass the 3D inductive biases used in previous methods from 3D rep- resentations (c.g.. NeRF, 3DGS) to network designs (e.g., epipolar projections, plane sweeps) wweeps)-addressingg novel view synthesis with fully data-driven approach. While the encoder-decoder model offers faster inference due to its independent latent representation, the decoder-only LVSM achieves superior quality, scalabil- ity, and zero-shot generalization, outperforming previous state-of-the-art meth- ods by 1.5 3.5 dB PSNR. Comprehensive evaluations across multiple datasets demonstrate that both LVSM variants achieve ttate-of-the-ar novel view synthesis quality. Notably, our models surpass all previous methods even with reduced computational resources (1-2 GPUs). Please see our website for more results: https://haian- www.iiggigtt....m..o. wwtprreects//ivs..
INTRODUCTION
Novel view synthesis is long-standing challenge in vision and graphics. For decades, the community has generally relied on various 3D inductive biases, incorporating 3D priors and handcrafted structures to simplify the task and improve synthesis quality. Recently, NeRF, 3D Gaussian Splatting (3DGS). and their variants (Mildenhall et al. 2020; Barron et al. 2021 Müller et al., 2022 Chen et al. 2022, Xu et al. 2022 Kerbl et al. 2023] Yu et al. 2024b) have significantly advanced the field by introducing new inductive biases through carefully designed 3D representations (e.g., continuous volumetric fields and Gaussian primitives) and rendering equations (e.g., ray marching and splatting with alpha blending), reframing view synthesis as the optimization of the representations using rendering losses per-scene basis. Other methods have also built generalizable networks to estimate these representations or directly generate novel-view images in feed-forwand manner, often incorporating additional 3D inductive biases, such as projective epipolar lines or plane-sweep volumes, in their architecture designs (Wang et al. 2021a a Yu ct al. 2021
Chen et al. 2021 Suhail
al. 2022b Charatan al. 2024, Chen et al.) 2024).
While effective, these 3D inductive biases inherently limit model flexibility, constraining their adaptability more diverse and complex scenarios that do not align with predefined priors or handcrafted structures. Recent large reconstruction models (LRMs) (Hong et al. 2024 Li et al. 2023
Wei et al. 2024. /Zhang et al., 2024) have made notable progress in removing architecture-level biases by leveraging large transformers without relying epipolar projections plane-sweep volumes, achieving state-of-the novel view synthesis quality. However, despite these advances, LRMs still rely on representation-levll biases such NeRFs, meshes, or 3DGS, along with their respective rendering equations- that limit their potential generalization and scalability.
In this work, we aim to minimize 3D inductive biases and push the boundaries of novel view synthesis with a fully aata-driven approach. While some prior works have attempted to reduce 3D inductive biases (Sitzmann al. 2021; Sajjadi et al. 2022 Kulhánek al. 2022). they still face challenges

---

Published as a conference paper at ICLR 2025
Scene Level View Synthesis
Object Lovel View Synthesis
Input Images
GS.LRMM
Owr
Ground truth
Our rendered novel views LVSM supports feed-forward novel view synthesis from sparse posed image inputs (even from a single view) on both objects and scenes. LVSM achieves significant quality improvements compared with the previous SOTA method, ie., GS-LRM (Zhang et al. 2024). (Please zoom in for more details.)
Single image input
Input Images
Our rendered views
Figure I:
in scalability and achieving high rendering quality. We propose the Large View Synthesis Model (LVSM), novel transformer based framework that synthesizes novel-view images from posed sparse-view inputs without predefined rendering equations or 3D structures, enabling accurate, training-efficeet, and scalable novel view synthesis with photo-realistic quality (as shown in Fig- D
To this end, we first introduce an encoder-decoder LVSM, removing handcrafted 3D representations and their rendering equations. We use an encoder transformer to map the input (patchified) multi-view image tokens into fixed number of 1D latent tokens, independent of the number of input views. These latent tokens are then processed by decoder transformer, which uses target-view Plücker rays as positional embeddings to generate the target view's 's image tokens, ultimately regressing the output pixel colors from final linear layer. The encoder-decoder LVSM jointly learns reconstructor (encoder), a scene representation (latent tokens), and a renderer (decoder) directly from data. By removing the need for predefined inductive biases in rendering and representation, LVSM offers improved generalization and achieves higher quality compared to NeRF- and GS-based approaches.
Nevertheless, the encoder-decodes LVSM retains key inductive bias: the use of an intermediate, albeit fully learned, scene representation. To further push the boundary, we propose a decoder-only LVSM, which adopts single-stream transformer directly convert input view tokens and target pose tokens into target view tokens, bypassing any intermediate representations. The decoder-only LVSM integrates novel view synthesis process into a holistic data-driven framework. achieving simultancous scene reconstruction and rendering in a fully implicit manner with minimal 3D inductive bias.
We present a comprehensive evaluation of variants both LVSM architectures. Notably, our models, trained on 2 or 4 input views, demonstrate strong zero-shot generalization to an unseen number of views, ranging from single input to more than 10. Thanks minimal inductive biases, our decoder- only model consistently outperforms the encoder-decoder variant in terms of quality, scalability, and zero-shot capability with varying numbers of input views. On the other hand, the encoder-decoder model achieves much faster inference speed due to its use of fixed- length latent scene representation. Both models, benefiting from reduced 3D inductive biases, outperform previous methods, achieving state-of-the-art novel view synthesis quality across multiple object-level and scene-level benchmark datasets. Specifically, our decoder-only LVSM surpasses previous state-of-the-ard methods, such GS-LRM, by substantial margin of 1.5 to 3.5 dB PSNR Our final models were trained 64 A100 GPUs for 3-7 days. depending on the data type and model architecture, but we found that even with just 1-2 A100 GPUs for training, model (with decreased model and batch size) still outperforms all previous methods trained with equal or even more compute resources.
RELATED WORK
View Synthesis. Novel view synthesis (NVS) has been studied for decades. Image- based rendering (IBR) methods perform view synthesis by weighted blending of input reference images using proxy
2

---

Published as a conference paper at ICLR 2025
geometry (Debevec et al., 1996 Heigl et al.
1999 Sinha et al. 2009)
Light field methods build slice
of the 4D plenoptic function from dense view imputs (Gorllrr et al.
1996| Levoy & Hanraham 1996;
Davis et al. 2012). Recent learning- based IBR methods incorporate convolutional networks to predict
blending weights (Hedman et al. 2018 Zhou et al., 2016; 2018) or use predicted depth maps (Choi et al., 2019). However, the renderable region is usually ly constrained to be near the input views. Other
work leverages multiview-steree reconstructions to render larger viewpoint changes (Jancssek &
Pajdla 201T; Chaurasia et al. 2013 Penner& & Zhang 2017). In contrast, we use more scalable
network designs to learn generalizable priors from larger, real-world data. Moreover, we perform rendering at the image patch level, achieving better model efficiency and rendering quality.
Optimizing 3D Representations, NeRF (Mildenhall et al. 2020) introduced neural volumetric 3D representation with differentiable volume rendering. enabling neural scene reconstruction by minimizing rendering losses and setting a new standard in NVS. Later work improved NeRF with better rendering quality (Barron et al. 202T Verbin et al., 2022 Barron et al.] 2023), faster optimiza- tion or rendering speed (Reiser et al. |202|] Hedman et al. Reiser et al. (023), and looser requirements on the input views (Niemeyer et al. 2022,] Martin-Brualle et al. 2021 Wang et al.) 2021b). Other work has explored hybrid representations that combine implicit NeRF content with explicit 3D information, e.g.. in the form of voxels, as in DVGO (Sun et al.) 2022). Spatial complexity can be further decreased by using sparse voxels (Liu al. 2020: Fridovich- Keil al. 2022), volume
decomposition (Chan et al. 2022; Chen et al.] [022] 2022) and hashing techniques (Müller et al.
2022). Another line of work investigates explicit point-based representations (Xu al. [002]] Zhang
iet al. 2022; Feng et al. 2022). Gaussian Splatting (Kerbl et al. (023) extends these 3D points to
3D Gaussians, improving both rendering quality and speed. In contrast, we perform NVS using
large transformer models (optionally with a learned latent scene representation) without requiring the inductive bias of using prior 3D representations, nor any per-scene optimization process.
Generalizable View Synthesis Methods. Generalizable methods enable fast NVS inference by using neural networks, trained across scenes, to predict novel views or an underlying 3D repre- sentation in a cct al. 202 and IBRNet (Wang et al.] 2021a) predict volumetric 3D representations from input views, utilizing 3D-specific priors like epipolar lines plane sweep cost volumes. Later methods
manner. For example, PixeINeRF (Yu et al.. 202TD. MVSNeRF (Chen)
improve performance under (unposed) sparse views (Liu et al. 2022 Johari et al.. 2022] Jiang et al.,
2024 Szymanowicz et al. 2024b) Diang al., (023). while other work extends to 3DGS representa-
tions Charatan et al. (2024); |Szymanowicz et al. (2024a); Chen et al. (2024): Tang et al. (202).. On
the other hand, approaches that attempt to directly learn geometry-free rendering function (Suhail
ct al. 2022a Sajjadi et al., 2022] Sitzmann et al. [021 Rombach et al., 2021 Kulhánek et al.
2022) lack model capacity and scalability, preventing them from capturing high-frequency detail. Specifically, Scene Representation Transformers (SRT) (Saijadi et al. 2022) also aims to avoid explicit, handcrafted 3D representations and instead learns latent representation via transformer, similar to our encoder-decoder model. However, some of SRT's model and rendering designs lead to less effective performance, such as the CNN-based token extractor and the use of cross-attention in the decoder. In contrast, our models are fully transformer-based with bidirectional self-attenioon (see detailed discussion in Sec. 4.4). Additionally, we introduce more scalable decoder-only architecture that effectively learns the NVS function with minimal 3D inductive bias, without intermediate latent representation.
Recently, 3D large reconstruction models (LRMs) have emerged (Hong et al. 2024
al. 2023
Wang et al. 2023a| Xu et al.] 2023) Wei et al. 2024 Zhang et al. 2024 Xie et al.] 2024), utilizing scalable transformer architectures Vaswani et al. 2017 trained large datasets to learn generic 3D priors. While these methods avoid using epipolar projection or cost volumes in their architectures, they still rely on existing 3D representations like tri-plane NeRFs, meshes, or 3DGS, along with their corresponding rendering equations, limiting their flexibility. In contrast, our approach eliminates these 3D inductive biases, aiming to learn rendering function (and optionally scene representation) directly from data. This leads to more scalable models and significantly improved rendering quality.
In addition to the deterministic methods mentioned above, recent generative NVS models support NVS using image/video diffusion models (Watson et al. 2022, Liu et al.) 2023a Gao et al. [024]] Zheng& & Vedaldi 2024, Kong et al.) [2224] Voleti et al. 2025). ). Note that our LVSM models are deterministic, and thus are fundamentally different from these generative models. We discuss this in detail in the Appendix. A.6
3

---

Published as a conference paper at ICLR 2025
Decoder-only Architedure
Encoder Peender-eeederrtee D Architecture
Iaput Views & Plücker Rays
Synthesized Target Vew
Encoder Transfomer
Decoder- only Transformer
Lojant
D Decoder Transformet
Tokent
Linear & Unpalchily
Target. View Plicker Rays
Figure 2: LVSM model architecture. LVSM first patchifies the posed input images into tokens. The target view to be synthesized is represented by its Plücker ray embeddings and is also tokenized. The input view and target tokens sent a full transformer-based model predict tokens that are used to regress the target view pixels. We study two LVSM transformer architectures, Decoder-only architecture (left) and a Srccode--Decoder architecture (right).
3
METHOD
We first provide an overview of our method in Sec. ecc.3.1 then describe two different transformer-based We also provide detailed architectural diagrams in Appendix A.44
model variants Sec. 3.2
3.1 OVERVIEW
Given N sparse input images with known camera poses and intrinsics, denoted as {[I, E,, K;) 1...., N). LVSM synthesizes target image with novel target camera extrinsics E' and intrinsics K'. Each input image has shape HxX33 where Il and W are the image height and width (and
there are 3 color channels).
Framework. As shown in Fig. 2 our LVSM method uses end-to-end transformer model to directly render the target image. LV SM starts by tokenizing the imput images. We first compute pixel-wise Plücker ray embedding (Plucker 1865) for each input view using the camera poses and intrinsics. We denote these Plücker ray embeddings as (P: E xxxxWx6, x6 = 1, ...... .). We patchify the RGB images and Plücker ray embeddings into non-overlapping patches, following the image tokenization layer of VIT (Dosovitskiy al.] 2020). We denote the image and Plücker ray patches of input image I, as {Lg E [Pxpx3] = I,.... HW/p"") and (Pij E RPxpx6 = 1..... HW/ /r"), respectively, where p is the patch size. For each patch, we concatenate its image patch and Plücker ray embedding patch, reshape them into ID vector, and use a linear layer to map it into an input patch token Xyj:
X Linear imput (Lyy (Ly.PP Pul) E 1Rd
where d the latent size, and [ means concatenation.
Similarly, LVSM represents the target pose to be synthesized as its Plücker ray embeddings p e H computed from the given target extrinsies E' and intrinsics Kt, We use the same patchify method and another linear layer to map it to the Plücker ray tokens of the target view, denoted as:
q Linear (Pt) IRd
where P is the Plücker ray embedding the jth patch the target view.
We flatten the input tokens into a ID token sequence, denoted as Iı,..., FL, where I, - N HW/p the sequence length of the input image tokens. We also flatten the target query tokens as q) 44 from the ray embeddings, with 4 HW/p /p² as the sequence length.
LVSM then synthesizes novel view by conditioning on the input view tokens using a full transformer model M:
V1...., D1, M ((g...... ... /1. (I..,, x.).
4

---

Published as a conference paper at ICLR 2025
Specifically, the output token f is an updated version of q₁, containing information for predicting the pixel values of the j"h patch of the target view. More details of model M are described in Sec. 3.2
We recover the spatial structure of output tokens using the inverse operation of the flatten operation. To regress RGB values of the target patch, we employ a linear layer followed by sigmoid function:
I Sigmoid(Linear.nn) (リ₂) E R39*.
(4)
We reshape the predicted RGB values back to the 2D patch in Roxpx3 and then form the synthesized novel view I' by performing the same operation all target patches independently.
Loss Function. Following prior works (Zhang al. 2024 Hong et 2024). we train LVSM with photometric novel view rendering losses:
== MSE(i", I') + Perceptual r):
(5)
where À is the weight for balancing the perceptual loss (Johnson et al. 2016).
3.2 TRANSFORMER-BANE MODEL ARCHITECTURE
In this subsection, we present two LVSM architectures encoder-decoder and decoder-only both designed to minimize 3D inductive biases. Following their name, "encoder-decoder first converts input images to an intermediate latent representation before decoding the final image pixels, whereas "decoder-only" directly outputs the synthesized target view without an intermediate representation, further minimizing inductive bias in its design. Unlike most standard language model transform- ers (Vaswani et al., 2017; Jaegle al. 2021 Radford et al. 2019). which typically use full attention for encoders and causal/cross attention for decoders, we adopt dense full self-attention across all our encoder and decoder architectures. The naming of our models is based on their output characteristics, instead of being strictly tied to the transformer architecture they utilize. Please refer to Appendix A. for further detailed discussion of the naming.
Architecture. The encoder-decoder LVSM comes with a learned latent scene representation for view synthesis, avoiding the use of NeRF, 3DGS, and other representations. The encoder first maps the input tokens to an intermediate 1D array of latent tokens (serving as a latent scene representation). The decoder then predicts the outputs, conditioning on the latent tokens and target pose.
Similar the triplane tokens in LRMs (Hong et al. 2024 Wei al. 2024). we use / learnable latent tokens {ER E Bk = 1....!!! 1} to aggregate information from input tokens {xx)]. The encoder, denoted as Transformer Enc- uses multiple transformer layers with self-attention. We concatenate } and } as the input to Transformer Enc, which performs information aggregation between them to update {ex]. The output tokens that correspond to the latent tokens, denoted as {2x] are used as the intermediate latent scene representation. The other tokens (updated from {r;}. denoted as )x are unused and discarded.
The decoder uses multiple transformer layers with self-attention. In detail, the inputs are the concatenation of the latent tokens { and the target view query tokens {4 By applying self- attention transformer layers over the input tokens, we get output tokens with the same sequence
length as the input. The output tokens that corresponds to the target tokens .1...., qlg are treated as
final outputs y,...., Vlq. and the other tokens (updated from {2r}. denoted {2(}) are unused. This architecture can be formulated as:
2 21..... J/ 21
Transformer.se (x) Th: C1,
et)
(6)
21 21 81. U1, = ( Z1: 91
91.).
(7)
Decoder-Only Architecture. Our alternate, decoder-only model further eliminates the need for an intermediate scene representation. Its architecture similar to the decoder in the encoder-decoder architecture but differs in inputs and model size. We concatenate the two sequences of input tokens {æ,) and target tokens {- The final output {y, } is the decoder's corresponding output for the target tokens {Ø;} The other tokens (updated from (æ:). denoted as (x(}) are unused and discarded. This architecture can be formulated as:
2/,..., 2/4 Ul...... Vi, Transformer Dec-only (2,,..., . Rt, ........4... 41,) Here Transformer Dec-only has multiple full self-attention transformer layers.
(8)
5

---

Published as a conference paper at ICLR 2025
Table 1: Quantitative comparisons on object-level (left) and scene-level (right) view synthesis. For the object-level comparison, we matched the baseline settings with GS-LRM (Zhang et al. (024)) in both input and rendering under both resolution of 256 (Re)-256) and 512 (Res-512). For the scene-leve comparison, we use the same validation dataset used by pixelSplat (Charatan al. 2024
which has 256 resolution.
ABO PSNR
Collisa
2022a
GSO
Dowss et 20
ReulE-enn11 10k Zhou et 2018
1PIPS"
PSNR
130PS
PSNR
5S53
Tiiluse L8M
Ta stal RIME (Res-12)
27.50
02390
0.895
26.54
0.491
0,064
peselNekt al 0021
20,43
0.589
0,550
Zhine
(Rc
20.09
0.925
0.050
CPNNR San
24.11
0.255
24.78
0.213
Ours Docide-Uily
(47)-117)
30.10
00.338
0.928
pinelSppaa
0.861
0.136
LGM [isss at 2004
(ee 256)
20.79
02813
0.158
21.44
0,832
0,122
MVS
0.138
GB-LRM
22rag ot an 2000
(Rc-156)
28.95
0.926
0.074
29.30
0.944
0,511
GS-LRM
Zhing
0.114
30.35
0,925
0.832
29:19
0.322
0/046
Our EICIEG-Urr.. -as-aa
3.114
Ous Decodes Ouly (Rc)
0.944
31.71
0,077
Ous Decodes Only
EXPERIMENTS
We first describe the datasets we use and baseline methods we compare to, then present the results of LVSM for both object-level and scene-level novel view synthesis.
4.1 DATASETS
We train (and evaluate) LVSM on object-level and scene-level datasets separately.
Object-level Datasets. We use the Objaverse dataset (Deitke et al. 2023) to train LVSM. We
follow the rendering settings in GS-LRM (Zhang et al. 2024) and render 32 random views of 730K
objects. We test on two object-level datasets. Google Scanned Objects (Downs et al., 2022) (GSO)
and Amazon Berkeley Objects (Collins et al. 20226) (ABO), GSO and ABO contain I, 1,099 and 1,000
objects, respectively. Following Instant3D (Li et al. 2023) and GS-LRM (Zhang et 2024). we use 4 sparse views as test inputs and another 10 views as target images.
Scene-level Datasets. We use the RealEstate 10K dataset (Zhou et al. 2018), which contains 80K video clips curated from 10K Youtube videos of both indoor and outdoor scenes. We follow the train/test data split used in pixelSplat (Charatan et al. 2024).
4.2 TRAINING DETAILS
Improving Training Stability. We observe that LVSM training crashes with plain transformer layers (Vaswani al. 2017) due to exploding gradients. We empirically find that using QK- Norm (Henry et al. (020) in the transformer layers stabilizes training. This observation is consistent with Bruce et al. (2024) and Esser al. (2024). We also skip optimization steps with gradient norm 5.0 in addition to the standard 1.0 gradient clipping.
Efficient Training Techniques. We use FlastAttention-v2 (Dao 2023) in the xFormers (Lefaudeux ct al. 2022), gradient checkpointing (Chen et al. 2016). and mixed -eeed-peeiisoon training with 1 data type to accelerate training.
Other Details. For more model and training details, please refer to Appendix A.2 A.2 and detailed
model architecture diagram (Fig. g
4.3 COMPARISON TO BASELINES
In this section, we describe our experimental setup and datasets (Sec. 4.1). introduce our model training details (Sec. 4.2) report evaluation results (Sec. 4.3) and perform an ablation study (Sec. 4.4p
Object-Level Results. We compare with Instant3D's Triplane-LRM (Li et al.) 2023) and GS- LRM (Zhag et al., 2024) at resolution of 512. As shown on the left side of Table [ our LVSM method achieves the best performance. In particular, at 512 resolution, our decoder-only LVSM achieves a 3 dB and 2.8 dB PSNR gain against the best prior method GS-LRM on ABO and GSO, respectively; our encoder-decoder LVSM achieves performance comparable to GS-LRM.
We also compare with LGM (Tang et al.) 2024) at resolution of 256, the official LGM model is trained with that resolution. We also report the performance of models trained on the resolution of 256. Compared with the best prior work GS- GS-L our decoder-only LVSM demonstrates 3.5 dB and 2.2 dB PSNR gain on ABO and GSO, respectively; our encoder-decoder LVSM yields slightly better performance than GS-L.RM.

---

Published as a conference paper at ICLR 2025
Tapet Images
TriplaneLLRe
GS-LAM
Ours Encoder- Encoder-eDoodke
Ours Dacoder-Oaly
Ground teath
Figure 3: Object-level visual comparison at 512 resolution. Given 4 sparse input posed images (leftmost column), compare our high-res object-level novel-view rendering results with two baselines: Instant3D's Triplane-LRM (Li al. 2023) and GS-LRM (Res-512) (Zhang al. 2024). Both Encoder-Decoder and Decoder- Only models exhibit fewer floaters (first example) fewer blurry artifacts (second example), compared to the baselines. Our Decoder-Only model effectively handles complex geometry, including small holes (third example) and thin structures (fourth example). Additionally, it preserves high- geg-ffreqency texture details (last example).
These significant performance gains validate the effectiveness of our design target of removing 3D inductive bias. More interestingly, the larger performance gain on ABO suggests that LVSM can handle challenging materials, which are difficult for current handcrafted 3D representations, The qualitative results in Fig. and Fig. also validate the high degree of realism of LVSM, especially for examples with specular materials, detailed textures, and thin, complex geometry.
Scene-Level Results. We compare
scene-level inputs with PixelNeRF (Yu et al., 202
GPNR (Suhail et al.) 2022a), Du et 2023), pixelSplat (Charatan et al. 2024), MVSplat (Chen et al. 2024) and GS-LRM (Zhang et al. 2024). As shown the right side of Table our decoder-only LVSM shows a 1.6 dB PSNR gain compared with the best prior work. GS-LRM. Our encoder-decoder LVSM also demonstrates comparable results to GS-LRM. These improvements
be observed qualitatively in Fig. 4 where LVSM has fewer floaters and better performance on thin structures and specular materials, consistent with the object-level results. These outcomes again validate the efficacy of our design of using minimal 3D inductive bias.
LVSM Trained with Only 1 GPU. Limited computing is a key bottleneck for academic research. To show the potential of LVSM using academic-level resources, we train LVSM on the scene- level dataset (Zhou al.) 2018) following the setting of pixelSplat (Charatan et al. 2024) and MVSplat (Chen et al. 2024), with only single 100 80G GPU for 7 days. In this experiment, we use smaller decoder-only model (denoted LVmM-) with transformer layers and a smaller batch size of 64 (in contrast to the default setting of 24 layers and batch size 512). Our decoder- only LVSM-small shows a performance of 27.66 dB PSNR, 0.870 SSIM, and 0.129 LPIPS. This performance surpasses the prior best I-GPU-trained model, MVSplat. with 1.3 dB PSNR gain. We also train a decoder-only LVSM (12 transformer layers, batch size 64) with 2 GPUs for days, exhibiting performance of 28.56 dB PSNR, 0.889 SSIM, and 0. 12 LPIPS. This performance is
7

---

Published as a conference paper at ICLR 2025
Imput images
pixelSphtt
MV Splat
GS LRM
Ours Encoder Docader
Ours Deuoder Only
Ground truth
and decoder-only LVSM on soene- level view synthesis, comparing them the prior leading bascline methods, namely pixelSplat (Charatan al. 2024. MVSplat (Chen 2024). and GS-LRM (Zhang al. 2024). Our results exhibit fewer texture and geometric artifacts, feature more realistic specular reflections, and are closer to ground truth images.
Figure 4: Scene-level visual comparison. We evaluate
even better than GS-LRM with 24 transformer layers trained on 64 GPUs. These results show the promising potential of LVSM for academic research.
4.4 ABLATION STUDIES
Model Size. In Tab. g we ablate the model size designs of both LVSM variants on both object and scene level. To save resources, these experiments run with GPUs and a total batch size of 64,
For the encoder-decoder LVSM, we maintain the total number of transformer layers while allocating different number of layers the encoder and decoder. We observe that using helps performance while using more encoder layers harms performance. We hypothesize that this because the encoder uses the latent representation compress the input image information, and a deeper encoder makes this compression process harder to learn, resulting in greater compression errors. This observation suggests that using the inductive bias of the encoder and intermediate latent representation may not be optimal for the final quality, aligning with our observation that the decoder-only variant outperforms the encoder-decoder variant.
decoder layers
For the decoder-only LVSM, we experiment with using different numbers of transformer layers and model sizes in the decoder. The experiment verifies that the decoder-only LVSM shows increasing performance when using more transformer layers, and validates the scalability of the decoder-only LVSM.
Model Architecture. As shown in Tab. 3 we evaluate the effectiveness of our model designs. We also visualize the equivalent attention mask for each design Fig. 9 for better illustration. To save resources, the encoder-decoder experiments here and a decreased number of training target views of 8. The decoder-only experiment run with our original setup.
with 32 GPUs, total batch size of 256,
8

---

Published as a conference paper at ICLR 2025
Table 2: Ablations studies on model sizes. The following experi- Table 3: Ablations studies on model architecture.
ments are all run with 8 GPUs and a total batch size of 64.
GSD
Downs et 3500
2002
PSNR:
12:P5
Realbsnne
Porams
PSNK
SSI
LPIPS
|SNNK
SSIM
LPIPS
Oune Deosee-DDoor
0,831
Ours Encoder Decoder 18)
1738
26.48
0,501
0,065
28.32
0.888
0.117
Oun wio laters'
26.63
0.503
0.061
(uns Uncoder becoder (12
1738
01889
0,076
27.39
(1:00)
0.157
prodiction
36.27
0.897
0,072
Our Erander Decodar (18
175M
24.74
0.673
0
26.30
0.855
0.152
Our wi pure ciess at: decoder
24.00
03876
0,085
Ours Decoder Only 124 layers
171M
27.06
0.910
0.085
25.89
0.894
0.308
RediEssuno (0k (2hna et B0LK
Ouee Dasolk=OOill (18 layers
12884
26.31
0.007
0.087
28.77
0.892
0.100
PSNB:
S500
Ours DesodesCully (12 layens
S6M
26.11
0.896
0.065
28.61
0.850
0.111
Ous Decodd: Only
29.67
0.906
0,696
Our Decoder-Uhily layers)
4330
24.15
0.855
0.092
27.02
0.809
0.129
Oun oo penporch prodiction
0.887
0.101
Our encoder-decoder L.SM leverages an encoder to transform input images into a set of 1D tokens that serve as an intermediate latent representation of the 3D scene. A decoder can then render novel view images from this latent representation. While the encoder-decoder LVSM shares high-level conceptual similarities with SRT (Sajjadi et al. (022)- latent tokens an intermediate latent representation without an explicit 3D representation- encoder-decoder LVSM introduces very distinct architecture that significantly enhances performance. In the following paragraphs, we evaluate many of our design decisions and also consider some alternate components based on SRT, showing that our design yields significant improvements.
are based on transformers and use ID
To generate input tokens for the encoder, we draw inspiration from ViT (Dosovitskiy et al., (020)) and recent LRMs (Wei et al. [2024] (Zhang et al. 2024). Specifically, we tokenize the input images and their poses by simply splitting the concatenated input views and Plücker ray embeddings into non-overlapping patches. In contrast, SRT relies on shallow convolutional neural networks (CNNs) to extract patch features, which are then flattened into tokens. As ablation study, tried SRT's CNN-based tokenizing method and observed that it makes training more unstable with a larger grad norm, which leads to worse performance. As demonstrated in Tab. 3, replacing our simple tokenizer with SRT's CNN-based tokenizer degrades performance (ours w/ CNN tokenizer).
Tokenization Strategy for Encoder Input:
Fixed-Length Latent Encoding for Efficient Rendering: Our encoder employs self-attention to progressively compress the information from posed input images into a fixed-length set of ID latent tokens. This design ensures consistent rendering speed, regardless of the number of input images, as shown in Fig. 6 This differs from approaches like SRT where latent token size increases linearly with input views, reducing rendering efficiency as the number of views grows.
The decoder of our encoder-decoder model utilizes pure (bidirectional) self-attention, allowing latent tokens and output target image tokens to attend each other. This enables i) latent tokens to be updated by fusing information from themselves and from other tokens, which also means the parameters of the decoder are a part of the scene representation; ii) output patch pixels can also attend to other patches for joint updates, ensuring the global awareness of the rendered target image. We ablate our full-attention design choice by experimenting with different attention mechanisms, illustrated in Fig. 9 As shown in Tab. disabling either the latents' updating (ours w/o latents' updating) or the joint updating of direct output pixel patches (ours w/ per-patch prediction) significantly degrades performance. SRT cannot support either mechanism because it employs a decoder with pure cross-attention. We experiment with an LVSM variant by adopting SRT's decoder designs. As shown in Tab. 3 (ours w/ pure cross-att decoder), this modification leads to reduced performance.
Bidirectional Self-Attention Decoder:
The decoder-only LVSM further pushes towards eliminating inductive bias, bypassing an intermediate representation altogether. It adopts single-stream transformer to directly convert the input multi-view tokens into target view tokens, treating view synthesis like eequence-to-sequee which is fundamentally different from prior work. We ablate the importance of the joint prediction of target image patches in the decoder-only LVSM. We design variant where the colors of each target pose patch are predicted independently. without applying self-attention across other target pose patches. We achieve this by letting each transformer layer's 's key and value matrices only consist of updated input image tokens, while both the updated input image tokens and target pose tokens form the query matrices. As shown on the bottom part of Tab. 3 this variant shows worse performance, with a 0.7 dB PSNR degradation. This result demonstrates the importance of using both input and target tokens as context tokens for information exchange using the simplest full self-attention transformer, which is consistent with our philosophy of reducing inductive bias.
translation task,

---

Published as a conference paper at ICLR 2025
Rendering 100
30
Oun Decodar-Dall
Uncoder
Oun Dovolar-Only
Ous Encoder Decode:
Rea time
GS-IRM
20
10
14
16
Input view sumneet
Enput ViOW numther
Figure 5: Zero-shot generalization to different number of input images the GSO dataset (Downs) al. 2022). We note that models are trained with just 4 input views.
Figure 6: Rendering FPS with different number of input images. We test the FPS on the object level under 256 256 resolution. We refer rendering as the decoding process, which synthesizes novel views from latent tokens or input images.
4.5 DISCUSSION
Zero-shot Generalization to More Input Views. We compare our LVSM with GS-LRM by taking different numbers of imput views during inference. We report the results on object-level inputs. Note that these models are trained with 4 input views, and tested on varying numbers of input views in a zero-shot manner. As shown in Fig. 5 our decoder-only LVSM shows increasingly better performance when using more input views, verifying the scalability of our model design at test time. Our encoder-decoder LVSM shows a similar pattern as GS-LRM, i.e., it exhibits a performance drop when using more than 8 input views. We conjecture that this is due the inductive bias of the encoder-decode design, i.c., using intermediate representation as a compression of input information limits performance. In addition, our single-inpll results (input view number 1) are competitive and even outperform some of the baseline that takes 4 images as input. These performance patterns validate our design target of using minimal 3D inductive bias for learning fully data-driven rendering model and cohere with our discussion in Sec. 3.2
Encoder-Decoder versus Decoder-Only. As mentioned in Sec. 3 the decoder-only and encoder- decoder architectures exhibit different trade-off's in speed, quality, and potential.
The encoder-decodee model transforms 2D image inputs into a fixed-length set of 1D latent tokens, which serve as a compressed representation of the 3D scene. This approach simplifies the decoder, reducing its model size. Furthermore, during the rendering/decoding process, the decoder always receives fixed number of tokens, regardless of the number of input images, ensuring a consistent rendering speed. As result, this model offers improved rendering efficiency, as shown in Fig. 6 Additionally, the use of 1D latent tokens as the latent representation for the 3D scene opens up the possibility of integrating this model with generative approaches for 3D content generation on its latent space. Nonetheless, the compression process can result in information loss, as the fixed latent token length is usually smaller than the original image token length, imposing an upper bound on performance. This characteristic of the encoder-decodee LVSM mirrors prior encoder-decoder LRMs; however, unlike those LRMs, our encoder-decoder LVSM does not have an explicit 3D structure.
In contrast, the decoder-only model learns direct mapping from the input image the target novel view, showcasing better scalability. For example, as the number of input images increases, the model can leverage all available information, resulting in improved novel view synthesis quality. However, this property also leads to a linear increase in input image tokens, leading to quadratic growth in computational complexity and limiting rendering speed.
we observe that our LVSM also works with a single input view for many cases, even though the model is trained with multi-view images during training. This observation shows the capability of LVSM to understand the 3D world, e.g., understanding depth, rather than performing purely pixel-level view interpolation.
Single Input Image. As shown in our project page, Fig. I and Fig. 5
CONCLUSION
In this work, we presented the Large View Synthesis Model (LVSM), a transformer-batt approach designed to minimize 3D inductive biases for scalable and generalizable novel view synthesis. Our two representations like NeRF and 3D Gaussian Splatting, allowing the model to learn priors directly from data, leading to more flexible and scalable novel view synthesis. The decoder-only LVSM, with its minimal inductive biases, excels in scalability, zero-shot generalization, and rendering quality, while the encoder-decoder LVSM achieves faster inference due to its fully learned latent scene representation. Both models demonstrate superior performance across diverse benchmarks and mark an important step towards general and scalable novel view synthesis in complex, real-world scenarios.
and lecoder-onll-bbyaas. physical-rendering--e
3D
10

---

Published as a conference paper at ICLR 2025
Acknowledgements. We thank Kalyan Sunkavalli for helpful discussions and support. This work was done when Haian Jin, Hanwen Jiang, and Tianyuan Zhang were research interns at Adobe Research. This work was also partly funded by the National Science Foundation [IS-2211259, 20-22.10084).
REFERENCES
Jonathan T. Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Partin-Bruuall,, and Pratul P. Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. ICCV, 2021.
Jonathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul P. Srinivasan, and Peter Hedman. Zip-nerf: Anti-aliased grid-based neural radiance fields. ICCV, 2023.
Jake Bruce, Michael D Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, et al. Genie: Generative interactive environments. In Forty-firsi International Conference on Machine Learning, 2024.
Eric R Chan, Connor Z Lin, Matthew Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas Guibas, Jonathan Tremblay, Sameh Khamis, et al. Efficient geometry-ane aware 3d generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16123-16133, 2022.
the
David Charatan, Sizhe Lester Li, Andrea Tagliasacchi, and Vincent Sitzmann. pixelsplat: 3d gaussian splats from image pairs for scalable generalizable 3d reconstruction. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp- 19457-19467, 2024.
Gaurav Chaurasia, Sylvain Duchene, Olga Hornung. and George Drettakis. Depth synthesis and local warps for plausible image-based navigation. ACM transactions on graphics (TOG), 32 (3):1-12, 2013.
Anpei Chen, Zexiang Xu, Fuqiang Zhao, Xiaoshuai Zhang, Fanbo Xiang, Jingyi Yu, and Hao Su. Mvsnerf: Fast generalizable radiance field reconstruction from multi-view stereo, 2021. URL https://arxiv org abs 2t1.://....rraaa//1..11....1. 15595
Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. Tensorf: Tensorial radiance fields. In European conference on computer vision, pp. 333-350. Springer, 2022.
Anpei Chen, Zexiang Xu, Xinyue Wei, Siyu Tang, Hao Su, and Andreas Geiger. Factor fields: A unified framework for neural fields and beyond. arXiv preprint arXiv:2302.01226, 2023.
Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost. arXiv preprint 16Xiv:160.006774 2016.
Yuedong Chen, Haofei Xu, Chuanxia Zheng, Bohan Zhuang. Mare Pollefeys, Andreas Geiger, Tat-Jen Cham, and Jianfei Cai. Mvsplat: Efficient 3d gaussian splatting from sparse multi-view images, 2024.
14627
Inchang Choi, Orazio Gallo, Alejandro Troccoli, Min H Kim, and Jan Kautz. Extreme view synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7781-7790, 2019.
Jasmine Collins, Shubham Goel, Kenan Deng, Achleshwar Luthra, Leon Xu, Erhan Gundogdu, Xi Zhang, Tomas F, Yago Vicente, Thomas Dideriksen, Himanshu Arora, Matthieu Guillaumin, and Jitendra Malik. Abo: Dataset and benchmarks for real-world object understanding, 2022a.
(RRLhhtt:://// http://aaxvvvv..o org/ wwt.//ssllll.ao 06199
Jasmine Collins, Shubham Goel, Kenan Deng, Achleshwar Luthra, Leon Xu, Erhan Gundogdu, Xi Zhang, Tomas F Yago Vicente, Thomas Dideriksen, Himanshu Arora, al. Abo: Dataset and benchmarks for rorl-world 3d object understanding. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 21 21126-21 136, 2022b.
11

---

Published as a conference paper at ICLR 2025
Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint 23000:22007.0891,,, 2023.
Abe Davis, Mare Levoy, and Fredo Durand. Unstructured light fields. Computer Graphics Forum,
pp. 305-314, 2012.
Paul E. Debevec, Camillo Jose Taylor, and Jitendra Malik. Modeling and rendering architecture from photographs: A hybrid geometry- and image-based approach. Seminal Graphics Papers: Pushing the Boundaries, Volume 2, 1996. URL wtt.accpee//eeeeeeeett..o.oo.o..
[CorpusI:: 26094115
Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of anno- tated 3d objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13142-13153, 2023.
Jacob Devlin, Ming-Wer Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, 2019. URL httpp:://axxvv.org
0e5518800048055
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition scale, ArXiv, abs/2010.11929, 2020. URL https: /appaapp....... wew.sesesnnccoollaa.... 225039882
CrggCorppssDD
Laura Downs, Anthony Francis, Nate Koenig, Brandon Kinman, Ryan Hickman, Krista Reymann, Thomas B. McHugh, and Vincent Vanhoucke. Google scanned objects: A high-quality dataset of 3d scanned household items, 2022. URL htt.th:ttt///rtttt. org/ /2004. .1921.1..11..
Yilun Du, Cameron Smith, Ayush Tewari, and Vincent Sitzmann. Learning render novel views from wide-baseline stereo pairs, 2023. URL
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first International Conference on Machine Learning. 2024.
Wanquan Feng, Jin Li, Hongrui Cai, Xiaonan Luo, and Juyong Zhang. Neural points: Point cloud representation with neural fields for arbitrary upsampling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18633-18642, 2022.
Sara Fridovich-Keil, Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance fields without neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5501-5510, 2022.
Ruiqi Gao*, Aleksander Holynski Philipp Henzler, Arthur Brussee, Ricardo Martin-Brualla, Pratul P. Srinivasan, Jonathan T. Barron, and Ben Poole*. Cat3d: Create anything in 3d with multi-view diffusion models. Advances in Neural Information Processing Systems, 2024.
Steven J. Gortler, Radek Grzeszezuk, Richard Szeliski, and Michael Cohen. The lumigraph. Proceedings of the r3r annual conference Computer graphics and interactive techniques, 1996. URL https:://///... app. http::/://app.ssmanneeco...o
Hao He, Yinghao Xu, Yuwei Guo, Gordon Wetzstein, Bo Dai, Hongsheng Li, and Ceyuan Yang. Cameractrl: Enabling camera control for text-to-video generation, 2024.
Peter Hedman, Julien Philip, True Price, Jan- Mun-Michael Frahm, George Drettakis, and Gabriel Brostow. Deep blending for free- viee-vien image-based rendering. ACM Transactions on Graphics (ToG), 37(6):1-15, 2018
Peter Hedman, Pratul P Srinivasan, Ben Mildenhall, Jonathan T Barron, and Paul Debevec. Baking neural radiance fields for real-time view synthesis. Proceedings of the IEEE/CVF international conference on computer vision, pp. 5875-5884, 2021.
12

---

Published as a conference paper at ICLR 2025
Benno Heigl, Reinhard Koch, Mare Pollefeys, Joachim Denzler, and Luc Van Gool. Plenoptic modeling and rendering from image sequences taken by hand-held camera. In Mustererkenmung 1999: 21. DAGM- MSG-Syyposium Bonn, 15.-17. September 1999, pp. 94-101. Springer, 1999.
Alex Henry, Prudhvi Raj Dachapally, Shubham Pawar, and Yuxuan Chen. Query-key normalization for transformers. arXiv preprini arXiv:2010.042455 2020.
Yicong Hong, Kai Zhang, Jiuxiang Gu, Sai Bi, Yang Zhou, Difan Liu, Feng Liu, Kalyan Sunkavalli, Trung Bui, and Hao Tan. Lrm: Large reconstruction model for single image to 3d, 2024. URL https://arxe org/ abs 2311. 24400 04400
Andrew Jaegle, Sebastian Borgeaud. Lean-Baptin Alayrac, Carl Doersch, Catalin Ionescu, David Ding, Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, et al. Perceiver io: A general architecture for structured inputs & outputs. arXiv preprint 1rXiv:2107.14999 2021.
Michal Jancosek and Tomas Pajdla. Multi-view reconstruction preserving weakly-supported surfaces. In CVPR 2011, pp. 3121-3128 IEEE, 2011.
Hanwen Jiang, Zhenyu Jiang, Yue Zhao, and Qixing Huang. Leap: Liberate sparse-view 3d modeling from camera poses. arXiv preprint arXiv:2310.01410, 2023.
Hanwen Jiang, Zhenyu Jiang, Kristen Grauman, and Yuke Zhu. Few-view object reconstruction with unknown categories and camera poses. In 2024 International Conference on 3D Vision (3DV), pp. 31-41, IEEE, 2024.
Mohammad Mahdi Johari, Yann Lepoittevin, and François Fleuret. Geonerf: Generalizing nerf with geometry priors, 2022. URL wttp:/rgggababbo...o......oosba aaaaaass/.l..../ 13539
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14, pp. 694-711. Springer, 2016.
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering, ACM Transactions on Graphics, 42(4), July 2023. URL sam. inria. fr/ Sunggraph/ddgqusssee wttp:aasssssaasppatt..ii.
Diederik Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6980, 2014.
Xin Kong, Shikun Liu, Xiaoyang Lyu, Marwan Taher, Xiaojuan Qi, and Andrew J Davison. Eschernet: A generative model for scalable view synthesis. arXiv preprint arXiv:2402. 2024.
Jonáš Kulhánck, Erik Derner, Torsten Sattler, and Robert Babuška. Viewformer: Nerf-free neural rendering from few images using transformers. In European Conference on Computer Vision (ECCV), 2022.
Benjamin Lefaudeux, Francisco Massa, Diana I Liskovich, Wenhan Xiong, Vittorio Caggiano, Sean Naren, Min Xu, Jieru Hu, Marta Tintore, Susan Zhang, et al. xformers: A modular and hackable transformer modelling library, 2022.
Mare Levoy and Pat Hanrahan. Light field rendering. Proceedings of the 23rd annual con- ference on Computer graphics and interactive techniques, 1996. URL https://api
semant www.hlar........ Starg/Corpue Io:
Jiahao Li, Hao Tan, Kai Zhang, Zexiang Xu, Fujun Luan, Yinghao Xu, Yicong Hong, Kalyan Sunkavalli, Greg Shakhnarovich, and Sai Bi. Instant3d: Fast text-to-3d with sparse-view generation and large reconstruction model, 2023. URL UtLhttps:: /ttp://aaivv.v.... wttp:/vv..gggaa////111..3.
Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt. Neural sparse voxel fields. Advances in Neural Information Processing Systems, 33:15651-15663. 2020.
Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero-1-to-3; Zero-shot one image to 3d object, 2023a.
13

---

Published as a conference paper at ICLR 2025
Yuan Liu, Sida Peng, Lingjie Liu, Qianqian Wang, Peng Wang, Christian Theobalt, Xiaowei Zhou, and Wenping Wang. Neural rays for occlusion-awarr image- rendering. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattemn Recognition, pp. 7824-7833, 2022.
Yuan Liu, Cheng Lin, Zijiao Zeng, Xiaoxiao Long, Lingjie Liu, Taku Komura, and Wenping Wang. images from a single-view image. arXiv preprint
Syncdreamer: Generating arXiv:2309.03455, 2023b.
Xiaoxiao Long, Yuan-Chen Guo, Cheng Lin, Yuan Liu, Zhiyang Dou, Lingjie Liu, Yuexin Ma, Song-Hai Zhang, Marc Habermann, Christian Theobalt, et Wonder3d: Single image to 3d using cross-domain diffusion. arXiv preprint arXiv:23 10. 15008, 2023.
Ricardo Martin-Brualla, Noha Radwan, Mchdi SM Sajjadi, Jonathan Barron, Alexey Dosovitskiy, and Daniel Duckworth. Nerf in the wild: Neural radiance fields for unconstrained photo collections. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 7210- 7219, 2021.
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis, 2020. URL attps://arxie
08934
Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM Transactions on Graphics, -1 July 2022. ISSN 1557-7368. doi: 00.1145/52822333330122 URL URL http:://x.
1145/3528223 0115/3528223.3531227 3530127
Michael Niemeyer, Jonathan T Barron, Ben Mildenhall, Mehdi SM Sajjadi, Andreas Geiger, and Noha Radwan. Regnerf: Regularizing neural radiance fields for view synthesis from sparse inputs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 1480-5490, 2022.
Eric Penner and Li Zhang. Soft 3d reconstruction for view synthesis. ACM Transactions on Graphics (TOG), 36(6):1-11, 2017. 2017.
Julius Plucker. Xvii. new geometry of space. Philosophical Transactions of the Royal Society of London, pp. 725-791, 1865.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXivv2204.06125, 1(2):3 2022.
Christian Reiser, Songyou Peng, radiance fields with thousands of tiny mlps. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 14335-14344 2021.
Yiyi Liao, and Andreas Geiger. Kilonerf: Speeding up neural
Christian Reiser, Rick Szeliski, Dor Verbin, Pratul Srinivasan, Ben Mildenhall, Andreas Geiger, Jon Barron, and Peter Hedman. Merf: Memory-efficteee radiance fields for real-time view synthesis in unbounded scenes. ACM Transactions on Graphics (TOG), 42(4): 2023.
Robin Rombach, Patrick Esser, and Björ Ommer. Geometry -free view synthesis: Transformers and no 3d priors. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 14356-14366, 2021.
Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: Image-to-image diffusion models. In ACM SIGGRAPH 2022 conference proceedings, pp. 1-10, 2022a.
Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. IEEE transactions on pattern analysis and machine intelligence, 45(4):4713-472) 2022b.
14

---

Published as a conference paper at ICLR 2025
Mehdi SM Sajjadi, Henning Meyer, Etienne Pot, Urs Bergmann, Klaus Greff, Noha Radwan, Suhani Vora, Mario Lučić, Daniel Duckworth, Alexey Dosovitskiy, et al. Scene representation transformer: Geometry- novel view synthesis through set-latent scene representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2239, 2022.
Ruoxi Shi, Hansheng Chen, Zhuoyang Zhang, Minghua Liu, Chao Xu, Xinyue Wei, Linghao Chen, Chong Zeng, and Hao Su. Zero123+++ a single image to consistent multi-view diffusion base model, 2023a.
Yichun Shi, Peng Wang, Jianglong Ye, Long Mai, Kejie Li, and Xiao Yang. Mvdream: Multi-view diffusion for 3d generation. arXiv:2308. 165 2023b.
Sudipta Sinha, Drew Steedly, and Rick Szeliski. Piecewise planar stereo for image- based rendering. 2009.
In 2009 International Conference Computer Vision, pp.
Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene representation networks: Contin- uous 3d- dd-ssructure-awart neural scene representations. Advances in Neural Information Processing Systems, 32, 2019.
Vincent Sitzmann, Semon Rezchikov, Bill Freeman, Josh Tenenbaum, and Fredo Durand. Light field networks: Neural scene representations with single-cvaluation rendering. Advances in Neural Information Processing Systems, 1931331939399, 2021.
Mohammed Suhail, Carlos Esteves, Leonid Sigal, and Ameesh Makadia. Generalizable patch-based neural rendering. In European Conference Computer Vision, pp. 156-174. Springer, 2022a.
Mohammed Suhail, Carlos Esteves, Leonid Sigal, and Ameesh Makadia. Light field neural rendering, 2022b. URL
abs 2112.112111.. 09687
Cheng Sun, Min Sun, and Hwann-Tzong Chen. Direct voxel grid optimization: Super-fast conver- gence for radiance fields reconstruction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5459-5469, 2022.
Stanislaw Szymanowicz, Eldar Insafutdinov, Chuanxia Zheng, Dylan Campbell, Joao Henriques, Christian Rupprecht, and Andrea Vedaldi. Flash3d: Feed-forward generalisable 3d scene recon- struction from single image. arxiv, 2024a.
Stanislaw Szymanowicz, Christian Rupprecht, and Andrea Vedaldi. Splatter image: Ultra-fast single-view 3d reconstruction. Conference on Computer Vision and Pattern Recognition (CVPR), 2024b.
Jiaxiang Tang, Zhaoxi Chen, Xiaokang Chen, Tengfei Wang, Gang Zeng, and Ziwei Liu. Lgm: Large multi-view gaussian model for high-resolution 3d content creation. arXiv preprint arXiv:2402.05054, 2024.
Joseph Tung, Gene Chou, Ruojin Cai, Guandao Yang, Kai Zhang, Gordon Wetzstein, Bharath Hariharan, and Noah Snavely. Megascenes: Scene-level view synthesis at scale. In ECCV, 2024.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Ad- vances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL
files/paper/22177
fi File/ff5ee23355 7dee91 1 c4a84 Saaeaa45aa-Paper...
Dor Verbin, Peter Hedman, Ben Mildenhall, Todd Zickler, Jonathan T. Barron, and Pratul P. Srinivasan. Ref-NeRF: Structured view-dependent appearance for neural radiance fields. CVPR, 2022.
Vikram Voleti, Chun-Har Yao, Mark Boss, Adam Letts, David Pankratz, Dmitry Tochilkin, Christian Laforte, Robin Rombach, and Varun Jampani. Sv3d: Novel multi-view synthesis and 3d generation from a single image using latent video diffusion. In European Conference on Computer Vision, pp. 439 Springer, 2025.
15

---

Published as a conference paper at ICLR 2025
Peng Wang, Hao Tan, Sai Bi, Yinghao Xu, Fujun Luan, Kalyan Sunkavalli, Wenping Wang. Zexiang Xu, and Kai Zhang. Pf-lrm: Pose-free large reconstruction model for joint pose and shape prediction. arXiv preprint arXiv:2311.12024 2023a.
Qianqian Wang, Zhicheng Wang, Kyle Genova, Pratul Srinivasan, Howard Zhou, Jonathan T. Barron, Ricardo Martin-Brualle Noah Snavely, and Thomas Funkhouser. Ibrnet: Learning multi-view image-based rendering, 2021a. URL https: /ttp:///axvvvv.. wttp:oo/////a../.// httpp/////a..i.ooo/a////////////
13090
Zhouxia Wang. Ziyang Yuan, Xintao Wang. Yaowei Li, Tianshui Chen, Menghan Xia, Ping Luo, and Ying Shan. Motionctrl: A unified and flexible motion controller for video generation. 2023b.
Zirui Wang, Shangzhe Wu, Weidi Xie, Min Chen, and Victor Adrian Prisacariu. Nerf-: Neural radiance fields without known camera parameters. arXiv preprint arXiv:2102.00766, 2021b.
Daniel Watson, William Chan, Ricardo Martin-Brualla, Jonathan Ho, Andrea Tagliasacchi, and Mo- hammad Norouzi. Novel view synthesis with diffusson models, arXiv preprint arXiv:2210.04628,
2022.
Xinyue Wei, Kai Zhang, Sai Bi, Hao Tan, Fujun Luan, Valentin Deschaintre, Kalyan Sunkavalli, Hao Su, and Zexiang Xu. Meshlmm: Large reconstruction model for high-quality mesh. arXiv preprint arXiv:2404. 12385, 2024.
Desai Xie, Sai Bi, Zhixin Shu, Kai Zhang, Zexiang Xu, Yi Zhou, Sören Pirk, Arie Kaufman, Xin Sun, and Hao Tan. Lrm-zero: Training large reconstruction models with synthesized data. arXiv preprint (xXi::2406.09371, 2024.
Qiangeng Xu, Zexiang Xu, Julien Philip, Sai Bi, Zhixin Shu, Kalyan Sunkavalli, and Ulrich Neumann. Point-nerf: Point-based neural radiance fields. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5438 04488,548.11 2022.
Yinghao Xu, Hao Tan, Fujun Luan, Sai Bi, Peng Wang, Jiahao Li, Zifan Shi, Kalyan Sunkavalli, Gordon Wetzstein, Zexiang Xu, and Kai Zhang. Dmv3d: Denoising multi-view diffusion using 3d large reconstruction model, 2023.
Yunzhi Yan, Zhen Xu, Haotong Lin, Haian Jin, Haoyu Guo, Yida Wang, Kun Zhan, Xianpeng Lang, Hujun Bao, Xiaowei Zhou, and Sida Peng. Streetcrafter: Street view synthesis with controllable video diffusion models, 2024.
wttp:///wwrraab////.///// 13188
Jiayu Yang, Ziang Cheng, Yunfei Duan, Pan Ji, and Hongdong Li. Consistnet: Enforcing 3d consistency for multi-view images diffusion. arXiv, 2023.
Jianglong Ye, Peng Wang, Kejie Li. Yichun Shi, and Heng Wang. Consistent-I-to-2 Consistent image to 3d view synthesis via geometry-aware diffusion models. arXiv preprint arXiv:2310.03020, 2023.
Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixelnerf: Neural radiance fields from one or few images, 2021. URL CtL.ttts:: /ttp///axvvvv//v http///wwwww.oaaaaaa/o///a/ 2012. 021 90
Wangbo Yu, Jinbo Xing, Li Yuan, Wenbo Hu, Xiaoyu Li, Zhipeng Huang. Xiangjun Gao, Tien- Tsin Wong, Ying Shan, and Yonghong Tian. Viewcrafter: Taming video diffusion models for high-fidelity novel view synthesis. arXiv preprint 2409:24000000444 2024a.
Zehao Yu, Anpei Chen, Binbin Huang, Torsten Sattler, and Andreas Geiger. Mip-splatting: Alias-free 3d gaussian splatting. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 19447-19456, 2024b.
Kai Zhang, Sai Bi, Hao Tan, Yuanbo Xiangli, Nanxuan Zhao, Kalyan Sunkavalli, and Zexiang Xu. Gs-Imm: Large reconstruction model for 3d gaussian splatting, 2024. URL https: //arxiv
org/ ab///bb//a0 2404 19702
Qiang Zhang, Seung- Hwan Back, Szymon Rusinkiewicz, and Felix Heide. Differentiable point-based radiance fields for efficient view synthesis. In SIGGRAPH Asia 2022 Conference Papers, pp. 1-12, 2022.
16

---

Published as a conference paper at ICLR 2025
Chuanxia Zheng and Andrea Vedaldi. Free3d: Consistent novel view synthesis without 3d represen- tation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9720-97311 2024.
Tinghui Zhou, Shubham Tulsiani, Weilun Sun, Jitendra Malik, and Alexei A Efros. View synthesis by appearance flow. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings. Part IV 14, pp. 286-301. Springer, 2016.
Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnification: Learning view synthesis using multiplane images. In SIGGRAPH, 2018.
17

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 NAMING CLARIFICATION
Importantly, we clarify that the naming of 'encoder' and 'decoder' are based on their output characteristics--iee, the encoder outputs the latent while the decoder outputs the tareet-rather than being strictly tied to the transformer architecture they utilize. For instance, in the encoder- decoder model, the decoder consists of multiple transformer layers with self-attention (referred to as Transformer Encoder layers in the original transformer paper). However, we designate it as a decoder because its primary function is to output results. These terminologies align with conventions used in LLMs (Vaswani et al. 2017 Radford et al. 2019 Devlin et al. 2019). Notably, we apply self-attention to all tokens in every transformer block of both models without introducing special attention masks or other architectural biases, in line with our philosophy of minimizing inductive bias.
A.2 ADDITIONAL IMPLEMENTATION DETAILS
We train LVSM with 64 A100 GPUs with a batch size of 8 per GPU. We use cosine learning rate schedule with peak learning rate of 4e-4 and a warmup of 2500 iterations. We train LVSM for 80k iterations on the object and 100k on scene data. LVSM uses a image patch size of p = 8 and token dimension d 768. The details of the transformer layers follow GS-LRM(Zhang et al. 2024) with an additional QK-Norm. Unless noted, all models have 24 transformer layers, the same as GS-LRM. The encoder-decoder LVSM has 12 encoder layers and 12 decoder layers, with 3072 latent tokens. Note that our model size is smaller than GS-LRM, as GS-LRM uses a token dimension of 1024.
For object-level experiments, we use 4 input views and 8 target views for each training example by default. We first train with a resolution 256, which takes 4 days for the encoder-decods model and days for the decoder-only model. Then, we finetune the model with resolution of 512 for 10k iterations with a smaller learning rate of 4e-5 and a smaller total batch size of 128, which takes 2.5 days. For scene-level experiments We use 2 input views and 6 target views for each training example. We first train with resolution of 256, which takes about days for both encoder-decode and decoder-only models. Then, we finetune the model with resolution of 512 for 20k iterations with a smaller learning rate of le-4 and total batch size of 128 for 3 days. For both object and scene-level experiments, the view selection details and camera pose normalization methods follow GS-LRM. We use a perceptual loss weight À as 0.5 and 1.0 on scene-level and object-level experiments, respectively.
We do not use bias terms our model, for both Linear and LayerNorm layers. We initialize the model weights with a normal distribution of zero-mean and standard deviation of 0.02/ (idx+ 1)) +0.5. where idx means transform layer index.
We train our model with AdamW optimizer (Kingma 2014). The (1 and 32 are set to 0.9 and 0.95 respectively, following GS-LRM. We use a weight decay of 0.05 on all parameters except the weights of LayerNorm layers.
A.3 ADDITIONAL. VISUAL. RESULTS
We show the visualization of LVSM the object level with 256 resolution in Fig. 7 Consistent with the findings of the experiment with 512 resolution (Fig. 3 LVSM performs better than the baselines on texture details, specular material, and concave geometry.
A.4 DETAILED MODEL ARCHITECTURE
We have provided a detailed model architecture figure, as shown in Fig. g
A.5
ATTENTION MASK ILLUSTRATION FOR DIFFERENT DESIGN CHOICES
In Fig. Sec. 4.4
we visualize the corresponding attention masks for the various design choices discussed in
18

---

Published as a conference paper at ICLR 2025
I
Inpul imags
GS-LRM
DursDeoodeeOallyy
Figure 7: Object-level visual comparison at 256 resolution. Comparing with the two baselines: (Tag et al. 2024) and GS-LRM (Res-256) (Zhang al., 2024), both our Secoder--ecodee and Decoder-Only models have fewer floater artifacts (last example), and can generate more accurate view-dependent effects (third example). Our Decoder-Only model can better preserve the texture details (first two examples).
Decoder-Cnly
Decoder
Self-Att Transformer Details
Tagae Vie
Upeitnd Torget F
Al Todles
Aquand
Verianll Terget Ne
vS2 (Daoscie))
Takms
Qued
Dess dei for 2010
Concalontin
Uplaad
dt
Tekew
Upditdd Unowerr Takan
Text
Taget Vinu B.4)
Quay
(Enoide))
Lourable
Takaw
Vie fikee Rooe
Phacer Keys
Figure 8: Model Details. We introduce two architectures: (1) an encoder-decoder LVSM, which encodes input image tokens into a fixed number of ID latent tokens, functioning as a fully-learned latent scene representation, and decodes novel-view images from them; and (2) decoder-only LVSM, which directly maps input images to novel view outputs, completely eliminating intermediate scene representations. Both models consist of pure self-attention blocks.
19

---

Published as a conference paper at ICLR 2025
Denotes allowed
K[Latend(rpuu)
K(oulputt
K[Latent(rpuut
K(julput)
K(Laten))
K(output)
K[Latent)
K(output)
Ojoupuli (itttendpppu)
Ojoutpdt Of SLLatnnbinpud
Q[Latenn)
(jniitpt Q(Lator))
Qlouppu
Ours (bidiractional)
Ours w/ per-patch prediction
Ours w/ pure cross attention
Ours w/o latents updating
Figure 9: Attention Mask Visualization. Both our encoder-decoder and decoder-only models employ bidirectional self-attention modules. This figure visualizes the corresponding attention masks for the various design choices discussed in Sec. [44]] (We use green color for the columns of latent/input tokens and blue for the columns of output pixel patch tokens.) In our encoder-decoder architecture, the decoder utilizes pure self-attention, enabling latent tokens and different output target image tokens to jointly attend to each other. Consequently, latent tokens can be updated across transformer layers, while different output patch pixels can also attend to each other for joint updates. As shown in Table 3 disabling either the joint updating of output patch pixels (ours w/ per-patch prediction) or the latents updating (ours w/o latents' updating) significantly degrades performance. Prior work, SRT (Sajjadi et al., 2022), eliminates both mechanisms by employing a decoder with pure cross-attention (ours w/ pure cross-att decoder), leading to even worse performance. Similarly, in our decoder-only model, disabling the joint updating of output patch pixels (ours w/ per-patch prediction) also results in notable performance drop.
A.6 DISCUSSION OF DIFFERENCES WITH PRIOR GENERATIVE NVS MODELS
Motivated by the success of the previous NVS geometry-free approaches (Sitzmann et 2021
Sajjadi et al. 2022), and the effectiveness of diffusion models in image-to-image tasks Saharia
al. 2022a Ramesh et al. 2022 Saharia et al. 2022b). 3DiM (Watson et al. 2022] explores training image-to-imade
diffusion models for leject-level multi multi-view rendering to perform novel view synthesis without an explicit 3D representation. However, 3DiMs is trained from scratch using
limited 3D data (Sitzmann et al. 2019), limiting it to category-specine settings and without zero-shot generalization.
The following work Zero-1-to-3 (Liu et al. 2023a) adopts similar pipeline without 3D repre- sentation but fine-tunes its model from a pretrained 2D diffusion model using a larger 3D object dataset (Deitke et al. 2023), achieving better generalization and higher quality. However, Zero- 1-to- 3's view synthesis results suffer from inherent multi-view inconsistency because is a probabilistic model and it generates one target image at time independently.
To improve this inconsistency problem. several works (Liu et al. 2023b Yang et al. 2023 Ye et al.
2023 Tung et al., 2024) integrate additional forms of 3D inductive bias, such as 3D representation, epipolar attention, etc., into the diffusion denoising process, leading to increased computational cost. Other approaches (Li al. 2023 Shi et al. [203a b Long et al. (023) predict single image grid representing (specific) multi-view images with fixed camera pose, sacrificing the ability to control the camera. More recent works, including Free3D (Zheng Vedaldi 2024), EscherNet (Kong et al. 2024), CAT3D (Gao* et al. 2024), SV3D (Voleti et al. 2025), and some other video model based work (Wang et al., 2023b He et al. 2024 Yu et al.) 2024a] Yan et al.) 2024), jointly predict multiple target views with accurate camera control while ensuring view consistency by integrating cross-view attention. However, these methods guarantee consistency only for the finite set of jointly predicted views.
In contrast, our generalizable deterministic models do not possess the same inherent inconsistency issues of probabilistic models. As demonstrated in the video results our project webpage, after being trained on large-scale multi-view data, our models can independently generate each target image with precise camera control while maintaining view consistency wwitout relying the -iew attention mechanisms employed by previous generative models. This capability enables our models to generate unlimited number of consistent views for the observed regions of reconstructed scenes,
20

---

Published as a conference paper at ICLR 2025
unlike prior generative models. Nonetheless, deterministic models have their inherent limitations, i.e., they can't 't hallucinate unseen regions, which are discussed in Appendix A.7
A.7 LIMITATIONS
Our models deterministic, and like all prior deterministic approaches (Chen et al. 2021 Wang et alal, 200 (Sajaadi et al.) 2022; Wang et al. 2023a; Zhang et al. (2024), they struggle to produce high-quality results in unseen regions. Previous 3D- deterministic models typically generate blurry artifacts for those regions due uncertainty, whereas our model often generates noisy and flickering artifacts with fixed patterns. To illustrate this, we provide video examples of related failure cases webpage. Incorporating generative techniques combining generative methods with our model could help solve this issue, which we leave as promising future direction.
Additionally, model's performance degrades when provided with images with aspect ratios and resolutions different from those seen during training. For instance, when trained on 512 x 512 images and tested on 512 x 960 input images, observe high-quality novel view synthesis the center of the output but blurred regions at the horizontal boundaries that extend beyond the training aspect ratio. We hypothesize that this limitation arises because model is trained center-cropped images. Specifically, the Plücker ray density is smaller at the boundaries the image's longer side, and since our model is not trained on such data, it struggles generalize. Expanding the training dataset to include more diverse image resolutions and aspect ratios could help address this issue.
21


Reviews:
[{"review_id": "ZV6e5aTHYJ", "reviewer": "Reviewer_h9gs", "summary": "This work presented the Large View Synthesis Model (LVSM), which aims to achieve novel view synthesis (NVS) via a pure transformer architecture, bypassing the need for additional 3D inductive bias. In particular, LVSM explored an encoder-decoder model design and a decoder-only one, where the former is more efficient in inference with a more compact learned latent space, and the latter is more effective and scalable regarding visual quality. Experiments of object-level datasets and scene-level datasets demonstrate the superiority of the introduced LVSM.", "questions": "Kindly refer to [Weaknesses]", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "* The idea of achieving high-quality photorealistic NVS with minimal 3D inductive bias is brave. It is also impressive that LVSM implements this brave idea with a straightforward yet effective pure Transformer-based architecture.\n* Experiments on several benchmarks demonstrate the effectiveness of the introduced LVSM\n* The paper is well structured, and it is easy to follow.", "weaknesses": "* More discussion with Scene Representation Transformer (SRT) [Sajjadi et. al, CVPR 22]. LVSM seems to be a ‘reimplementation’ of SRT with more recent modules, which significantly limits the novelty of LVSM. The discussions in L141-L146 cannot convince me about the key contribution of LVSM. A more thorough analysis is suggested below.\n  * The introduction should clearly reveal the similarities and differences between SRT and LVSM. The motivation (minimal 3D inductive bias) and architecture (encoder-decoder) of LVSM are similar to SRT, and it seems that the key difference is that LVSM adopts more advanced modules from LRM.\n  * Different observations of ‘decoder-only’ architecture between LVSM and SRT.  SRT also explores the ‘decoder-only’ designs (see Sec. 4.3 “No Encoder Transformer” in SRT), which shows that ‘decoder-only’ performs worse than its ‘encoder-decoder’ counterparts. This observation happens to be contradicted to that of LVSM. It would be interesting to provide further analysis about what leads to this different conclusion despite similar settings.\n  * More analysis is needed to understand why LVSM achieves good quality on NVS.  It is hard to understand why LVSM gets much cleaner images while SRT gets only blurry ones. Is it because LVSM is trained with more data? Besides, it would be beneficial to visualise the attention of LVSM (similar to Fig. 5 in SRT).\n  * Noisy poses or unknown poses settings on LVSM. SRT achieves reasonably good results when the camera poses are noisy or even unknown (Fig. 4 in SRT). How does LVSM perform under similar noisy pose or pose-free settings? \n\n\n* Assessing the geometrical accuracy. For the object-level data, it would be better to reconstruct the 3D mesh using the rendered novel views, similar to Fig. 5 in latentSplat [Wewer et al., ECCV 24]. For the scene-level data, e.g., the single-view case in Fig. 1, it would be better to show the error map between the rendered and ground truth views (similar to Fig.6 in MVSplat [Chen et al. ECCV 24]), which makes it easier to understand how well the rendered views align with the given camera poses.\n\n\n* Performance on more complex datasets. L527 claims that LVSM is not simply doing view interpolation. Since the results are mainly shown on simple data, e.g., RealEstate10K with zoom-in / zoom-out trajectories, it cannot justify this claim. It would be better to show some results on more complex datasets, e.g., MipNeRF360, Tanks and Temples.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xORFFxcBbK", "reviewer": "Reviewer_jDDo", "summary": "This paper introduces two view synthesis models: one with an encoder-decoder architecture and another with a decoder-only design. Both models treat view synthesis as a predictive sequence-to-sequence task, using Plucker embeddings for camera position encoding. Without relying on intermediate 3D structures, the models achieve impressive results across object-level and scene-level view synthesis.", "questions": "See limitations.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "The paper is well-motivated and and very well-written, though certain technical details could benefit from additional clarity (outlined below). The visual results are striking, as shown on the authors’ website, and I appreciate the authors provide additional results with limited GPU-hours, making reproduction more feasible for academic labs. Overall, this work is a valuable contribution to view synthesis research.", "weaknesses": "1. Related Works. While the paper covers key prior work on 3D representation and few-shot view synthesis, it would benefit from a discussion of generative multi-view methods, especially recent works like Free3D (CVPR 2024, also uses Plucker embedding to encode camera poses) and EscherNet (CVPR 2024, also can be inferenced with varying number of reference/target views). These methods also do not rely on intermediate 3D representations, treating view synthesis as a sequence-to-sequence problem. Adding some proper discussions on how the proposed methods differ from these could provide valuable context for readers, and strength the community focussing on sequence-to-sequence learning type view synthesis.\n\n2. Architecture Design Details.\n- Encoder-Decoder Architecture: Could the authors clarify the choice of compressing input tokens into a fixed-length representation with latent tokens? What are the benefits over using uncompressed reference tokens (only with linear complexity)?\n- Decoder-Only Architecture: Are attention masks fully bi-directional? It would be interesting to see if introducing asymmetric masking strategies (e.g., limiting specific views to certain tokens) could enhance generalization to different numbers of reference views. Additionally, if attention is fully bi-directional, the distinction between reference and target views seems blurred, as the loss can be computed from all views, (conditioned on all other views)?\n\n3. Experiments\n\n- Unified Model Training: Could the model be trained jointly on both object- and scene-level data instead of using separate models? Demonstrating this capability would advance the model’s utility toward a more general-purpose view synthesis framework.\n- Varying Reference Images: The authors suggest the model performs well with varying numbers of reference images. To strengthen this claim, I recommend evaluating on a NeRF-Synthetic dataset instead of GSO in Fig. 5 (similarly as shown in EscherNet), which includes more complex objects in terms of textures and lighting. This would also enable clearer comparisons to other state-of-the-art, scene-specific methods like InstantNGP and 3D Gaussian Splatting that leverage 3D representations.\n- Single-Image View Synthesis: Single-image view synthesis is a common use case, so it would be valuable to include a comparison to methods specifically designed for single-image scenarios to showcase the model’s adaptability and strong generalization.\n- Plucker Embedding Generalization: Could the authors explore how well the Plucker embeddings generalize to different spatial coordinates? For instance, in scene-level experiments, would generation quality remain consistent if reference/target camera poses are applied with the same camera transformation, such as a 30-degree elevation or a 1-meter translation?\n\n4. Limitations\nA dedicated limitations section would help readers identify areas for improvement. Potential limitations include:\n\n- Predictive Modeling: The predictive approach in LVSM may restrict outputs to interpolation, limiting extrapolation capabilities, especially in scene-level tasks (as shown in the website). Maybe a generative model variant trained on larger datasets address this? Can the authors provide some extrapolation results?\n- Extreme Reference View Limits: How well does LVSM handle extremes in reference view numbers (e.g., only one view or over 100 views)? This could be an insightful addition, especially if single-view predictions are inconsistent or if performance degrades with many reference views.\n\nI am happy to further raise the score if the authors can adequately address my concerns.\n\n===================\n\nUpdated score based on the authors' rebuttal.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6PsUCZS04Q", "reviewer": "Reviewer_X42b", "summary": "This paper studies the task of synthesizing novel views from a set of views without explicit 3D representation.\nThe key idea is to use a transformer architecture to bypass the 3D inductive biases.\nTo achieve this goal, an encoder-decoder and a decoder-only architectures are proposed to conditional on image and camera pose for novel view synthesis. \nExperiments are trained on object-level Objaverse dataset, and scene-level RealEstate10K dataset, and tested on object-level Google Scanned Objects and Amazon Berkeley Objects and scene-level RealEstate10K dataset.\nThey demonstrate reasonable results on novel view synthesis, outperforming the existing state-of-the-art approaches.", "questions": "- Why this simple architecture performs so good compared to SRT? What's the key contribution, only transformer vs. CNN? \n- What's the performance on other scene-level datasets?\n- How about the 3D consistency?\n- The decoder-only transformer also be used for CAT3D, while they are in latent space for diffusion. If we train the similar architecture, but on diffusion-based architecture, what's the performance? In particular, could we use it to build a large foundational 3D model upon the pre-trained 2D diffusion models? The model not only works well on a special trained dataset, but can be generalised well to arbitrary images.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "### S1 -- Good results on an interesting task\n- The task of synthesizing novel views from a set of input views is interesting and very challenging. The proposed method seems to work well on both object-level dataset and scene-level dataset.\n- Base on the visual results shown in Figure 3 and 4, the proposed deterministic pipeline also can imagine the new content which is invisible from the input views.\n\n### S2 -- Simple ideas and careful implementations\n- There are two main transformer-based architectures. \n    - One is an encoder-decoder architecture, which used the transformer encoder to encode all input views into latent space, and then use a query camera pose to get the target view images. This architecture is very similar to SRT, except here the encoder is transformer architecture.\n    - The second idea is to design a decoder-only architecture. This reduces the necessary of the one representation from the encoder.\n- These two ideas are carefully implemented and validated through ablation studies. In particular, the decoder-only architecture seems quite effective in achieving a set of input views, from 1-10.\n\n### S3 -- Good writing\n- The paper is very well written, with clear motivations, sufficient technical explanations and illustrative visualization.", "weaknesses": "### W1 --- Significant is not well demonstrated\n- The proposed idea is a very specific, minor change to SRT -- basically using a slightly different transformer encoder or decoder to replace the original CNN. Fundamentally, I am not fully convinced that it is even crucial to use only transformer architecture than the CNN-based feature extraction and then do the transform.\n- This small change seems to lead to a large improvement on both object-level and scene-level datasets. However, if we train the original SRT on the same dataset with the same computational GPUs, what's the performance? \n- A fair comparison to the highly related work (SRT) should be provided. The discussion in L140-145 is also not a very strong claim. \n- The decoder-only architecture is very interesting and useful, but some related work like 3DIM, Free3D, CAT3D also used it in the diffusion for a set of views with the Plücker Rays embedding as conditional. Why this architecture is better than these difussion-based methods, which also used the transformer in pixel or latent space?\n\n### W2 --- More results should have been expected\n- I expected more visual results on scene-level cross-domain datasets. Figure 4 shows only result on RelEstate10k, which is a relative easy dataset. How about the performance on the scene-level datasets, such as DL3DV, Mip-NeRF 360, or other traditional NeRF Datasets?\n- This deterministic model can also provide sharp results for invisible regions, which is quite interesting. However, the authors only highlight them in the object-level results, how about the scene-level results? \n- Does the scene-level model perform good for the extrapolation, instead of interpolation?\n- In 3DIM, Free3D, SV3D, CAT3D, they used some temporal attention to ensure the consistency of generated images. How do the authors active the 3D consistency in the design? Besides, the multi-view rendered results as in 3DIM and Free3D or reconstructed 3D as in SV3D and CAT3D will make the paper stronger to show 3D consistency and structure, while they want to bypass the 3D representation.\n- It will be helpful to provide some visual examples of the failure cases.\n- The ablations studies are only related to the different layers of the transformer, and the attention architecture. If the authors argue the large contribution of transformer compared to CNN, a fair ablation should be made to use SRT-related architecture under the same experimental settings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XLy3ueJbBL", "reviewer": "Reviewer_NhwW", "summary": "The authors propose to train a generalizable novel view synthesis model with minimal 3D inductive biases, training on posed multi-view data to map images and camera rays to novel views, with target rays as queries.  This is supervised by a simple photometric and perceptual novel view loss.  \nThe architecture uses ViT-like image tokenization to embed images and corresponding Plucker rays, and the target views are decoded by Plucker ray queries.  Two Transformer architectures are considered – decoder-only, and encoder-decoder.\nThese are compared to prior work, and differences in performance between the two choices are compared and contrasted.", "questions": "Suggestions - clarifying the scaling (model size, compute) discussion (with model parameter counts), and comparison to prior work.\nAlso if possible it would be good to have the encoder-decoder vs decoder-only comparisons for both scene- and object-level (i.e. Table 2).", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The model variants introduced compare well to prior work; the large improvements compared to baselines for the decoder-only variant are impressive.  The investigation of both encoder-decoder and decoder-only as modeling choices (with the discussed tradeoffs) is also interesting and seems novel.  The discussion of the effects of compute (model size, compute available) on performance, while disorganized, is quite interesting.", "weaknesses": "There is limited novelty in the approach of removing handcrafted 3D representations – as the authors point out this was done in SRT [1], with a less effective and scalable architecture, as well as in the stereo case [2], which proposes a similar philosophy of input-level inductive biases, and in the multi-view case [3], where the view synthesis branch closely resembles LVSM.  Clarification of the novel elements of this architecture compared to prior work would be great, beyond the discussion of encoder-decoder vs decoder-only.\n\nThe presentation is somewhat messy.  The discussion of encoder vs encoder-decover in Section 3.2 takes up a lot of room, but much of it is standard Transformer information; this could be condensed to describe specifically how these concepts relate to the architecture rather than be a general introduction.  Also, there is information about model scale and compute scattered throughout the document, and the model sizes are not in terms of #params but layers.\n\n[1] Sajjadi, Mehdi SM, Henning Meyer, Etienne Pot, Urs Bergmann, Klaus Greff, Noha Radwan, Suhani Vora et al. \"Scene representation transformer: Geometry-free novel view synthesis through set-latent scene representations.\" CVPR 2022\n\n[2] Yifan, Wang, Carl Doersch, Relja Arandjelović, Joao Carreira, and Andrew Zisserman. \"Input-level inductive biases for 3d reconstruction.\" CVPR 2022\n\n[3] Guizilini, Vitor, Igor Vasiljevic, Jiading Fang, Rares Ambrus, Greg Shakhnarovich, Matthew R. Walter, and Adrien Gaidon. \"Depth field networks for generalizable multi-view scene representation.\" ECCV 2022", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "A70mS4ONWl", "reviewer": "Reviewer_Lz3H", "summary": "This paper proposes a novel transformer-based approach for scalable and generalizable novel view synthesis from sparse-view inputs. Two models are proposed. both of which bypass the 3D inductive biases used in previous methods and address novel view synthesis with a fully data-driven approach. Comprehensive evaluations across multiple datasets demonstrate that the work achieve state-of-the-art novel view synthesis quality.", "questions": "The architecture of decoder model is elaborated from L303-308, the attention layer in the transformer block seems to be self-attention. During both training and inference stage, I wonder how many input and target views are used for each batch sample, will the self-attention between target views affect the result? If I want to get the target view at P1 and P2, will inferencing them at the same feedforward pass be different from inferencing one by one?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. this paper minimizes 3D inductive biases for scalable and generalizable novel view synthesis;\n2. Two LVSM architectures—encoder-decoder and decoder-only—are designed to minimize 3D inductive biases;\n3. Both of the two models achieve impressive sparse novel view synthesis performance. \n4. The impressive result might provide a potential new representation for generalizable NVS task.", "weaknesses": "The architecture level is not well explained and might cause a bit confusing, I will elaborate it in questions part.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oHdZfPxZMZ", "reviewer": "Reviewer_jAgM", "summary": "This paper proposes LVSM, a general novel view synthesis (NVS) model for sparse-view inputs. It aims to improve the quality, efficiency, and scalability of NVS by minimizing 3D inductive bias. Different from previous works, LVSM directly utilizes transformer-based backbone to synthesize novel views without intermediate 3D representations and corresponding rendering equations. The author(s) introduce two versions of LVSM. The first model, encoder-decoder LVSM, encodes inputs into a scene representation token and decodes it into novel views. The second model, decoder-only LVSM, further removes the need for intermediate representation by adopting a single-stream transformer. Both models patchify the posed input view and use Plucker ray embeddings to tokenize the target view. Experiments show that LVSM trained on 2-4 input views has generalizability to an unseen number of views, and it outperforms the SOTA GS-LRM by 1.5 to 3.5dB PSNR.", "questions": "1.\tIt is commonly believed that 3D reconstruction improves view consistency for novel view synthesis. By removing 3D inductive bias, how does LVSM ensure consistency across novel views? Is this knowledge simply learned using attention mechanism? To demonstrate view consistency, you may want to provide some 3D-aware metrices for comparison, such as reprojection error or depth estimation, which will help my evaluation a lot.\n2.\tCould you please provide a more detailed diagram for the LVSM architecture? Specifically, such a diagram will help readers understand the methodology more easily. In the submission, only Figure 2 is available with very brief design.\n3.\tRefer to weakness.\n4. Will you release your codes and pretrained models after possible acceptance?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1.\tThe proposed models achieve impressive reconstruction results in terms of PSNR, SSIM, LPIPS, and qualitative evaluation.\n2.\tThe proposed models can be trained with a single A100 80G GPU. In contrast, existing general reconstruction models, like LRM, LGM, and GS-LRM, usually require a large number of computational resources.\n3.\tThe proposed model is generalizable to an unseen number of input views, from single view to more than 10 views.\n4.\tAblation studies show the scalability of LVSM, where transformers with more layers generally perform better in terms of the quantitative reconstruction results.", "weaknesses": "1.\tThe qualitative results are not consistent with the quantitative results. In specific, Table 1 shows that the encoder-decoder LVSM achieves similar quantitative results as GS-LRM, and sometimes worse in object-level datasets. However, Figure 3 and Figure 7 substantially outperform GS-LRM. I am worried about potential cherry picks in the qualitative results. It will help evaluation by showing more examples as well as some failure cases of LVSM.\n2.\tMissing experiments for efficiency comparison. Specifically, the paper claims the higher efficiency of LVSM than previous methods, while there is no comparison between LVSM and other NVS models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)