LVSM: A Large View Synthesis Model with Minimal 3D Inductive Bias

## Abstract

Abstract We propose the Large View Synthesis Model (LVSM), a novel transformer-based approach for scalable and generalizable novel view synthesis from sparse-view inputs. We introduce two architectures: (1) an encoder-decoder LVSM, which encodes input image tokens into a fixed number of 1D latent tokens, functioning as a fully learned scene representation, and decodes novel-view images from them; and (2) a decoder-only LVSM, which directly maps input images to novel-view outputs, completely eliminating intermediate scene representations. Both models bypass the 3D inductive biases used in previous methods—from 3D representations (e.g., NeRF, 3DGS) to network designs (e.g., epipolar projections, plane sweeps)—addressing novel view synthesis with a fully data-driven approach.
While the encoder-decoder model offers faster inference due to its independent latent representation, the decoder-only LVSM achieves superior quality, scalability, and zero-shot generalization, outperforming previous state-of-the-art methods by 1.5 to 3.5 dB PSNR. Comprehensive evaluations across multiple datasets demonstrate that both LVSM variants achieve state-of-the-art novel view synthesis quality. Notably, our models surpass all previous methods even with reduced computational resources (1-2 GPUs). Please see our website for more details: https://haian-jin.github.io/projects/LVSM/ .

## 1 Introduction

Novel view synthesis is a long-standing challenge in vision and graphics.
For decades, the community has generally relied on various 3D inductive biases, incorporating 3D priors and handcrafted structures to simplify the task and improve synthesis quality.
Recently, NeRF, 3D Gaussian Splatting (3DGS), and their variants (Mildenhall et al., ; Barron et al., ; Müller et al., ; Chen et al., ; Xu et al., ; Kerbl et al., ; Yu et al., ) have significantly advanced the field by introducing new inductive biases through carefully designed 3D representations (e.g., continuous volumetric fields and Gaussian primitives) and rendering equations (e.g., ray marching and splatting with alpha blending), reframing view synthesis as the optimization of the representations using rendering losses on a per-scene basis.
Other methods have also built generalizable networks to estimate these representations or directly generate novel-view images in a feed-forward manner, often incorporating additional 3D inductive biases, such as projective epipolar lines or plane-sweep volumes, in their architecture designs (Wang et al., ; Yu et al., ; Chen et al., ; Suhail et al., ; Charatan et al., ; Chen et al., ).

While effective, these 3D inductive biases inherently limit model flexibility, constraining their adaptability to more diverse and complex scenarios that do not align with predefined priors or handcrafted structures. Recent large reconstruction models (LRMs) (Hong et al., ; Li et al., ; Wei et al., ; Zhang et al., ) have made notable progress in removing architecture-level biases by leveraging large transformers without relying on epipolar projections or plane-sweep volumes, achieving state-of-the-art novel view synthesis quality. However, despite these advances, LRMs still rely on representation-level biases—such as NeRFs, meshes, or 3DGS, along with their respective rendering equations—that limit their potential generalization and scalability.

Figure: Figure 1: LVSM supports feed-forward novel view synthesis from sparse posed image inputs (even from a single view) on both objects and scenes. LVSM achieves significant quality improvements compared with the previous SOTA method, i.e., GS-LRM (Zhang et al., ). (Please zoom in for more details.)
Refer to caption: x1.png

In this work, we aim to *minimize 3D inductive biases* and push the boundaries of novel view synthesis with a fully data-driven approach. We propose the Large View Synthesis Model (LVSM), a novel transformer-based framework that synthesizes novel-view images from posed sparse-view inputs *without
predefined rendering equations or 3D structures, enabling accurate, efficient, and scalable novel view synthesis with photo-realistic quality* (see Fig. 1 for visual examples).

To this end, we first introduce an encoder-decoder LVSM, removing handcrafted 3D representations and their rendering equations.
We use an encoder transformer to map the input (patchified) multi-view image tokens into a fixed number of 1D latent tokens, independent of the number of input views. These latent tokens are then processed by a decoder transformer, which uses target-view Plücker rays as positional embeddings to generate the target view’s image tokens, ultimately regressing the output pixel colors from a final linear layer. The encoder-decoder LVSM jointly learns a reconstructor (encoder), a scene representation (latent tokens), and a renderer (decoder) directly from data. By removing the need for predefined inductive biases in rendering and representation, LVSM offers improved generalization and achieves higher quality compared to NeRF- and GS-based approaches.

However, the encoder-decoder LVSM still retains a key bias: the need for an intermediate, albeit fully learned, scene representation. To further push the boundaries, we propose a decoder-only LVSM, which adopts a single-stream transformer to directly convert the input multi-view tokens into target view tokens, bypassing any intermediate representations. The decoder-only LVSM integrates the novel view synthesis process into a holistic data-driven framework, achieving scene reconstruction and rendering simultaneously in a fully implicit manner with minimal 3D inductive bias.

We present a comprehensive evaluation of variants of both LVSM architectures.
Notably, our models, trained on 2-4 input views, demonstrate strong zero-shot generalization to an unseen number of views, ranging from a single input to more than 10. Thanks to minimal inductive biases, our decoder-only model consistently outperforms the encoder-decoder variant in terms of quality, scalability, and zero-shot capability with varying numbers of input views.
On the other hand, the encoder-decoder model achieves much faster inference speed due to its use of a fixed-length latent scene representation. Both models, benefiting from reduced 3D inductive biases, outperform previous methods, achieving state-of-the-art novel view synthesis quality across multiple object-level and scene-level benchmark datasets. Specifically, our decoder-only LVSM surpasses previous state-of-the-art methods, such as GS-LRM, by a substantial margin of 1.5 to 3.5 dB PSNR. Our final models were trained on 64 A100 GPUs for 3-7 days, depending on the data type and model architecture, but we found that even with just 1–2 A100 GPUs for training, our model (with a decreased model and batch size) still outperforms all previous methods trained with equal or even more compute resources.

## 2 Related Work

View Synthesis.
Novel view synthesis (NVS) has been studied for decades.
Image-based rendering (IBR) methods perform view synthesis by weighted blending of input reference images using proxy geometry (Debevec et al., ; Heigl et al., ; Sinha et al., ). Light field methods build a slice of the 4D plenoptic function from dense view inputs (Gortler et al., ; Levoy & Hanrahan, ; Davis et al., ). Recent learning-based IBR methods incorporate convolutional networks to predict blending weights (Hedman et al., ; Zhou et al., ; ) or using predicted depth maps (Choi et al., ). However, the renderable region is usually constrained to be near the input viewpoints. Other work leverages multiview-stereo reconstructions to enable rendering under larger viewpoint changes (Jancosek & Pajdla, ; Chaurasia et al., ; Penner & Zhang, ).
In contrast, we use more scalable network designs to learn generalizable priors from larger, real-world data. Moreover, we perform rendering at the image patch level, achieving better model efficiency, and rendering quality.

Optimizing 3D Representations.
NeRF (Mildenhall et al., ) introduced a neural volumetric 3D representation with differentiable volume rendering, enabling neural scene reconstruction by minimizing rendering losses and setting a new standard in novel view synthesis.
Later work improved NeRF with better rendering quality (Barron et al., ; Verbin et al., ; Barron et al., ), faster optimization or rendering speed (Reiser et al., ; Hedman et al., ; Reiser et al., ), and looser requirements on the input views (Niemeyer et al., ; Martin-Brualla et al., ; Wang et al., ).
Other work has explored hybrid representations that combine implicit NeRF content with explicit 3D information, e.g., in the form of voxels, as in
DVGO (Sun et al., ).
Spatial complexity can be further decreased by using sparse voxels (Liu et al., ; Fridovich-Keil et al., ), volume decomposition (Chan et al., ; Chen et al., ; ), and hashing techniques (Müller et al., ).
Another line of works investigates explicit point-based representations (Xu et al., ; Zhang et al., ; Feng et al., ).
Gaussian Splatting (Kerbl et al., ) extends these 3D points to 3D Gaussians, improving both rendering quality and speed. In contrast, we perform novel view synthesis using large transformer models (optionally with a learned latent scene representation) without the need of any inductive bias of using prior 3D representations or any per-scene optimization process.

Generalizable Feed-forward Methods.
Generalizable methods enable fast NVS inference by using neural networks, trained across scenes, to predict the novel views or an underlying 3D representation in a feed-forward manner.
For example, PixelNeRF (Yu et al., ), MVSNeRF (Chen et al., ) and IBRNet (Wang et al., ) predict volumetric 3D representations from input views, utilizing 3D-specific priors like epipolar lines or plane sweep cost volumes.
Later methods improve performance under (unposed) sparse views (Liu et al., ; Johari et al., ; Jiang et al., ; ),
while other work extends to 3DGS representations Charatan et al. (); Chen et al. (); Tang et al. ().
On the other hand, approaches that attempt to directly learn a rendering function (Suhail et al., ; Sajjadi et al., ; Sitzmann et al., ; Rombach et al., ) have proven not to be scalable and lack model capacity, preventing them from capturing high-frequency details. Specifically, SRT (Sajjadi et al., ) intends to remove the use of handcrafted 3D representations and learn a latent representation instead, similar to our encoder-decoder model. However, it utilizes CNN feature extraction and adopts a cross-attention transformer with non-scalable model and rendering designs. In contrast, our models are fully transformer-based with self-attention, and we introduce a more scalable decoder-only architecture that can effectively learn the novel view synthesis function with minimal 3D inductive bias, without an intermediate latent representation.

Recently, 3D large reconstruction models (LRMs) have emerged (Hong et al., ; Li et al., ; Wang et al., ; Xu et al., ; Wei et al., ; Zhang et al., ; Xie et al., ), utilizing scalable transformer architectures (Vaswani et al., ) trained on large datasets to learn generic 3D priors. While these methods avoid using epipolar projection or cost volumes in their architectures, they still rely on existing 3D representations like tri-plane NeRFs, meshes, or 3DGS, along with their corresponding rendering equations, limiting their potential.
In contrast, our approach eliminates these 3D inductive biases, aiming to learn a rendering function (and optionally a scene representation) directly from data. This leads to more scalable models and significantly improved rendering quality.

## 3 Method

We first provide an overview of our method in Sec. 3.1, then describe two different transformer-based model variants in Sec. 3.2.

### 3.1 Overview

Given $N$ sparse input images with known camera poses and intrinsics, denoted as $\{(\mathbf{I}_{i},\mathbf{E}_{i},\mathbf{K}_{i})|i=1,\ldots,N\}$, LVSM synthesizes target image $\mathbf{I}^{t}$ with novel target camera extrinsics $\mathbf{E}^{t}$ and intrinsics $\mathbf{K}^{t}$. Each input image has shape $\mathbb{R}^{H\times W\times 3}$, where $H$ and $W$ are the image height and width (and there are $3$ color channels).

Figure: Figure 2: LVSM model architecture. LVSM first patchifies the posed input images into tokens. The target view to be synthesized is represented by its Plücker ray embeddings and is also tokenized. The input view and target tokens are sent to a full transformer-based model to predict the tokens that are used to regress the target view pixels. We study two LVSM transformer architectures, as a Decoder-only architecture (left) and a Encoder-Decoder architecture (right).
Refer to caption: x2.png

#### Framework.

As shown in Fig. 2, our LVSM method uses an end-to-end transformer model to directly render the target image.
LVSM starts by tokenizing the input images. We first compute a pixel-wise Plücker ray embedding (Plucker, ) for each input view using the camera poses and intrinsics.
We denote these Plücker ray embeddings as $\{\mathbf{P}_{i}\in\mathbb{R}^{H\times W\times 6}|i=1,\ldots,N\}$. We patchify the RGB images and Plücker ray embeddings into non-overlapping patches, following the image tokenization layer of ViT (Dosovitskiy, ). We denote the image and Plücker ray patches of input image $\mathbf{I}_{i}$ as $\{\mathbf{I}_{ij}\in\mathbb{R}^{p\times p\times 3}|j=1,\ldots,HW/p^{2}\}$ and $\{\mathbf{P}_{ij}\in\mathbb{R}^{p\times p\times 6}|j=1,\ldots,HW/p^{2}\}$, respectively, where $p$ is the patch size.
For each patch, we concatenate its image patch and Plücker ray embedding patch, reshape them into a 1D vector, and use a linear layer to map it into an input patch token
$\mathbf{x}_{ij}$:

$$ $\displaystyle\mathbf{x}_{ij}=\text{Linear}_{\mathit{input}}([\mathbf{I}_{ij}, \mathbf{P}_{ij}])\in\mathbb{R}^{d},$ (1) $$

where $d$ is the latent size, and $[\cdot,\cdot]$ means concatenation.

Similarly, LVSM represents the target pose to be synthesized as its Plücker ray embeddings $\mathbf{P}^{t}\in\mathbb{R}^{H\times W\times 6}$, computed from the given target extrinsics $\mathbf{E}^{t}$ and intrinsics $\mathbf{K}^{t}$. We use the same patchify method and another linear layer to map it to the Plücker ray tokens of the target view, denoted as:

$$ $\displaystyle\quad\mathbf{q}_{j}=\text{Linear}_{\mathit{target}}(\mathbf{P}^{t }_{j})\in\mathbb{R}^{d},$ (2) $$

where $\mathbf{P}^{t}_{j}$ is the Plücker ray embedding of the $j^{\text{th}}$ patch in the target view.

We flatten the input tokens into a 1D token sequence, denoted as $x_{1},\ldots,x_{\mathit{l}_{x}}$, where $\mathit{l}_{x}\!=\!NHW/p^{2}$ is the sequence length of the input image tokens. We also flatten the target query tokens as $q_{1},\ldots,q_{\mathit{l}_{q}}$ from the ray embeddings, with $\mathit{l}_{q}=HW/p^{2}$ as the sequence length.

LVSM then synthesizes novel view by conditioning on the input view tokens using a full transformer model $M$:

$$ $\displaystyle y_{1},\ldots,y_{\mathit{l_{q}}}=M(q_{1},\ldots,q_{\mathit{l_{q}} }|x_{1},\ldots,x_{\mathit{l_{x}}}).$ (3) $$

Specifically, the output token $y_{j}$ is an updated version of $q_{j}$, containing the information to predict the pixel RGB values of the $j^{\text{th}}$ patch of the target view. The implementation details of model $M$ are described in Sec. 3.2 next.

We recover the spatial structure of output tokens using the inverse operation of the flatten operation. To regress RGB values of the target patch, we employ a linear layer followed by a Sigmoid function:

$$ $\displaystyle\hat{\mathbf{I}}_{j}^{t}=\text{Sigmoid}{(\text{Linear}_{out}(y_{j }))}\in\mathbb{R}^{3p^{2}}.$ (4) $$

We reshape the predicted RGB values back to the 2D patch in $\mathbb{R}^{p\times p\times 3}$, and then we get the synthesized novel view $\hat{\mathbf{I}}^{t}$ by performing the same operation on all target patches independently.

#### Loss Function.

Following prior works (Zhang et al., ; Hong et al., ), we train LVSM with photometric novel view rendering losses:

$$ $\displaystyle\mathcal{L}=\text{MSE}(\hat{\mathbf{I}}^{t},\mathbf{I}^{t})+ \lambda\cdot\text{Perceptual}(\hat{\mathbf{I}}^{t},\mathbf{I}^{t}),$ (5) $$

where $\lambda$ is the weight for balancing the perceptual loss (Johnson et al., ).

### 3.2 Transformer-Based Model Architecture

In this subsection, we present the two LVSM architectures—encoder-decoder and decoder-only—both designed to minimize 3D inductive biases.
Following their name, ‘encoder-decoder’ first converts input images to a latent representation before decoding the final image colors, whereas ‘decoder-only’ directly outputs the synthesized target view without an intermediate representation, further minimizing inductive biases in its design.

To better illustrate the two models, we draw an analogy between them and their counterparts in the large language model (LLM) domain. In detail, the encoder-decoder model is inspired by the encoder-decoder framework in the original Transformer model (Vaswani et al., ) and Perceiver-IO (Jaegle et al., ) though with different model details, e.g. more decoder layers and removal of cross-attention layers. And the decoder-only model resembles a GPT-like decoder-only architecture but without adopting a causal attention mask (Radford et al., ).

Importantly, we clarify that the naming of ‘encoder’ and ‘decoder’ are based on their output characteristics—i.e., the encoder outputs the latent while the decoder outputs the target—rather than being strictly tied to the transformer architecture they utilize.
For instance, in the encoder-decoder model, the decoder consists of multiple transformer layers with self-attention (referred to as Transformer Encoder layers in the original transformer paper). However, we designate it as a decoder because its primary function is to output results.
These terminologies align with conventions used in LLMs (Vaswani et al., ; Radford et al., ; Devlin, ).
Notably, we apply self-attention to all tokens in every transformer block of both models, without introducing special attention masks or other architectural biases, in line with our philosophy of minimizing inductive bias.

#### Encoder-Decoder Architecture.

The encoder-decoder LVSM comes with a learned latent scene representation for view synthesis, avoiding the use of NeRF, 3DGS, and other representations.
The encoder first maps the input tokens to an intermediate 1D array of latent tokens (functioning as a latent scene representation). Then the decoder predicts the outputs, conditioning on the latent tokens and target pose.

Similar to the triplane tokens in LRMs (Hong et al., ; Wei et al., ), we use $l$ learnable latent tokens $\{e_{k}\in\mathbb{R}^{d}|k=1,...,l\}$ to aggragate information from input tokens $\{x_{i}\}$. The encoder, denoted as $\mathrm{Transformer}_{\mathit{Enc}}$, uses multiple transformer layers with self-attention. We concatenate $\{x_{i}\}$ and $\{e_{k}\}$ as the input of $\mathrm{Transformer}_{\mathit{Enc}}$, which performs information aggregation between them to update $\{e_{k}\}$.
The output tokens that correspond to the latent tokens, denoted as $\{z_{k}\}$, are used as the intermediate latent scene representation. The other tokens (updated from $\{x_{i}\}$, denoted as $\{x^{\prime}_{i}\}$) are unused and discarded.

The decoder uses multiple transformer layers with self-attention.
In detail, the inputs are the concatenation of the latent tokens $\{z_{k}\}$ and the target view query tokens $\{q_{j}\}$.
By applying self-attention transformer layers over the input tokens, we get output tokens with the same sequence length as the input.
The output tokens that corresponds to the target tokens $q_{1},\ldots,q_{\mathit{lq}}$ are treated as final outputs $y_{1},\ldots,y_{\mathit{lq}}$, and the other tokens (updated from $\{z_{i}\}$, denoted as $\{z^{\prime}_{i}\}$) are unused.
This architecture can be formulated as:

$$ $\displaystyle x^{\prime}_{1},\ldots,x^{\prime}_{\mathit{l_{x}}},z_{1},\ldots,z _{\mathit{l}}$ $\displaystyle=\mathrm{Transformer}_{\mathit{Enc}}(x_{1},\ldots,x_{\mathit{l_{x }}},e_{1},\ldots,e_{\mathit{l}})$ (6) $\displaystyle z^{\prime}_{1},\ldots,z^{\prime}_{l},y_{1},\ldots,y_{\mathit{l_{ q}}}$ $\displaystyle=\mathrm{Transformer}_{\mathit{Dec}}(z_{1},\ldots,z_{l},q_{1}, \ldots,q_{\mathit{l_{q}}}).$ (7) $$

#### Decoder-Only Architecture.

Our alternate, decoder-only model further eliminates the need for an intermediate scene representation. Its architecture is similar to the decoder in encoder-decoder architecture but differs in inputs and model size.
We concatenate the two sequences of input tokens $\{x_{i}\}$ and target tokens $\{q_{j}\}$.
The final output $\{y_{j}\}$ is the decoder’s corresponding output for the target tokens $\{q_{j}\}$. The other tokens (updated from $\{x_{i}\}$, denoted as $\{x^{\prime}_{i}\}$) are unused and discarded. This architecture can be formulated as:

$$ $\displaystyle x^{\prime}_{1},\ldots,x^{\prime}_{\mathit{l_{x}}},y_{1},\ldots,y _{\mathit{l_{q}}}=\mathrm{Transformer}_{\mathit{Dec\mbox{-}only}}(x_{1},\ldots ,x_{\mathit{l_{x}}},q_{1},\ldots,q_{\mathit{l_{q}}})$ (8) $$

Here the $\mathrm{Transformer}_{\mathit{Dec\mbox{-}only}}$ has multiple full self-attention transformer layers.

## 4 Experiments

### 4.1 Datasets

We train (and evaluate) LVSM on object-level and scene-level datasets separately.

Object-level Datasets. We use the Objaverse dataset (Deitke et al., ) to train LVSM. We follow the rendering settings in GS-LRM (Zhang et al., ) and render 32 random views for $730$K objects. We test on two object-level datasets, i.e., Google Scanned Objects (Downs et al., ) (GSO) and Amazon Berkeley Objects (Collins et al., ) (ABO). In detail, GSO and ABO contain $1099$ and $1000$ objects, respectively. Following Instant3D (Li et al., ) and GS-LRM (Zhang et al., ), we have 4 sparse views as testing inputs and another 10 views as target images.

Scene-level Datasets. We use the RealEstate10K dataset (Zhou et al., ), which contains $80$K video clips curated from $10$K Youtube videos of both indoor and outdoor scenes. We follow the train/test data split used in pixelSplat (Charatan et al., ).

### 4.2 Training Details

Improving Training Stability.
We observe that the training of LVSM crashes with plain transformer layers (Vaswani et al., ) due to exploding gradients.
We empirically find that using QK-Norm (Henry et al., ) in the transformer layers stabilizes training.
This observation is consistent with Bruce et al. () and Esser et al. ().
We also skip optimization steps with gradient norm > 5.0 besides the standard 1.0 gradient clipping.

Efficient Training Techniques. We use FlashAttention-v2 (Dao, ) in the xFormers (Lefaudeux et al., ), gradient checkpointing (Chen et al., ), and mixed-precision training with Bfloat16 data type to accelerate training.

Other Details.
We train LVSM with 64 A100 GPUs using a batch size of 8 per GPU. We use a cosine learning rate schedule with a peak learning rate of $4\text{e-}4$ and a warmup of 2500 iterations. We train LVSM for 80k iterations on the object and 100k on scene data. LVSM uses a image patch size of $p=8$ and token dimension $d=768$. The details of transformer layers follow GS-LRM (Zhang et al., ) with an additional QK-Norm.
Unless noted, all models have 24 transformer layers, the same as GS-LRM. The encoder-decoder LVSM has $12$ encoder layers and $12$ decoder layers. Note that our model size is smaller than GS-LRM, as GS-LRM uses a token dimension of $1024$.

For object-level experiments, we use 4 input views and 8 target views for each training example by default. We first train with a resolution of $256$, which takes 4 days for the encoder-decoder model and 7 days for the decoder-only model. Then we finetune the model with a resolution of $512$ for 10k iterations with a smaller learning rate of $4\text{e-}5$ and a smaller total batch size of 128, which takes 2.5 days. For scene-level experiments
We use 2 input views and 6 target views for each training example. We first train with a resolution of $256$, which takes about 3 days for both encoder-decoder and decoder-only models. Then we finetune the model with a resolution of $512$ for 20k iterations with a smaller learning rate of $1\text{e-}4$ and a total batch size of 128 for 3 days.
For both object and scene-level experiments, the view selection details and camera pose normalization methods follow GS-LRM. We use a perceptual loss weight $\lambda$ as 0.5 and 1.0 on scene-level and object-level experiments, respectively.

**Table 1: Quantitative comparisons on object-level (left) and scene-level (right) view synthesis. For the object-level comparison, we matched the baseline settings with GS-LRM (Zhang et al., ) in both input and rendering under both resolution of $256$ (Res-256) and resolution of $512$ (Res-512). For the scene-level comparison, we use the same validation dataset used by pixelSplat (Charatan et al., ), which has $256$ resolution.**
|  | ABO (Collins et al., ) | GSO (Downs et al., ) |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ |
| Triplane-LRM (Li et al., ) (Res-512) | 27.50 | 0.896 | 0.093 | 26.54 | 0.893 | 0.064 |
| GS-LRM (Zhang et al., ) (Res-512) | 29.09 | 0.925 | 0.085 | 30.52 | 0.952 | 0.050 |
| Ours Encoder-Decoder (Res-512) | 29.81 | 0.913 | 0.065 | 29.32 | 0.933 | 0.052 |
| Ours Decoder-Only (Res-512) | 32.10 | 0.938 | 0.045 | 32.36 | 0.962 | 0.028 |
| LGM (Tang et al., ) (Res-256) | 20.79 | 0.813 | 0.158 | 21.44 | 0.832 | 0.122 |
| GS-LRM (Zhang et al., ) (Res-256) | 28.98 | 0.926 | 0.074 | 29.59 | 0.944 | 0.051 |
| Ours Encoder-Decoder (Res-256) | 30.35 | 0.923 | 0.052 | 29.19 | 0.932 | 0.046 |
| Ours Decoder-Only (Res-256) | 32.47 | 0.944 | 0.037 | 31.71 | 0.957 | 0.027 |

### 4.3 Evaluation against Baselines

Figure: Figure 3: Object-level visual comparison at 512 resolution. Given 4 sparse input posed images (leftmost column), we compare our high-res object-level novel-view rendering results with two baselines: Instant3D’s Triplane-LRM (Li et al., ) and GS-LRM (Res-512) (Zhang et al., ) . Both our Encoder-Decoder and Decoder-Only models exhibit fewer floaters (first example) and fewer blurry artifacts (second example), compared to the baselines. Our Decoder-Only model effectively handles complex geometry, including small holes (third example) and thin structures (fourth example). Additionally, it preserves the details of high-frequency texture (last example).
Refer to caption: x3.png

In this section, we describe our experimental setup and datasets (Sec. 4.1), introduce our model training details (Sec. 4.2), report evaluation results (Sec. 4.3) and perform an ablation study (Sec. 4.4).

Object-Level Results. We compare with Instant3D’s Triplane-LRM (Li et al., ) and GS-LRM (Zhang et al., ) at a resolution of $512$. As shown on the left side of Table 1, our LVSM achieves the best performance against all prior works. In particular, at $512$ resolution, our decoder-only LVSM achieves a 3 dB and 2.8 dB PSNR gain against the best prior method GS-LRM on ABO and GSO, respectively; our encoder-decoder LVSM achieves performance comparable to GS-LRM.

We also compare with LGM (Tang et al., ) at the resolution of $256$, as the official LGM model is trained with an input resolution of $256$. We also report the performance of models trained on resolution of $256$. Compared with the best prior work GS-LRM, our decoder-only LVSM demonstrates a 3.5 dB and 2.2 dB PSNR gain on ABO and GSO, respectively; our encoder-decoder LVSM shows a slightly better performance than GS-LRM.

These significant performance gains validate the effectiveness of our design target of removing 3D inductive bias. More interestingly, the larger performance gain on ABO shows that our method can handle challenging materials, which are difficult for current handcrafted 3D representations. The qualitative results in Fig. 3 and Fig. 9 also validate the high degree of realism of LVSM, especially for examples with specular materials, detailed textures, and thin, complex geometry.

Figure: Figure 4: Scene-level visual comparison. We evaluate our encoder-decoder and decoder-only models on scene-level view synthesis, comparing them against the prior leading baseline methods, namely pixelSplat (Charatan et al., ), MVSplat (Chen et al., ), and GS-LRM (Zhang et al., ). Our methods exhibit fewer texture and geometric artifacts, generate more accurate and realistic specular reflections, and are closer to the ground truth images.
Refer to caption: x4.png

Scene-Level Results. We compare with prior works pixelNeRF (Yu et al., ), GPNR (Suhail et al., ),  Du et al. (), pixelSplat (Charatan et al., ), MVSplat (Chen et al., ) and GS-LRM (Zhang et al., ). As shown on the right side of Table 1, our decoder-only LVSM shows a 1.6 dB PSNR gain compared with the best prior work GS-LRM. Our encoder-decoder LVSM also demonstrates comparable results to GS-LRM. These improvements can be observed qualitatively in Fig. 4, where LVSM has fewer floaters and better performance on thin structures and specular materials, consistent with the object-level results. These outcomes again validate the efficacy of our design of using minimal 3D inductive bias.

LVSM Trained with Only 1 GPU.
Limited computing is a key bottleneck for academic research.
To show the potential of LVSM using academic-level resources, we train LVSM on the scene-level dataset (Zhou et al., ) following the setting of pixelSplat (Charatan et al., ) and MVSplat (Chen et al., ), with only a single A100 80G GPU for 7 days.
In this experiment, we use a smaller decoder-only model (denoted LVSM-small) with $6$ transformer layers and a smaller batch size of $64$ (in contrast to the default one with $24$ layers and batch size $512$).
Our decoder-only LVSM-small shows a performance of 27.66 dB PSNR, 0.870 SSIM, and 0.129 LPIPS. This performance surpasses the prior best 1-GPU-trained model MVSplat, with a 1.3 dB PSNR gain. We also train the decoder-only LVSM (12 transformer layers, batch size 64) with 2 GPUs for 7 days, exhibiting a performance of 28.56 dB PSNR, 0.889 SSIM, and 0.112 LPIPS. This performance is even better than GS-LRM with 24 transformer layers trained on 64 GPUs.
These results show the promising potential of LVSM for academic research.

Figure: Figure 5: Ablations studies on model sizes. The following experiments are run with 8 GPUs.

### 4.4 Ablation Studies

Model Size. As shown in Table 6, we ablate the model size designs of both LVSM variants.
For the encoder-decoder LVSM, we maintain the total number of transformer layers while allocating a different number of layers to the encoder and decoder. We observe that using more decoder layers helps the performance while using more encoder layers harms the performance. We conjecture the reason is that the encoder uses the latent representation as the compression of input images and this compression process is hard to learn from data using more encoder layers, leading to more compression errors. This is an interesting observation showing that using the inductive bias of the encoder and intermediate latent representation may not be optimal.

For the decoder-only LVSM, we experiment with using different numbers of transformer layers in the decoder.
The experiment verifies that decoder-only LVSM demonstrates an increasing performance when using more transformer layers.
This phenomenon validates the scalability of the decoder-only LVSM.
We also provide the decoder-only model scaling results on object-level reconstruction in Table 10 of Appendix, where the larger models perform relatively better.

Model Architecture. We evaluate the effectiveness of our design objective, which emphasizes minimal inductive bias at the model architecture level. To do this, we replace the self-attention transformer layers with variants that incorporate human-designed priors.

We first ablate the importance of updating latent representation in the decoder of the encoder-decoder LVSM. We design a variant, where the input for each transformer layer of the decoder are the target tokens and raw latent tokens predicted by the encoder, rather than the updated latent tokens outputted from the previous transformer layer. Thus, we use fixed latent tokens in the decoder without updating. We perform experiments at the object level. As shown on the top part of Table 6, this variant shows a degraded performance with 2 dB less PSNR. This result can be explained by that the transformer themselves is a part of the latent representation, as they are used to update the latent representation in the decoder.

We then ablate the importance of joint prediction of target image patches in the decoder-only LVSM. We design a variant where the colors of each target pose patch are predicted independently, without applying self-attention across other target pose patches. We achieve this by letting each transformer’s layer’s key and value matrices only consist of the updated input image tokens, while both the updated input image tokens and target pose tokens form the query matrices. As shown on the bottom part of Table 6, this variant shows a worse performance with a 0.7 dB PSNR degradation. This result demonstrates the importance of using both input and target tokens as the context of each other for information propagation using the simplest full self-attention transformer, in line with our philosophy of reducing inductive bias.

### 4.5 Discussions

Figure: Figure 7: Zero-shot generalization to different number of input images on the GSO dataset (Downs et al., ). We note that all models are trained with just 4 input views.
Refer to caption: extracted/5946542/figures/NVS_PSNR_under_different_input_view_number_gso.png

Zero-shot Generalization to More Input Views. We compare our LVSM with GS-LRM
by taking different numbers of input views to the training.
We report the results on the object level.
Note that these models are trained only with $4$ input views and test on other input view numbers in a zero-shot manner.
As shown in Fig. 8, our decoder-only LVSM shows increasingly better performance when using more input views, verifying the scalability of our model design at test time. Our encoder-decoder LVSM shows a similar performance pattern with GS-LRM, i.e., exhibiting a performance drop when using more than 8 input views. We conjecture the reason is the inductive bias of the encoder-decoder design, i.e. using intermediate representation as a compression of input information, limits the performance.
In addition, our single-input result (input view number = 1) is competitive and even beats some of our baseline which takes 4 images as input.
These performance patterns validate our design target of using minimal 3D inductive bias for learning a fully data-driven rendering model and cohere with our discussion in Sec. 3.2.

Encoder-Decoder versus Decoder-Only.
As we mentioned in Sec. 3, the decoder-only and encoder-decoder architectures exhibit different trade-offs in speed, quality, and potential.

The encoder-decoder model transforms 2D image inputs into a fixed-length set of 1D latent tokens, which serve as a compressed representation of the 3D scene. This approach simplifies the decoder, reducing its model size. Furthermore, during the rendering/decoding process, the decoder always receives a fixed number of tokens, regardless of the number of input images, ensuring a consistent rendering speed. As a result, this model offers improved rendering efficiency, as shown in Fig. 8. Additionally, the use of 1D latent tokens as the latent representation for the 3D scene opens up the possibility of integrating this model with generative approaches for 3D content generation on its 1D latent space.
Nonetheless, the compression process can result in information loss, as the fixed latent tokens length is usually smaller than the original image tokens length, which imposes an upper bound on performance. This characteristic of the encoder-decoder LVSM mirrors prior encoder-decoder LRMs, whereas our LVSM does not have an explicit 3D structure.

In contrast, the decoder-only model learns a direct mapping from the input image to the target novel view, showcasing better scalability. For example, as the number of input images increases, the model can leverage all available information, resulting in improved novel view synthesis quality. However, this property also leads to a linear increase in input image tokens, causing the computational cost to grow quadratically and limiting the rendering speed.

Single Input Image. As shown in our project page, Fig. 1 and Fig. 8, we observe that our LVSM also works with a single input view for many cases, even though the model is trained with multi-view images during training. This observation shows the capability of LVSM to understand the 3D world, e.g. understanding depth, rather than performing purely pixel-level view interpolation.

## 5 Conclusions

In this work, we presented the Large View Synthesis Model (LVSM), a transformer-based approach designed to minimize 3D inductive biases for scalable and generalizable novel view synthesis. Our two architectures—encoder-decoder and decoder-only—bypass physical-rendering-based 3D representations like NeRF and 3D Gaussian Splatting, allowing the model to learn priors directly from data, leading to more flexible and scalable novel view synthesis. The decoder-only LVSM, with its minimal inductive biases, excels in scalability, zero-shot generalization, and rendering quality, while the encoder-decoder LVSM achieves faster inference due to its fully learned latent scene representation. Both models demonstrate superior performance across diverse benchmarks and mark an important step towards general and scalable novel view synthesis in complex, real-world scenarios.

#### Acknowledgements.

We thank Kalyan Sunkavalli for helpful discussions and support. This work was done when Haian Jin, Hanwen Jiang, and Tianyuan Zhang were research interns at Adobe Research. This work was also partly funded by the National Science Foundation (IIS-2211259, IIS-2212084).

## Appendix A Appendix

We include additional results, ablations, and model details.

### A.1 Additional Implementation Details

We do not use bias terms in our model, for both Linear and LayerNorm layers. We initialize the model weights with a normal distribution of zero-mean and standard deviation of $0.02/(2*(\textit{idx}+1))**0.5$, where idx means transform layer index.

We train our model with AdamW optimizer (Kingma, ). The $\beta_{1}$, and $\beta_{2}$ are set to 0.9 and 0.95 respectively, following GS-LRM. We use a weight decay of 0.05 on all parameters except the weights of LayerNorm layers.

### A.2 Discussion on Zero-shot Generalization

We notice that LVSM has the problem of generalizing (zero-shot) to input images with aspect ratios that are different from training data, e.g., training with $512\times 512$ images and inferring on $512\times 960$ input images. We observe high-quality novel view synthesis quality at the center of synthesized novel views but also observe blurred regions at the horizontal boundaries that exceed the training image aspect ratio. We conjecture the reason is that our model is trained with center-cropped images in our implementation. In detail, as the Plücker ray will have a smaller density at the boundary of the long side of images, and our model is not trained with these data, it can not generalize well to them.

### A.3 Additional Visual Results

We show the visualization of LVSM at the object level with $256$ resolution in Fig. 9. Consistent with the findings of the experiment with $512$ resolution (Fig. 3), LVSM performs better than the baselines on texture details, specular material, and concave geometry.

Figure: Figure 9: Object-level visual comparison at 256 resolution. Comparing with the two baselines: LGM(Tang et al., ) and GS-LRM (Res-256) (Zhang et al., ), both our Encoder-Decoder and Decoder-Only models have fewer floater artifacts (last example), and can generate more accurate view-dependent effects (third example). Our Decoder-Only model can better preserve the texture details (first two examples).
Refer to caption: x5.png

### A.4 Additional Ablation Results

Additionally, we ablate our decoder-only LVSM on the object-level dataset. As shown in Table 10, decoder-only LVSM demonstrates better performance with more transformer layers, similar to our observation in Table 6. Interestingly, we find that decoder-only LVSM scales better, i.e., exhibits more performance gain with more transformer layers, compared with scene-level results in Table 6. We conjecture the reason is that the input views of the object-level data have larger-baseline cameras, in contrast with smaller-baseline cameras in the scene-level dataset RealEstate10K (Zhou et al., ).
Thus, the novel view synthesis task on the scene-level RealEstate10K dataset is easier and the performance saturates beyond the certain number of transformer layers.

Figure: Figure 10: Model size ablation studies on the object-level reconstruction. The following experiments are run with 8 GPUs.

## Peer Reviews

confidence: 5
contribution: 4
initial_rating: 6
presentation: 4
questions: Kindly refer to [Weaknesses]
soundness: 4
strengths: * The idea of achieving high-quality photorealistic NVS with minimal 3D inductive bias is brave. It is also impressive that LVSM implements this brave idea with a straightforward yet effective pure Transformer-based architecture.
* Experiments on several benchmarks demonstrate the effectiveness of the introduced LVSM
* The paper is well structured, and it is easy to follow.
summary: This work presented the Large View Synthesis Model (LVSM), which aims to achieve novel view synthesis (NVS) via a pure transformer architecture, bypassing the need for additional 3D inductive bias. In particular, LVSM explored an encoder-decoder model design and a decoder-only one, where the former is more efficient in inference with a more compact learned latent space, and the latter is more effective and scalable regarding visual quality. Experiments of object-level datasets and scene-level datasets demonstrate the superiority of the introduced LVSM.
weaknesses: * More discussion with Scene Representation Transformer (SRT) [Sajjadi et. al, CVPR 22]. LVSM seems to be a ‘reimplementation’ of SRT with more recent modules, which significantly limits the novelty of LVSM. The discussions in L141-L146 cannot convince me about the key contribution of LVSM. A more thorough analysis is suggested below.
  * The introduction should clearly reveal the similarities and differences between SRT and LVSM. The motivation (minimal 3D inductive bias) and architecture (encoder-decoder) of LVSM are similar to SRT, and it seems that the key difference is that LVSM adopts more advanced modules from LRM.
  * Different observations of ‘decoder-only’ architecture between LVSM and SRT.  SRT also explores the ‘decoder-only’ designs (see Sec. 4.3 “No Encoder Transformer” in SRT), which shows that ‘decoder-only’ performs worse than its ‘encoder-decoder’ counterparts. This observation happens to be contradicted to that of LVSM. It would be interesting to provide further analysis about what leads to this different conclusion despite similar settings.
  * More analysis is needed to understand why LVSM achieves good quality on NVS.  It is hard to understand why LVSM gets much cleaner images while SRT gets only blurry ones. Is it because LVSM is trained with more data? Besides, it would be beneficial to visualise the attention of LVSM (similar to Fig. 5 in SRT).
  * Noisy poses or unknown poses settings on LVSM. SRT achieves reasonably good results when the camera poses are noisy or even unknown (Fig. 4 in SRT). How does LVSM perform under similar noisy pose or pose-free settings? 


* Assessing the geometrical accuracy. For the object-level data, it would be better to reconstruct the 3D mesh using the rendered novel views, similar to Fig. 5 in latentSplat [Wewer et al., ECCV 24]. For the scene-level data, e.g., the single-view case in Fig. 1, it would be better to show the error map between the rendered and ground truth views (similar to Fig.6 in MVSplat [Chen et al. ECCV 24]), which makes it easier to understand how well the rendered views align with the given camera poses.


* Performance on more complex datasets. L527 claims that LVSM is not simply doing view interpolation. Since the results are mainly shown on simple data, e.g., RealEstate10K with zoom-in / zoom-out trajectories, it cannot justify this claim. It would be better to show some results on more complex datasets, e.g., MipNeRF360, Tanks and Temples.

---

confidence: 5
contribution: 4
initial_rating: 6
presentation: 3
questions: See limitations.
soundness: 4
strengths: The paper is well-motivated and and very well-written, though certain technical details could benefit from additional clarity (outlined below). The visual results are striking, as shown on the authors’ website, and I appreciate the authors provide additional results with limited GPU-hours, making reproduction more feasible for academic labs. Overall, this work is a valuable contribution to view synthesis research.
summary: This paper introduces two view synthesis models: one with an encoder-decoder architecture and another with a decoder-only design. Both models treat view synthesis as a predictive sequence-to-sequence task, using Plucker embeddings for camera position encoding. Without relying on intermediate 3D structures, the models achieve impressive results across object-level and scene-level view synthesis.
weaknesses: 1. Related Works. While the paper covers key prior work on 3D representation and few-shot view synthesis, it would benefit from a discussion of generative multi-view methods, especially recent works like Free3D (CVPR 2024, also uses Plucker embedding to encode camera poses) and EscherNet (CVPR 2024, also can be inferenced with varying number of reference/target views). These methods also do not rely on intermediate 3D representations, treating view synthesis as a sequence-to-sequence problem. Adding some proper discussions on how the proposed methods differ from these could provide valuable context for readers, and strength the community focussing on sequence-to-sequence learning type view synthesis.

2. Architecture Design Details.
- Encoder-Decoder Architecture: Could the authors clarify the choice of compressing input tokens into a fixed-length representation with latent tokens? What are the benefits over using uncompressed reference tokens (only with linear complexity)?
- Decoder-Only Architecture: Are attention masks fully bi-directional? It would be interesting to see if introducing asymmetric masking strategies (e.g., limiting specific views to certain tokens) could enhance generalization to different numbers of reference views. Additionally, if attention is fully bi-directional, the distinction between reference and target views seems blurred, as the loss can be computed from all views, (conditioned on all other views)?

3. Experiments

- Unified Model Training: Could the model be trained jointly on both object- and scene-level data instead of using separate models? Demonstrating this capability would advance the model’s utility toward a more general-purpose view synthesis framework.
- Varying Reference Images: The authors suggest the model performs well with varying numbers of reference images. To strengthen this claim, I recommend evaluating on a NeRF-Synthetic dataset instead of GSO in Fig. 5 (similarly as shown in EscherNet), which includes more complex objects in terms of textures and lighting. This would also enable clearer comparisons to other state-of-the-art, scene-specific methods like InstantNGP and 3D Gaussian Splatting that leverage 3D representations.
- Single-Image View Synthesis: Single-image view synthesis is a common use case, so it would be valuable to include a comparison to methods specifically designed for single-image scenarios to showcase the model’s adaptability and strong generalization.
- Plucker Embedding Generalization: Could the authors explore how well the Plucker embeddings generalize to different spatial coordinates? For instance, in scene-level experiments, would generation quality remain consistent if reference/target camera poses are applied with the same camera transformation, such as a 30-degree elevation or a 1-meter translation?

4. Limitations
A dedicated limitations section would help readers identify areas for improvement. Potential limitations include:

- Predictive Modeling: The predictive approach in LVSM may restrict outputs to interpolation, limiting extrapolation capabilities, especially in scene-level tasks (as shown in the website). Maybe a generative model variant trained on larger datasets address this? Can the authors provide some extrapolation results?
- Extreme Reference View Limits: How well does LVSM handle extremes in reference view numbers (e.g., only one view or over 100 views)? This could be an insightful addition, especially if single-view predictions are inconsistent or if performance degrades with many reference views.

I am happy to further raise the score if the authors can adequately address my concerns.

===================

Updated score based on the authors' rebuttal.

---

confidence: 5
contribution: 2
initial_rating: 6
presentation: 3
questions: - Why this simple architecture performs so good compared to SRT? What's the key contribution, only transformer vs. CNN? 
- What's the performance on other scene-level datasets?
- How about the 3D consistency?
- The decoder-only transformer also be used for CAT3D, while they are in latent space for diffusion. If we train the similar architecture, but on diffusion-based architecture, what's the performance? In particular, could we use it to build a large foundational 3D model upon the pre-trained 2D diffusion models? The model not only works well on a special trained dataset, but can be generalised well to arbitrary images.
soundness: 3
strengths: ### S1 -- Good results on an interesting task
- The task of synthesizing novel views from a set of input views is interesting and very challenging. The proposed method seems to work well on both object-level dataset and scene-level dataset.
- Base on the visual results shown in Figure 3 and 4, the proposed deterministic pipeline also can imagine the new content which is invisible from the input views.

### S2 -- Simple ideas and careful implementations
- There are two main transformer-based architectures. 
    - One is an encoder-decoder architecture, which used the transformer encoder to encode all input views into latent space, and then use a query camera pose to get the target view images. This architecture is very similar to SRT, except here the encoder is transformer architecture.
    - The second idea is to design a decoder-only architecture. This reduces the necessary of the one representation from the encoder.
- These two ideas are carefully implemented and validated through ablation studies. In particular, the decoder-only architecture seems quite effective in achieving a set of input views, from 1-10.

### S3 -- Good writing
- The paper is very well written, with clear motivations, sufficient technical explanations and illustrative visualization.
summary: This paper studies the task of synthesizing novel views from a set of views without explicit 3D representation.
The key idea is to use a transformer architecture to bypass the 3D inductive biases.
To achieve this goal, an encoder-decoder and a decoder-only architectures are proposed to conditional on image and camera pose for novel view synthesis. 
Experiments are trained on object-level Objaverse dataset, and scene-level RealEstate10K dataset, and tested on object-level Google Scanned Objects and Amazon Berkeley Objects and scene-level RealEstate10K dataset.
They demonstrate reasonable results on novel view synthesis, outperforming the existing state-of-the-art approaches.
weaknesses: ### W1 --- Significant is not well demonstrated
- The proposed idea is a very specific, minor change to SRT -- basically using a slightly different transformer encoder or decoder to replace the original CNN. Fundamentally, I am not fully convinced that it is even crucial to use only transformer architecture than the CNN-based feature extraction and then do the transform.
- This small change seems to lead to a large improvement on both object-level and scene-level datasets. However, if we train the original SRT on the same dataset with the same computational GPUs, what's the performance? 
- A fair comparison to the highly related work (SRT) should be provided. The discussion in L140-145 is also not a very strong claim. 
- The decoder-only architecture is very interesting and useful, but some related work like 3DIM, Free3D, CAT3D also used it in the diffusion for a set of views with the Plücker Rays embedding as conditional. Why this architecture is better than these difussion-based methods, which also used the transformer in pixel or latent space?

### W2 --- More results should have been expected
- I expected more visual results on scene-level cross-domain datasets. Figure 4 shows only result on RelEstate10k, which is a relative easy dataset. How about the performance on the scene-level datasets, such as DL3DV, Mip-NeRF 360, or other traditional NeRF Datasets?
- This deterministic model can also provide sharp results for invisible regions, which is quite interesting. However, the authors only highlight them in the object-level results, how about the scene-level results? 
- Does the scene-level model perform good for the extrapolation, instead of interpolation?
- In 3DIM, Free3D, SV3D, CAT3D, they used some temporal attention to ensure the consistency of generated images. How do the authors active the 3D consistency in the design? Besides, the multi-view rendered results as in 3DIM and Free3D or reconstructed 3D as in SV3D and CAT3D will make the paper stronger to show 3D consistency and structure, while they want to bypass the 3D representation.
- It will be helpful to provide some visual examples of the failure cases.
- The ablations studies are only related to the different layers of the transformer, and the attention architecture. If the authors argue the large contribution of transformer compared to CNN, a fair ablation should be made to use SRT-related architecture under the same experimental settings.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: Suggestions - clarifying the scaling (model size, compute) discussion (with model parameter counts), and comparison to prior work.
Also if possible it would be good to have the encoder-decoder vs decoder-only comparisons for both scene- and object-level (i.e. Table 2).
soundness: 3
strengths: The model variants introduced compare well to prior work; the large improvements compared to baselines for the decoder-only variant are impressive.  The investigation of both encoder-decoder and decoder-only as modeling choices (with the discussed tradeoffs) is also interesting and seems novel.  The discussion of the effects of compute (model size, compute available) on performance, while disorganized, is quite interesting.
summary: The authors propose to train a generalizable novel view synthesis model with minimal 3D inductive biases, training on posed multi-view data to map images and camera rays to novel views, with target rays as queries.  This is supervised by a simple photometric and perceptual novel view loss.  
The architecture uses ViT-like image tokenization to embed images and corresponding Plucker rays, and the target views are decoded by Plucker ray queries.  Two Transformer architectures are considered – decoder-only, and encoder-decoder.
These are compared to prior work, and differences in performance between the two choices are compared and contrasted.
weaknesses: There is limited novelty in the approach of removing handcrafted 3D representations – as the authors point out this was done in SRT [1], with a less effective and scalable architecture, as well as in the stereo case [2], which proposes a similar philosophy of input-level inductive biases, and in the multi-view case [3], where the view synthesis branch closely resembles LVSM.  Clarification of the novel elements of this architecture compared to prior work would be great, beyond the discussion of encoder-decoder vs decoder-only.

The presentation is somewhat messy.  The discussion of encoder vs encoder-decover in Section 3.2 takes up a lot of room, but much of it is standard Transformer information; this could be condensed to describe specifically how these concepts relate to the architecture rather than be a general introduction.  Also, there is information about model scale and compute scattered throughout the document, and the model sizes are not in terms of #params but layers.

[1] Sajjadi, Mehdi SM, Henning Meyer, Etienne Pot, Urs Bergmann, Klaus Greff, Noha Radwan, Suhani Vora et al. "Scene representation transformer: Geometry-free novel view synthesis through set-latent scene representations." CVPR 2022

[2] Yifan, Wang, Carl Doersch, Relja Arandjelović, Joao Carreira, and Andrew Zisserman. "Input-level inductive biases for 3d reconstruction." CVPR 2022

[3] Guizilini, Vitor, Igor Vasiljevic, Jiading Fang, Rares Ambrus, Greg Shakhnarovich, Matthew R. Walter, and Adrien Gaidon. "Depth field networks for generalizable multi-view scene representation." ECCV 2022

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: The architecture of decoder model is elaborated from L303-308, the attention layer in the transformer block seems to be self-attention. During both training and inference stage, I wonder how many input and target views are used for each batch sample, will the self-attention between target views affect the result? If I want to get the target view at P1 and P2, will inferencing them at the same feedforward pass be different from inferencing one by one?
soundness: 4
strengths: 1. this paper minimizes 3D inductive biases for scalable and generalizable novel view synthesis;
2. Two LVSM architectures—encoder-decoder and decoder-only—are designed to minimize 3D inductive biases;
3. Both of the two models achieve impressive sparse novel view synthesis performance. 
4. The impressive result might provide a potential new representation for generalizable NVS task.
summary: This paper proposes a novel transformer-based approach for scalable and generalizable novel view synthesis from sparse-view inputs. Two models are proposed. both of which bypass the 3D inductive biases used in previous methods and address novel view synthesis with a fully data-driven approach. Comprehensive evaluations across multiple datasets demonstrate that the work achieve state-of-the-art novel view synthesis quality.
weaknesses: The architecture level is not well explained and might cause a bit confusing, I will elaborate it in questions part.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: 1.	It is commonly believed that 3D reconstruction improves view consistency for novel view synthesis. By removing 3D inductive bias, how does LVSM ensure consistency across novel views? Is this knowledge simply learned using attention mechanism? To demonstrate view consistency, you may want to provide some 3D-aware metrices for comparison, such as reprojection error or depth estimation, which will help my evaluation a lot.
2.	Could you please provide a more detailed diagram for the LVSM architecture? Specifically, such a diagram will help readers understand the methodology more easily. In the submission, only Figure 2 is available with very brief design.
3.	Refer to weakness.
4. Will you release your codes and pretrained models after possible acceptance?
soundness: 4
strengths: 1.	The proposed models achieve impressive reconstruction results in terms of PSNR, SSIM, LPIPS, and qualitative evaluation.
2.	The proposed models can be trained with a single A100 80G GPU. In contrast, existing general reconstruction models, like LRM, LGM, and GS-LRM, usually require a large number of computational resources.
3.	The proposed model is generalizable to an unseen number of input views, from single view to more than 10 views.
4.	Ablation studies show the scalability of LVSM, where transformers with more layers generally perform better in terms of the quantitative reconstruction results.
summary: This paper proposes LVSM, a general novel view synthesis (NVS) model for sparse-view inputs. It aims to improve the quality, efficiency, and scalability of NVS by minimizing 3D inductive bias. Different from previous works, LVSM directly utilizes transformer-based backbone to synthesize novel views without intermediate 3D representations and corresponding rendering equations. The author(s) introduce two versions of LVSM. The first model, encoder-decoder LVSM, encodes inputs into a scene representation token and decodes it into novel views. The second model, decoder-only LVSM, further removes the need for intermediate representation by adopting a single-stream transformer. Both models patchify the posed input view and use Plucker ray embeddings to tokenize the target view. Experiments show that LVSM trained on 2-4 input views has generalizability to an unseen number of views, and it outperforms the SOTA GS-LRM by 1.5 to 3.5dB PSNR.
weaknesses: 1.	The qualitative results are not consistent with the quantitative results. In specific, Table 1 shows that the encoder-decoder LVSM achieves similar quantitative results as GS-LRM, and sometimes worse in object-level datasets. However, Figure 3 and Figure 7 substantially outperform GS-LRM. I am worried about potential cherry picks in the qualitative results. It will help evaluation by showing more examples as well as some failure cases of LVSM.
2.	Missing experiments for efficiency comparison. Specifically, the paper claims the higher efficiency of LVSM than previous methods, while there is no comparison between LVSM and other NVS models.

Meta-review: The paper introduces LVSM, a transformer-based approach for novel view synthesis that effectively minimizes 3D inductive biases while achieving state-of-the-art performance across object- and scene-level datasets. Reviewers praised the impressive visual and quantitative results, the thoughtful architectural design of both encoder-decoder and decoder-only variants, and the extensive ablation studies that clarify the contributions of key model components. The model's scalability, efficient training on limited computational resources, and generalizability to varying input views were also highlighted as significant strengths, making this work a valuable contribution to the novel view synthesis community.

Decision: Accept (Oral)