The Less You Depend, The More You Learn: Synthesizing Novel Views from Sparse, Unposed Images without Any 3D Knowledge
Authors: Haoru Wang
Kai Ye
Minghan Qin
Yangyan Li
Wenzheng Chen
Baoquan Chen
Venue: ICLR 2026 Poster (2026)
Keywords: novel view synthesis
feed-forward
scaling behavior
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Recent advances in feed-forward Novel View Synthesis (NVS) have led to a divergence between two design philosophies: bias-driven methods, which rely on explicit 3D knowledge, such as handcrafted 3D representations (e.g., NeRF and 3DGS) and camera poses annotated by Structure-from-Motion algorithms, and data-centric methods, which learn to understand 3D structure implicitly from large-scale imagery data. This raises a fundamental question: which paradigm is more scalable in an era of ever-increasing data availability? In this work, we conduct a comprehensive analysis of existing methods and uncover a critical trend that the performance of methods requiring less 3D knowledge accelerates more as training data increases, eventually outperforming their 3D knowledge-driven counterparts, which we term “the less you depend, the more you learn.” Guided by this finding, we design a feed-forward NVS framework that removes both explicit scene structure and pose annotation reliance. By eliminating these dependencies, our method leverages great scalability, learning implicit 3D awareness directly from vast quantities of 2D images, without any pose information for training or inference. Extensive experiments demonstrate that our model achieves state-of-the-art NVS performance, even outperforming methods relying on posed training data. The results validate not only the effectiveness of our data-centric paradigm but also the power of our scalability finding as a guiding principle.

THE LESS YOU DEPEND, THE MORE YOU LEARN: SYNTHESIZING NOVEL VIEWS FROM SPARSE, UNPOSED IMAGES WITH MINIMAL 3D KNOWLEDGE
Haoru Wang"
Kai Ye*
Minghan Qin ByteDance
Peking University
Peking University
bu52du8stu. www.sttu...u...uuuc.oo edu.cn
ye_kaißpku.eer eew.cp.ku.eddcc edu.cc
mingkaan/byteedacce.co
Yangyan Lit
Wenzheng Chen
Baoquan Chen Peking University
Ant Group
Peking University & Beijing
yangyan, gyoaptg
Academy of Artificial Intelligence
ba oquua @plcu edd..ooo
wenzhengchen@pku.. wetpeeed.pi.........mm co
The Less You Depend on Explicit 3D Knowledge
Blas-driven
Data-centric
Handerafted Regresentation
Representation
Gaussian Splating
Predefincal Beniering
Lesmable Benikring
Depundency
Camera Poses Amsotated by SIM
Camera Poses Learned by Network
COLMAP
Transfonmer
Stroture Srom Mocoz
Leamable Laterz Pose
MVSplat
S
LVSM
Ours
31
31
27
27
PSNK
27
23
23
23
MVSplat
LVSM
UP-LVSM
19
19
1k
4k
16k
66k.
Ik
4k
16k
66k
ik
4k
16k
66k
Training
Training Secnes
Training
The More You Learn from Abundant Data
ABSTRACT
Recent advances in ford-forward Novel View Synthesis (NVS) have led to a divergence between design philosophies: bias-driven methods, which rely on explicit 3D knowledge, such as handcrafted 3D representations (e.g., NeRF and 3DGS) and camera poses annotated by crructure-from-Motiie and data-centric methods, which learn to understand 3D structure implicitly from large-scale imagery data. This raises a fundamental question: which paradigm is more scalable in an era of ever-increasing data availability? In this work, we conduct comprehensive analysis of existing methods and uncover a critical trend that the performance of methods requiring less 3D knowledge accelerates more as training data increases, eventually outperforming their 3D knowledge-driven counterparts, which term "the less you depend, the more you learn." Guided by this finding, we design feed-forward NVS framework that removes both explicit scene structure and pose annotation reliance. By eliminating these dependencies, our method leverages great scalability, learning implicit 3D awareness directly from vast quantities of 2D images, without any pose information for training or inference. Extensive experiments demonstrate that our model achieves state-of-the-arr NVS performance, even outperforming methods relying posed training data. The
algorithms,
"Equal contribution. 'Equal advisory.

---

results validate not only the effectiveness of our data-centric paradigm but also the power of our scalability finding as guiding principle.
INTRODUCTION
Novel View Synthesis (NVS). a long- long-standing challenge in computer vision and graphics, aims to render high-fidelity, unseen views of a scene from a collection of 2D images. Traditional solution typically involves Structure-rrom-Motion (SfM) (Wu et al., 2011; Schonberger & Frahm, 2016) to estimate camera parameters of each view, followed by per-scene fitting of representations like Neural Radiance Fields (Mildenhall et al., 2020, NeRF) 3D Gaussian Splatting (Kerbl et al., 2023, 3DGS). Recently, the field has experienced a rapid shift towards a feed-forward paradigm (Yu et al., 2021; Charatan et al., 2024; Chen et al., 2024; Ye et al., 2025; Jin et al., 2025; Jiang al., 2025), where the specific scene representations are directly predicted by neural networks instead of by a seffient-descent-baeeee optimization. Leveraging powerful priors learned from large-scale datasets, these approaches can synthesize compelling novel views from sparse, wide-baseline, or even entirely unposed input images, bypassing the restrictive assumptions of their optimization- based predecessors.
Reviewing recent advances in feed-forward NVS reveals a key distinction between two design philosophies. The bias-driven one (Yu al., 2021; Charatan et al., 2024; Chen et al., 2024; Ye et al., 2025) explicitly injects 3D knowledge, such as human inductive biases (e.g.. handcrafted rendering formulas and predefined 3D representations) or intermediate 3D clues estimated by heuristic algorithms (e.g., camera parameters obtained by COLMAP (Schonberger & Frahm, 2016)) directly into the method architecture. The alternative, the dataa-centiic approaches (Sajjadi et al., 2023; Jin et al., 2025; Jiang et al., 2025), seek to learn 31) knowledge implicitly, allowing spatial understanding to be distilled directly from vast quantities of 2D image data. This divergence raises fundamental questions about the future of the field: which paradigm proves more effective and scalable, especially in an era of increasingly abundant data?
In this work, investigate the relationship between explicit 3D knowledge dependencies and data scalability to address these questions. We categorize existing methods (Charatan et al., 2024; Chen al., 2024; Ye al., 2025; Jin al., 2025) by their dependence 3D knowledge and systematically analyze their performance across varying data regimes. Our experiments reveal a consistent and critical trend: methods that require less explicit 3D knowledge demonstrate superior data scalability. Their performance accelerates more significantly as the amount of training data increases, eventually surpassing their 3D knowledge-driven counterparts. This finding highlights fundamental trade-off: while explicit 3D knowledge provides a useful scaffold for training on limited data, it creates a performance bottleneck at scale. We conclude that reducing dependence on 3D knowledge is essential for developing truly scalable NVS approaches.
Building on these insights, we propose UP-LVSM, a data-centric NVS framework designed to unlock scalability by eliminating 3D knowledge dependencies. Using a pure Transformer architecture similar to Jin et al. (2025), UP-LVSM models scenes implicitly within a latent space, bypassing the need for predefined 3D structures. We further identify camera poses annotated by pipelines as an indirect form of 3D knowledge that hinders scalability. To address this, we introduce novel Latent Plücker Learner to infer camera geometry directly from images in self-supervised manner, further bypassing the need for pose annotations during training.
By shedding these dependencies 3D knowledge, UP-LVSM fully leverages data scaling to synthesize photorealistic and 3D-consistent novel views from sparse, unposed images-even without any pose supervision during training. Experiments demonstrate that UP-LVSM outperforms state-of- the-art approaches that rely on explicit scene structure or pose annotations. This not only confirms the viability of minimizing 3D knowledge but also establishes new path toward scalable and generalizable novel view synthesis learned purely from 2D observations.
Our primary contributions are summarized follows:
1. We perform a systematic analysis of NVS methods through the lens of 3D knowledge, uncovering the key principle that reducing dependence on such knowledge the key unlocking scalability.
2. We propose UP-LVSM, a novel data-centria NVS framework that effectively learns spatial reason- ing from unposed 2D images without requiring explicit 3DD representations or pose supervision.
2

---

Extensive experiments demonstrate that framework achieves both superior scalability and state-of- the-art performance, validating our key hypothesis and the effectiveness of the data-centric paradigm.
REVISITING 3D KNOWLEDGE IN FEED-FORWARD NOVEL VIEW SYNTHESIS
2.1 PRELIMINARIES
Novel View Synthesis The goal of novel view synthesis (NVS) is to reconstruct a 3D scene representation, denoted as S, from a given set of 2D images and their corresponding camera poses, {(T', ((T' pt) This reconstructed scene is then used to render novel image T from viewpoint PT. The process is typically supervised by reconstruction loss L(T, T) that measures the difference between the rendered image T and the ground-truth image T. This can be formally expressed as:
target
(I) where A is the scene reconstruction function and R is the rendering function. In settings with dense observations (N 2 50), the scene is typically modeled using explicit 3D representations like Neural Radiance Fields (NeRFs) (Mildenhall et al., 2020) or 3D Gaussian Splatting (3DGS) (Kerbl et al., 2023). The differentiable nature of these representations allows the reconstruction function A to be implemented as a per-scene optimization process (Mildenhall et al., 2020; Barron al., 2022; Kerbl ct al., 2023; Yu et al., 2024), which effectively yields photorcalistic novel views.
S == A A=,
I.....,,,. P T= PT),
Ferw-Forward NVS Despite their promise, per-scend optimization approaches are limited by their reliance on dense observations, making them less suitable for under-constrained settings where inputs are sparse (typically N < 5) or camera poses Py are unavailable. To address this, feed-forward approaches (Yu et al., 2021: Charatan et al., 2024; Chen et al., 2024; Ye et al., 2025; Jin et al., 2025) employ a neural network as the reconstruction function A. By training on large-scale multi-viww datasets (Zhou et al., 2018; Deitke et al.. 2023; Ling et al., 2024), these methods learn powerful priors to compensate for the ambiguity of sparse inputs, enabling reconstruction of the scene in a single forward pass. More discussions are provided in Appendix B.
2.2 3D KNOWLEDGE DEPENDENCE IN FEED-FORWARD NVS
In reviewing recent advances in feed forward NVS, the methods can be characterized by their varying reliance on 3D knowledge, which typically manifests in two key aspects: explicit scene structure and pose annotation availability. Table demonstrates the categorization.
Method
Explicit Scene Structure
Modeling
R. Modeling
Problem Settng
Pl
PixeINeRF (Yu et 2021) 2021)
NeRF
Volumetric Rendering
PixelSppaat (Charatan al., 2024)
3DGS
Gaussian Splatting
posed
MVSplat (Chen et al., 2024)
3DGS
Gaussian Splatting
y
LVSM (Jin et al., 2025)
Latent
Learnable Network
Ours (PT-IVSM)
NoPoSplat (aa (Ye et al., 2025)
V
3DGS
Latent
Gaussian Splatting
Learnable Network
posed posed-target target
/
SPFSplat* (Hunng & Mikolajezyk, 2025)
3DGS
Gaussian Splatting
X
Rayzer* (Jang et al., 2025)
Latent
Learnable Network
unposed
Ours (UP-LVSM)
Lateut
Learnable Network
X
Table 3D Knowledge in Feed-Forward NVS. We characterize recent feed-forward NVS methods (edenotes concurrent work) based on their varying dependence on explicit 3D knowledge (i.e., the choice of S and R. modeling) and pose availability (i.e., whether Pr and PPr are provided).
Explicit Scene Structure As demonstrated in Table 1, this refers to the integration of explicit 3D representations handcrafted rendering operations directly into the NVS architecture. Methods based on established 3D structures, such as PixeINeRF (Yu et al., 2021), PixelSplat (Charatan et al., 2024), MVSplat (Chen al., 2024), NoPoSplat (Ye et al., 2025), and SPFSplat (Huang & Mikolajezyk, 2025), incorporate explicit representations like NeRF (Mildenhall et al., 2020) or 3DGS (Kerbl et al., 2023) along with their associated rendering equations to model the scene and the render function R. These architectural choices enforce a strong geometric consistency based on principles like volumetric rendering or plane sweeps, thereby explicitly injecting 3D knowledge into method designs. In contrast, approaches like LVSM (Jin et al., 2025) and Rayzer (Jiang al., 2025), treat scene modeling as learning problem, representing the scene implicitly as latent tokens and allowing its modeling be learned implicitly from data.
3

---

Pose Annotation Availability Another key aspect is the availability of camera pose annotations for input views Pr and target views PT. This depen- dency defines the problem's constraints, leading to three primary settings, as detailed in Figure I and Table I: (1) posed: camera poses are required for both input and target views. (2) posed-target: in- put view poses are unknown, but target view poses are required for supervision. (3) unposed: no cam- pose information is assumed for either input target views. Detailed problem setting definitions can be found in Appendix A. While the first two set- tings rely on datasets with poses often generated by
Scene Regne
Render
Training
Input Viems
Recon. Module
Thrget
Target Var
Target Poss
Input Piwe
Posed target
Repr.
Render Medule
Traising
Input Views
Recon.
Target Pars
Tergen larget
SfM pipelines like COLMAP (Schonberger & Frahm, 2016), it is important to recognize that SfM itself is built heuristic algorithms and geometric inductive biases and often produces incorrect estimations. Con- sequently, relying on these poses is an indirect form of dependence on 3D knowledge. In this context, only the unposed setting truly operates without such dependence, typically relying on a latent pose learned in self-supe manner (Sajjadi et al., 2023).
Unposed
Repe.
Render Medule
Training
Ingue Views
Recon. Module
Lavet Pate
Exp
Poss Learner
Figure 1: Pose Annotation Availability.
2.3 EXPLICIT VS. LEARNED 3D KNOWLEDGE: A DIVERGENCE
Viewing predefined 3D structure and pose availability as forms of explicit 3D knowledge reveals a fundamental divergence in feed-forward NVS design. On one side, the bias-driven paradigm relies on explicitly injecting 3D knowledge as human inductive biases. On the other, the data-centric paradigm allows this knowledge to be learned implicitly from large-scale imagery. While both paradigms have proven effective, a critical question remains: which is more scalable and learns more effectively in an era of increasing data abundance? This work argues that reducing dependence on explicit 3D knowledge leads to superior scalability and, ultimately, better performance. We term this principle "the less you depend, the more you learn" ", and provide a detailed analysis to substantiate this hypothesis in the following section.
THE LESS YOU DEPEND, THE MORE YOU LEARN
In this section, we present our analysis to validate the hypothesis that the less you depend, the more you learn. Specifically, the less you depend refers to reducing reliance on 3D knowledge in methodology design, including explicit scene structure and camera pose annotations. Meanwhile, the more you learn refers to scalability, which is defined as how performance improves the amount of training data increases. By examining the relationship between performance and data quantity for different methods, we find the performance of methods that requires less 3D knowledge accelerates more as data scales.
Dataset While our experiments are conducted across diverse datasets (Zhou et al., 2018; Ling et al., 2024; Liu et al., 2021; Deitke et al., 2023), explained later in Section 3.3, we select the RealEstate 10K dataset (Zhou et al., 2018) as the representative to introduce our exper- imental setup, which contains real-world imagery over 70K scenes. To evaluate scalability, we create four training subsets of increasing size (little, medium, large, and full, summarized in Table 2). while using a single, consistent test set for all evaluations to ensure fair comparison.
Subset
Train
Test
Little
1202
Medium
4121
7286
I Large Full
16449
66033
Table 2: Number of scenes in RealEstate subsets.
Representative Methods We select methods that span different levels of scene structural biases and vary in problem settings to represent different dependencies on 3D knowledge. In the posed setting, contrast the structural bias-driven MVSplat (Chen et al., 2024) with the bias-fit LVSM (Jin et al., 2025). In the posed posed-targer setting, we select NoPoSplat (Ye et al., 2025) as the bias-driven representative, As established bias-free method exists in the posed-target setting, we simply
4

---

adapt LVSM for this setting to further enhance the comprehensiveness of analysis, denoting it as PT-LVSM. We leave adaptation details in Appendix to maintain the flow of our main analysis.
Experimental Results We evaluate these methods trained different subsets of the RealEst 1 10K dataset to assess their scalability, as shown in Table 3. We quantify scalability as the average gain in NVS metrics (PSNR, SSIM, LPIPS) for every 4x increase in training data.
Method
Bins free
Py-free
Little Subset
Medium Suboet
PSNR1 SSIM1 LPIPS]
Large Subset
Fall Subset
Arg. Gain 1 APSNR ASSIM ALPIPS
1012X
/0.873/0
26.5/0374/00. /0.125
27.60/0.57
NoPoSplat
0.54D/ 0.142
0.139
0.004
PLANSM
20.80 0.659 0.251
22.9210.731 / 00184
24.54/07881 /0.159
26.00 0.825 00.35
1.72/ 0.055/ 00.031
Table 3: Scalability Comparisons ReallEstate 10K (Zhou al., 2018).
3.1 DEPENDENCY ANALYSIS ON EXPLICIT SCENE STRUCTURE
As shown in Table and Figure 2, our experiments reveal a clear trade-off between explicit scene structure and data scalability. Methods with explicit scene structure (i.e.. MVSplat and NoPoSplat) excel in low-data regimes (e.g., IK scenes) but fail to scale effectively with more data. Conversely, implicit methods (i.e., LVSM and PT-LVSM), while initially underperforming, demonstrate substan- tial performance gains as the training set grows to 66K scenes. This confirms our hypothesis that reducing reliance on explicit 3D structures is crucial for unlocking data scalability.
28
0.90
0.18
Posed Setting
26
0.86
LPIPS
0.14
0.82
0.10
LVSM
IVSM
LVSM
0.78
0.06
1k
4k
16k
66k
1k
4k
16k
66k
1k
4k
16k
66k
Training Scenes
Training
Training Scenes
Tärget- Target-posed posed Setting
LPIPS
0.18
0.7
0.10
NaPal
NoPaSplai PE-LYSM
NaPaSppaa PI-LNSM
PI-LVSM
0.02
1k
4k
16k
66k
1k
4k
16k
66k
1k
4k
16k
66k
Training Scenes
Training Scenes
Training Scenes
Figure 2: Scalability Comparison on Varying Choices of Scene Structure.
2 Input Views
GT Target View
1k Training
16k Training
66K Training
NoPoSplat
PI-LYSM
Figure 3: Intuitive Explanation. On posed-target setting, both NoPoSplat and PT-LVSM fail to infer correct spatial structure when trained with IK scenes, resulting in artifacts at the right bottom of target views. While bias-driven NoPoSplat consistently makes mistakes, PT-LVSM significantly improves when training data scales up from 1K to 66K, eventually outperforming NoPoSplat.
Intuitively, strong structural biases act as a necessary scaffold when training data is scarce, compensating for a lack information. However, as data becomes abundant, these same biases become restrictive, limiting the model's ability to learn complex patterns directly from the data and thus hindering generalization. As illustrated in Figure 3, performance of the data-centric method improves with more data, while the bias-driven approach stagnates.
Discussion
5

---

3.2 DEPENDENCY ANALYSIS ON POSE ANNOTATIONS
Beyond 3D bias, we find that reliance pose annotations also critically impacts scalability. As shown in Table and Figures 4, PT-LVSM demonstrates significantly better scalability than LVSM. Although both data-centric methods with weak 3D biases, they differ in a key aspect: LVSM requires input images annotated with camera pose Pz, whereas PT-LVSM does not.
Discussion Benefiting from known camera poses that provide strong 3D clues, posed methods like LVSM should theoretically have a significantly higher performance ceiling. However, it is observed that our free-free PT-LVSM quickly closes this gap as data scales. We attribute this to noise in pose annotations. As discussed in Section 2.2, pose annotations in real-world datasets (Zhou et al., 2018; Yao et al., 2020; Yeshwanth et al., 2023; Ling al., 2024) are typically generated by Structure-from Prrom-Motion tools (Wu et al., 2011; Schonberger & Frahm, 2016) that are built on geometric inductive biases, thereby introducing noise and inconsistencies. We argue that relying on these poses during training is an indirect form of 3D knowledge dependence, which creates bottleneck scale. See Appendix for detailed explanation.
29
IASM PRENSM
20 1k
16k
66k
Training Ncenes
Figure 4: Scalability of LVSM and PT-LVSM.
3.3 UNLOCKING DATA- CENTRIC FEED-FORWAAD FORWARD NVS
Motivation Our analysis indicates that data scalability in feed-forward NVS is fundamentally limited by dependencies on explicit scene structure and camera pose annotations. As PT-LVSM still relies target view poses PT for its posed-target setting (Figure I), critical question remains: can we achieve even greater scalability and surpass the performance ceiling of pose-dependent methods like LVSM by removing this final dependency? We answer this by proposing UP-LVSM, novel fred-forward NVS framework that learns 3D knowledge implicitly from 2D images without any pose annotations, which will detail in Section 4.
0.96
0.26
PSNR
26
SIN
01.4
0.20
LYSM FEEYSM UP-LVSM
0.72
0.14
0.60
0.08
4k
16k
66k
1k
4k
16k
66k
1k
4k
16k
66k
Training Scenes
Training Scenes
Training Scenes
Figure 5: Superior Scalability of UP-LVSM on RealEstate
Method
Pe-free
Py-free
#Scenos: IK
ReallFstatc LOK, (Zhou et al., 2018): /SSMI /IPIPS)
AScenes: 4K
#Sienno: 16K
#Seenes: 66K
APSIRR/ ASSIM ALPPIS Arg. Gain T
LVSM PE-LYSM
X
25.67 (083//831//0 D.
26.52/0851 /..1//85//0.33
37.1/0334:0. 20.1420..12
27.40/857//0.11
064/0614/000) G.2/ 263 2.3/00074/000 0.001
2080 / /O.550 /2//B.2 0.231
22.92 /0.154
24.54/0781 /0.19
26.00/0.25/0.. 0
UP-ENSM
21.03/0670/00 0,226
20.9700744000 /0.165
26
2690/0317/0.44)
28.8/
Method
Pe-froe
Py-fice
(Ling
SSIMI /LPIPS)
4Sornes:
APSNR/
Avg.
LVSM PT-VVSM UP LVSM
1661 FO531 /0.457
17.22/0572/0 /0.44223
19.14/0603
12/00035
Converged.
1658 0558:0410/0444
17
/947/0/641
115/000
16.45 0.387 0,477
0.556/0.581/ /.422
19.59/0.53/ 0.366
1.57/ /0.05
Method
Pr-frree
Py-free
AScenes: 0.2K
ACID (Liu et al.. 2021y; PENRY SSIMI/ /LPIPS)
#Scenes: 0.8K
AScenes: 3K
*Scenes: 13K
APSNR/ ASSIM SAPIPS Aog. Cain
LVSM PT-LYSM UP-LYSM
23.43/ 33443/0710 00.77/00226
25 25.96/ 0.759/ 0.2223
27.01 /.379 0.211
1.79 0,99 /0.017 4.17/0.102 130
Noc Converged,
18.1 (0.65/ 0.059
26.31 / /0760/
26.75/ 0.56/0966//00 0.
/ 0.643
26.38 /0.71/
27.21 /..1//0.00/0.11
0..224
Objavens (Deikke al., 2003): PSNR: /SSIM) LLPPS
APSNR/
Pc
free
PUbjeess:
#Ubjects:
#Objects: 32K
#Objects: 128K
Aug.
LVSM
24.58 0.177
2850/ 03887 10.000
29:33
EL99 0.896
30.22 11966 0.087
1.27 0.029 /OA2S
PELVSM UR-ENSM
21.6/
7088/0554/0..3 0.526
25.0 0.855 0.19
26.33
0..33//8852// 0,141
0 0.120
1.92/ 0.034/ /O008
/0712/ 0.403
25.38 70.75/0.277 00.275
26.00 16.00/0.000 1
26. 12.
D.329 1.156
2.11 / 0.040/ (L.85
Table 4: We conduct extensive experiments across diverse datasets (Zhou et al., 2018; Ling et al., 2024; Liu et al., 2021; Deitke et al., 2023) to validate our hypothesis.
Discussion As demonstrated in Figure 5 and Table 4, our UP-LVSM achieves consistently superior scalability by fully eliminating reliance on explicit 3D knowledge, which validates our hypothesis. Moreover, benefiting from great scalability, our UP-LVSM achieves -fate-of-the-art performance using only 2D supervision, which unlocks the full potential of data-driven NVS learning. More analysis about data scalability of our method is provided in Appendix E.

---

4 METHODOLOGY
As motivated in Section 3.3, we propose UP-LVSM (Unposed Large View Synthesis Model) to unlock scalability by eliminating the need for explicit scene structure and camera pose annotations. This, however, places our method in the challenging unposed setting (Figure 1), where the core difficulty lies learning without the explicit target pose supervision (Py) available in simpler settings like posed-target. To this end, we propose the Latent Plücker Learner, the core component in UP-LVSM that learns meaningful latent pose space in self-supervised manner. this section, we will detail these technical designs and provide experimental results to validate the effectiveness of our proposed method, highlighting its capability of synthesizing high-fidelity novel views directly from unstructured 2D image collections.
4.1 TRANSFORMER BASED ARCHITECTURE
As illustrated in Figure 6 (a), we build upon previous works (Jin et al., 2025; Wang et al., 2025), employing Transformer (Vaswani et al., 2017) to construct our forward neural networks as encoder-decoder architecture. As detailed in Figure 6 (b), it first encodes input images I into patchified tokens using DINOv2 (Oquab et al., 2023), and then employs Transformer networks to reconstruct scene latents. A decoder takes the scene latents and the camera pose information in latent Plücker as inputs to synthesize novel views. See Appendix for detailed network architecture.
Inputs E ge
Patdilfed Tokess
Concatenoted
Input
Scene
Synthesized
Latent
Target View
Flaiten
[Dacarded]
DINOV2
Encoder
Encoder
Decoder
Flatico
Trans-
(licarded)
former
Template Scone
gt
Lateat
I Latent Pläcker Learner
Trans-
(Dicarded))
Targel
Synthesieed
Decoder
Flanto
former
Dapairhity
Latent Plücker
Ground Turget Views
Linear
Target Views
Latent Pläcker ggg"sees
Concatenaned
Tokets
Calers
c aaoso
(a) Pipeline Overview
(b) Encoder- Decoder Implementation
Scene Latent
Tokens
Concatrnated
R. 2e (yrm
Paw Tokees
Lateut Placker Learner
Platien
MLLP
eapemell of
Latent Pläcker Learner
Decoder
DINO<2
Trans- former
Mlldect Rabedding
Target E grArs
Scene
Ground Truth Target Views
Plickkr
Synthesized Target View
Latent Pläcker
(c) Autoencoder-lik Latent Plücker Learning
(d) Latent Plücker Learner Implementation
Figure 6: UP-LVSM Architecture
4.2 LATENT PLÜCKER LEARNER
Background As previously discussed, UP-LVSM operates in the challenging unposed setting. where ground-truth poses Py are not provided. Target poses are fundamental conventional NVS (posed & posed-target settings), as they provide the explicit viewpoint conditioning required for the rendering process. This creates clear image-pose pairs for supervised learning. In absence of such ground-truth, the model must learn to infer latent poses in self-supervised manner, leveraging the implicit signal from multi-view imagery (i.e., images of the same scene serve as positive pairs). However, the key challenge lies in constraining the representational capacity of this learned latent pose. high-dimensional latent space risks severe information leakage, where the latent pose inadvertently encodes the target image itself rather than just the viewpoint. Conversely, low-dimensional space may lack the expressiveness required to guide fine-grained, pixel-accurate rendering.
Method We propose the Latent Plücker Learner to address this challenge using an autoencoder architecture that strategically manages information flow, as illustrated in Figure 6 (c). To prevent

---

information leakage from the target view, the learner first uses encoder to distill the image into a highly compact 7D latent pose token (translation x and quaternion Q) as detailed Figure 6 (d). This low-dimensional bottleneck constrains the latent space, making it unable retain specific image content. Conversely, to ensure this compact representation is expressive enough for rendering, the token is then analytically upsampled into fine-grained, pixel-level conditions by adapting the Plücker ray embedding (Plucker, 1865) to operate within learned latent space. This design provides rich, per-ray conditioning for the renderer while maintaining minimal of learnable pose parameters. By training this module with a shared latent space across scenes, our model effectively learns meaningful camera pose representation without any 3D supervision.
Plücker Ray Embedding Plücker ray embedding (Plucker, 1865) is an effective technique to embed camera pose information into pixel-aligned tokens. Given image IR H x3 the Plücker ray encodes its corresponding camera pose for each pixel P concat d, d) # where represents the camera center and d is the camera ray direction corresponding to the pixels.
4.3 EXPERIMENTS
Setup As method employs the DINOv2 tokenizer, the 256 256 resolution is incompatible with for patch size of 14. We therefore adopt the 224 x 224 resolution to align with DINOv2's native configuration. Note that to ensure strict fairness, baselines. Instead, we retrain all baseline methods from scratch using the exact same 224 x 224 resolution with patch size of 14 and the same training split as our UP-LVSM. All training follows the official implementations of each baseline. This experimental setup is also applied in the scalability experiments of Section 3.
do use official checkpoints of
NVS Performance We follow Ye et al. (2025) to evaluate NVS performance. Qualitative and quantitative results the RealEstate 10K dataset (Zhou et al., 2018) are shown in Figure 7 and Table 5, respectively. Despite trained without any 3D supervision, our UP-LVSM even outperforms previous pose-dependent methods, which have access the dataset-provided poses. These results demonstrate the effectiveness of our proposed framework, validating the feasibility of scaling 2D-only learning frameworks to unlock spatial reasoning without explicit scene structure or pose annotations.
More results are illustrated in Appendix F.
Inputs
MVSplat
LVSM
NoPoSplat
PT-LVSM
UP-LVSM
Reference
todf
Figure 7: Qualitative View Synthesis Comparisons.
8

---

Method
Input Pose Pr
PSNR
Large Overlap
SSIM
LPIPS
PSNK
Medium Overlap
S8IM
LPIPS
PSNK
Small Ovedlap
SSIM 1
LPIPS
PSNR
Pall Dval. SSIM
LPIPS
PisetNeRR'
20:94
0
0.517
20.38
0.559
11.40
19.27
0.535
0.565
20.33
0.572
0,549
PisziSpla
26.18
0.579
0.115
23.61
0.521
11:622
21.22
0.352
0.225
25.51
0.867
0.126
MVSplat
27.32
0.559
0.112
23.57
0.319
11:665
20.67
0.330
0.238
26.45
0.874
0.123
LVSM
28.58
0.587
0.108
25,00
0,530
0.149
22.71
0,765
0.202
27.50
0.574
0,317
NoPoSplut
25.84
0.554
1.133
23.67
0.505
11:17
21.58
0.550
0.231
25.46
0.854
0.137
Ous (PT (VSS)
26.47
0.529
0.130
34.27
0.775
11.17
22.03
0.720
0.224
26.00
0.825
0.135
Ous (UP LVSM)
29.51
0
0JMM
36.93
0.882
0.132
24.54
0.796
0.374
28.82
0
0,104
Table 5: Quantitative Comparisons on RealEstate 10K (Zhou et al., 2018). Following Ye et al. (2025),
we conduct evaluations across different overlap levels. Our UP-LVSM outperforms pose-dependent approaches, particularly in challenging cases where input views share minimal overlap.
Method
Trstn->Test
PSNR
SSIM
LPIPS.
Model
Source Target Pose Pr
SSIM
LPIPS
IVSM
ACID -CTD
27.03
0.301
(a) annotations
26,00
0.525
UP-IVSM
ACID-ACID
27.21
0.787
0.194
UP-IVSM
(b) Pose Estimator in Segjadi et al. (2023)
20.92
0.521
0.558
UP-LYSM
KEJOK -ACI
27.33
0.792
0.182
(c) Latent Plücker Learner
28.82
0.891
0.104
Table 6: Zero-Shot Generalization.
Table 7: Ablation Studies on the Latent Plücker Learner.
Zero-Shot Generalization We evaluate the zero-shot generalization of UP-LVSM by training on the RealEEstate 10K dataset and testing on the unseen ACID dataset (Liu et al., 2021), denoted as REIOK-ACIID. As demonstrated in Table 6, this zero-shot generalization even outperforms methods trained directly on ACID, highlighting the performance gains from the larger quantity of data in RealEstate 10K (66K scenes) over ACID (13K scenes), which reinforces our findings.
Ablation Studies To validate the effectiveness of
our Latent Plücker Learner, conduct ablation stud- ies by contrasting different source of target pose Py information during UP-LVSM's training, illus-
Input Views
Scene
Synthesized Target View
Latent
Encoder
Decoder
trated in Figure A. The choice of SfM annotations results in fallback to the gosed-target setting and the inaccuracies of SIM lead to a performance degrada- tion, as demonstrated in Table (a). We also replace the Latent Plücker Learner with the pose estimator from Sajjadi et al. (2023), which uses a key-value querying mechanism and masking strategy to encour- age learning a meaningful latent pose space. While effective in smaller data regimes, this design becomes unstable at scale and yields suboptimal results, as demonstrated in Table 7 (b). In contrast, our Latent Plücker Learner leverages fine-grained Plücker em- beddings together with bottlenecked architecture, naturally avoiding information leakage and delivering substantially better rendering quality.
(n) SIM Tools
COLMAP
Pose Information
GT Target Views
(b) Sjjadi
Maak
Figure A. Ablation Studies on the Latent Plücker Learner. (a) Directly using explicit poses annotated by SfM tools to provide pose information of target views. (b) Replacing the Latent Plücker Learner with special pose estimator network (Sajjadi et al., 2023).
4.4
MORE INVESTIGATION
Reference View
Source View
Artn, Weights
DINOv2 Sim.
Medel
(f 1811
18 m0 9
9)
1 60" o: 180"
CLIP (Radfoed er al. 2021)
MAE (3e 2022)
3.5
(Oguub 2003)
27.5
K.0
UP LVSM (Qus)
25.4
Table Correspondence Estimation Accuracy for 3D Awareness Probing (El Banani et al., 2024).
Model
PSNR@
Al 1
LPIPS@AII 1
DUSER (Wang al., 2024)
19.28
0.630
0.391
LVSM (lin et al., 2025)
18.73
0.590
0.415
UPP-VSMM (Churs)
18.11
0.601
0.409
Table 9: GTA Metrics for 3D Awareness Probing (Chen et al., 2025).
Figure 8: Attention Weight Visualization.
Probing 3D Awareness As our UP-LVSM is a fully implicit approach to NVS which forgoes explicit 3D knowledge, it is critical to ascertain whether it implicitly learns spatial relationships To this end, we probe the model's 3D awareness following the methodologies proposed in (El Banani

---

et al., 2024; Chen et al., 2025). Our quantitative assessment, summarized in Table 8 & 9, demonstrates competitive 3D awareness. For a qualitative analysis, we visualize the attention weights of UP- LVSM between the marked patch (red) in the reference view and each patch in the source view, as illustrated in Figure 8. The visualized weights demonstrate noticeable correspondence awareness, even compared DINOv2 feature similarity. Both analyses confirm that UP-LVSM successfully develops a considerable degree of 3D awareness. See Appendix J.1 for detailed explanation.
Probing Pose Accuracy We further evaluate the accuracy of the latent poses produced by our Latent Plücker Learner by training simple 2-layer MLP to map the latent poses to SE (3) space, supervised with pose annotations from RealEstate 10K. We report the accuracy of mapped poses in Table 10, where our method achieves pose accuracy comparable to the concurrent Rayzer (Jiang et al., 2025). Qualitatively, we use t-SNE (Van der Maaten & Hinton, 2008) to visualize difference between the latent space and the real-world SE (3) space in Figure 9, demonstrating that the two spaces can align through simple twisted domain transformation. See Appendix J.2 for details. Both results indicate that our model effectively learns the underlying 3D pose geometry using only 2D supervision.
Ground Truth Space
Latent Space
Model
Trans 60.1
Trans, Trans.e0.21
Trans, e0.3
Rayzer (Jiang al., 2025) UP-LVSM (Ours)
84.2
92.8
Gomal Erenk Saae
Spens
71.3
89.2
96.4
ASNNE ASNE Die
L-NNE
Model
Rot, @ 10" 1
Rot. Not.6:20"
Rot 30"
Rayzer (Jiang al., 2025)
100
UP-LVSM (Ours)
98.4
99.6
99.8
1-NNK Din
6-SNE Din
Figure 9: Visualization of the Learned La-
Table 10: Accuracy of Mapping Latent Pose to SE (3).
tent Space of Camera Poses.
Camera Control While UP-LVSM em- ploys the Latent Plücker Learner to eliminate pose annotation dependence for improved scal- ability, the implicit nature of its estimated la- tent camera poses hinders the explicit control of the rendering view. However, it is easy to extend UP-LVSM for camera-controlllble ren- dering in real-world scenarios by additionally learning a linear mapping from the SE (3) space to the learned latent space, as our La- tent Plücker Learner effectively encourages the model to learn a meaningful manifold (Ta- ble 10 and Figure 9). After a regular training stage, we fine-tune UP-LVSM with this lin- ear mapper using a small subset of posed data (1202 scenes in RealEstate10KK, 1.8% of the training dataset). Table A demonstrates little performance degradation of this finetuning stage. We further visualize the linearly mapped poses in Figure B, providing evidence that our design effectively supports transformation between explicit SE (3) cameras and the latent ones, allowing human-specified camera sequences to be directly mapped to latent Plücker representations for controllability. More details are provided in Appendix D.
Input Camera Sequence
Mapped in Latent Space
Figure 10: The linear transformation effectively
maps input camera sequence into latent space, fa- cilitating explicit camera control.
Model
PSNR:
SSIMI
IPIPS).
UP-LVSM (w. Latent Plücker Learne)
28. 82
0.891
0.104
UP-LVSM (w. Lineur Mapper)
28.41
0.886
Figure 11: NVS Performance with Mapped Poses.
5 CONCLUSION
In this work, we revisit the field of feed-forward novel view synthesis through the lens of 3D knowledge dependency. We first highlight the need to reduce dependence on 3D knowledge by analyzing the scaling behaviors of offtt-----hh-art methods, revealing a key trend: methods with less 3D dependence accelerate dramatically as data scales-thh the less you depend, the more you learn. Building on this, we propose a novel NVS framework bypassing the need of explicit scene structure and camera pose annotations. By eliminating these 3D knowledge dependencies, our method leverages data scaling to foster implicit 3D awareness from 2D imagery, even outperforming the 3D knowledge-driven counterparts, thereby validating the effectiveness of our data-centric paradigm.
10

---

REPRODUCIBILITY STATEMENT
The scalability analysis in Section 3, the training of our proposed method (UP-LVSM), and the experimental results in Section 4.3 & 4.4, all reproducible. Details of the scalability analysis provided in Section 3 and Appendix I.2 to ensure reproducibility. Details of training UP-LVSM are provided in Appendix I.I with the architecture illustrated in Section 4.1. Details of investigation in Section 4.4 are provided in Appendix J. Furthermore, we will release code for the training and evaluation of our UP-LVSM to facilitate future research for the community.
ACKNOWLEDGEMENT
This work is supported by the projects of Beijing Science and Technology Program (2251100008125028),
REFERENCES
Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 5470-5479 (079,,4999-20 2022.
Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterport3d: Learning from rgb-d data in indoor environments. arXiv preprint arXiv:1709.061556 2017.
Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information -ichi 3d model repository. arXiv preprint arXiv:1512.03012. 2015. 2015.
David Charatan, Sizhe Li, Andrea Tagliasacchi, and Vincent Sitzmann. pixelsplat: 3d gaussian splats from image pairs for scalable generalizable 3d reconstruction. In CVPR, 2024.
Yue Chen, Xingyu Chen, Anpei Chen, Gerard Pons-Moll, and Yuliang Xiu. Feat2gs: Probing visual foundation models with gaussian splatting. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 6348-668 06666,66,44 2025.
Yuedong Chen, Haofei Xu, Chuanxia Zheng, Bohan Zhuang, Mare Pollefeys, Andreas Geiger, Tat-Jen Cham, and Jianfei Cai. Mvsplat: Efficient 3d gaussian splatting from sparse multi-view images. arXiv preprint arXiv:2403. /4627, 2024.
Jasmine Collins, Shubham Goel, Kenan Deng, Achleshwar Luthra, Leon Xu, Erhan Gundogdu, Xi Zhang, Tomas Yago Vicente, Thomas Dideriksen, Himanshu Arora, et al. Abo: Dataset and benchmarks for real-world 3d object understanding. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 21 21126-2113 2136, 2022.
Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5828-5839, 2017.
Matt Deitke, Ruoshi Liu, Matthew Wallingford, Huong Ngo, Oscar Michel, Aditya Kusupati, Alan Fan, Christian Laforte, Vikram Voleti, Samir Yitzhak Gadre, et al. Objaverse-xl: A universe of 10m+ 3d objects. Advances in Neural Information Processing Systems, 36:35799-35813, 2023.
Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. image is worth 16x 16 words: Transformers for image recognition at scale. arXiv preprint P2Xi::2010111920 2020.
Laura Downs, Anthony Francis, Nate Koenig, Brandon Kinman, Ryan Hickman, Krista Reymann, Thomas B McHugh, and Vincent Vanhoucke. Google scanned objects: A high-quality dataset of 3d scanned household items. 2022 International Conference on Robotics and Automation (ICRA), pp. 2553-2560 IEFF, 2022.
11

---

Yilun Du, Cameron Smith, Ayush Tewari, and Vincent Sitzmann. Learning to render novel views from bide-baseline stereo pairs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4970-4980, 2023.
Mohamed El Banani, Amit Raj, Kevis-Kokitsi Maninis, Abhishek Kar, Yuanzhen Li, Michael Rubinstein, Deqing Sun, Leonidas Guibas, Justin Johnson, and Varun Jampani. Probing the 3D Awareness of Visual Foundation Models, In CVPR, 2024.
Zhiwen Fan, Panwang Pan, Peihao Wang, Yifan Jiang, Hanwen Jiang, Dejia Xu, Zehao Zhu, Dilin Wang, and Zhangyang Wang. Pose-free generalizable rendering transformer. arXiv preprint arXiv:2310.03704, 2023.
Zhiwen Fan, Wenyan Cong, Kairun Wen, Kevin Wang, Jian Zhang, Xinghao Ding, Danfei Xu, Boris Ivanovic, Marco Pavone, Georgios Pavlakos, et al. Instantsplat: Unbounded sparse-view pose-free gaussian splatting in 40 seconds. arXiv preprint arXiv:2403.20300 2(3):4, 2024.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 16000-16009, 2022.
Alex Henry, Prudhvi Raj Dachapally, Shubham Pawar, and Yuxuan Chen. Query-key normalization for transformers. arXiv preprint 10XXi:2010004245 2020.
Ranran Huang and Krystian Mikolajczyk. No pose at all: Self-supervised pose-free 3d gaussian splatting from sparse views. arXiv preprint arXiv:2508.01177 2025.
Hanwen Jiang, Hao Tan, Peng Wang, Haian Jin, Yue Zhao, Sai Bi, Kai Zhang, Fujun Luan, Kalyan Sunkavalli, Qixing Huang, and Georgios Pavlakos. Rayzer: A self-supervised large view synthesis model, 2025.
Haian Jin, Hanwen Jiang, Hao Tan, Kai Zhang, Sai Bi, Tianyuan Zhang, Fujun Luan, Noah Snavely, and Zexiang Xu. Lysm: A large view synthesis model with minimal 3d inductive bias. In The Thirteenth International Conference on Learning Representations, 2025. URL https: wwe.iiwweee..eeee..me // //ppppervieww.o// net. httr:/foummmmii wtt.ruuttt----PPttttetoo en.
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part I1 14, pp. 694-711. Springer, 2016.
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics, 42(4), July 2023. URL https: : / repo- sam. inria..... fr/ fungraph/ /dd gaadd-gussiii an splatt angg/t/
Zhengqi Li and Noah Snavely. Megadepth: Learning single-view depth prediction from internet photos. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2041-2050 2018
Zhengqin Li, Ting- Wei Yu. Shen Sang. Sarah Wang. Meng Song, Yuhan Liu, Yu- Yu-Ying Ych, Rui Zhu, Nitesh Gundavarapu, Jia Shi, et al. Openrooms: An open framework for photorcalistic indoor scene datasets. In Proceedings of the IEFE/CVF conference computer vision and pattern recognition, pp. 7190-7199, 2021.
Lu Ling, Yichen Sheng, Zhi Tu, Wentian Zhao, Cheng Xin, Kun Wan, Lantao Yu, Qianyu Guo, Zixun Yu, Yawen Lu, et al. D13dy-10k: A large-scale scene dataset for deep learning-based 3d vision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 22160-22169, 2024.
Andrew Liu, Richard Tucker, Varun Jampani, Ameesh Makadia, Noah Snavely, and Angjoo Kanazawa. Infinite nature: Perpetual view generation of natural scenes from single image. In Proceedings of the IEEE/CVF International Conference on Computer Vision pp. 14458-14467, 2021.
12

---

Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing scenes as neural radiance fields for view synthesis. The European Conference Computer Vision (ECCV), 2020.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprini urXiv:2304.07793, 2023.
Julius Plucker. Xvii. on a new geometry of space. Philosophical Transactions of the Royal Society of London, 155:725-791, 1865. doi: 00.1098/rst..1865.000 URL https:// http://oiaabbb..oaaa.ee wtt.ss..................1 0017.
royalsocietypabll/ witbhiiihhnnggg
Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PmLR, 2021.
Jeremy Reizenstein, Roman Shapovalov, Philipp Henzler, Luca Sbordone, Patrick Labatut, and David Novotny. Common objects in 3d: Large-scale learning and evaluation of real-life 3d category reconstruction. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 10901-10911, 2021.
Mike Roberts, Jason Ramapuram, Anurag Ranjan, Atulit Kumar, Miguel Angel Bautista, Nathan Paczan, Russ Webb, and Joshua M Susskind. Hypersim: A photorealistic synthetic dataset for holistic indoor scene understanding. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 10912-10922, 2021.
Robin Rombach, Patrick Esser, and Björn Ommer. Geometry-free view synthesis: Transformers and no 3d priors. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 14356-14366 14366, 2021.
Mehdi S. M. Sajjadi, Henning Meyer, Etienne Pot, Urs Bergmann, Klaus Greff, Noha Radwan, Suhani Vora, Mario Lucie, Daniel Duckworth, Alexey Dosovitskiy, Jakob Uszkoreit, Thomas Funkhouser, and Andrea Tagliasacchi. Scene Representation Transformer: Geometry-Free Novel View Synthesis Through Set-Latent Scene Representations. CVPR, 2022. URL https:// srt-paper. git............g io/.
Mehdi S. M. Sajjadi, Aravindh Mahendran, Thomas Kipf, Etienne Pot, Daniel Duckworth, Mario Lučié, and Klauss Greff. RUST: Latent Neural Scene Representations from Unposed Imagery. CVPR, 2023.
Johannes L Schonberger and Jan-Michael Frahm. Structurefroommmotion revisited. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4104-4113, 2016.
Brandon Smart, Chuanxia Zheng, Iro Laina, and Victor Adrian Prisacariu. Splatt3r: Zero-shot gaussian splatting from uncalibrated image pairs. arXiv preprint arXiv:2408.1300 13912, 2024.
Mohammed Suhail, Carlos Esteves, Leonid Sigal, and Ameesh Makadia. Generalizable patch-based neural rendering. In European Conference Computer Vision, pp. 156-174. Springer, 2022.
Joseph Tung, Gene Chou, Ruojin Cai, Guandao Yang, Kai Zhang, Gordon Wetzstein, Bharath Hariharan, and Noah Snavely. Megascenes: Scene-level view synthesis at scale. In ECCV, 2024.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones. Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
Jianyuan Wang, Minghao Chen, Nikita Karaev, Andrea Vedaldi, Christian Rupprecht, and David Novotny. Vggt: Visual geometry grounded transformer. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2025.
13

---

Qianqian Wang, Zhicheng Wang, Kyle Genova, Pratul P Srinivasan, Howard Zhou, Jonathan T Barron, Ricardo Martin- Bruaala, Noah Snavely, and Thomas Funkhouser. Ibrne:: Learning multi-view image-based rendering. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 4690-4699, 2021a.
Shuzhe Wang, Vincent Leroy, Yohann Cabon, Boris Chidlovskii, and Jerome Revaud. Dust3r: Geometric 3d vision made easy. In CVPR, 2024.
Zirui Wang. Shangzhe Wu, Weidi Xie, Min Chen, and Victor Adrian Prisacariu. NeRF--: Neural radiance fields without known camera parameters. arXiv preprint arXiv:2102.07064 2021b.
Christopher Wewer, Kevin Raj. Eddy llg, Bernt Schiele, and Jan Eric Lenssen. latentsplat: Autoen- coding variational gaussians for fast generalizable 3d reconstruction. In European Conference on Computer Vision, pp. 473- Springer, 2024.
Changchang Wu et al. Visualsfm: A visual structure from motion system, 2011.
Haofei Xu, Anpei Chen, Yuedong Chen, Christos Sakaridis, Yulun Zhang, Mare Pollefeys, Andreas Geiger, and Fisher Yu. Murf: multi-baseline radiance fields. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 20041-20054 2024.
Yao Yao, Zixin Luo, Shiwei Li, Jingyang Zhang, Yufan Ren, Lei Zhou, Tian Fang, and Long Quan. Blendedmvs: A large-scall scale dataset for generalized multi-view stereo networks. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 1790-1799, 2020.
Botao Ye, Sifei Liu, Haofei Xu, Xueting Li, Marc Pollefeys, Ming-Hsuan Yang, and Songyou Peng. No pose, no problem: Surprisingly simple 3d gaussian splats from sparse unposed images. In The Thirteenth International Conference on Learning Representations, 2025. URL https:
//openrevi ew net/ forum? d== 4o fakekdf.
Chandan Yeshwanth, Yueh-Cheng Liu, Matthias Nießner, and Angela Dai. Scannet++: A high- fidelity dataset of 3d indoor scenes. In Proceedings of the IEEE/CVF International Conference Computer Vision pp. 12 12-22, 2023.
Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixelNeRF: Neural radiance fields from one few images. In CVPR, 2021.
Xianggang Yu, Mutian Xu, Yidan Zhang, Haolin Liu, Chongjie Ye, Yushuang Wu, Zizheng Yan, Chenming Zhu, Zhangyang Xiong, Tianyou Liang, et al. Mvimgnet: A large-scale dataset of multi-view images. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9150-9161, 2023.
Zehao Yu, Anpei Chen, Binbin Huang, Torsten Sattler, and Andreas Geiger. Mip-splatting: Alias-free 3d gaussian splatting. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 19447-19456. 2024.
Kai Zhang, Sai Bi, Hao Tan, Yuanbo Xiangli, Nanxuan Zhao, Kalyan Sunkavalli, and Zexiang Xu. Gs-Irm: Large reconstruction model for 3d gaussian splatting. European Conference on Computer Vision, 2024.
Shangzhan Zhang, Jianyuan Wang, Yinghao Xu, Nan Xue, Christian Rupprecht, Xiaowei Zhou, Yujun Shen, and Gordon Wetzstein. Flare: Ferd-forward geometry, appearance and camera estimation from uncalibrated sparse views. arXiv preprint arXiv:2502.12.:. 2025.
Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnification: learning view synthesis using multiplane images. ACM Trans. Graph. 37(4), July 2018. ISSN 0730-0301. doi: 10.1145/1195517.2201333 URL https:://o/.t org 104.11141/90 3201323.
14

---

A PROBLEM SETTING DETAILS
In this section, we provide additional details regarding the problem settings described in Section 3 of the main paper, where we present our analysis to support the hypothesis: The less you depend, the more you learn. Clarifying these settings is essential, as analysis relies heavily on experimental results and comparative evaluations across state-of-the-al methods. While Figure I in the main paper illustrates the three problem settings categorize, and Table I lists the corresponding methods, the definitions of these settings-Sui as their inputs, outputs, and evaluation protocols- only briefly discussed for clarity. In this appendix, elaborate these aspects, with particular attention to distinctions between training, evaluation, and real-world deployment.
A.1 POSED SETTING
The posed setting is the most straightforward scenario, assuming that pose information is always available. During both training and evaluation, the pose the target view PT is provided by the dataset. In real-world applications, however, Py determined by a user query, reflecting natural camera control behavior.
critical aspect of this setting that the view synthesis problem is inherently pose-equivalent: any given instance with poses Pz, Py is functionally equivalent to one with poses H.Pr, P H' is an arbitrary transformation in SE(3). To ensure pose-equivalence during training, common practice is to apply camera pose normalization. This procedure treats the first input view as the reference, designating its pose as canonical and transforming all other camera poses from world coordinates into the canonical frame.
H'Pr, where
A.2 POSED-TARGET SETTING
The posed-target setting introduces subtle but important distinction. Unlike the posed setting, does not require the input view poses Pr for scene modeling, but it does require the target pose PT for view synthesis. Despite the pose-equivalence normalization discussed earlier, this setting inherently introduces auabiguity-specifieeee
how model reason about the spatial relationship between a posed target view and unposed input views?
To address this challenge, existing posed-target methods (Wang et al., 2021b; Fan et al., 2024; Ye et al., 2025) typically employ an evaluation- time pose alignment trick to ensure fair comparison on benchmarks such RealEstatelOK (Zhou al., 2018). For example, in NoPoSplat (Ye et al., 2025), the model first estimates a 3D Gaussian Splatting (3DGS) representation in a canonical space from two unposed input views. This reconstructed 3DGS is then frozen, and the target camera pose is optimized at inference time so that the synthesized target view aligns as closely as possible with the ground truth image. It is important to note that this procedure is used solely for benchmark evaluation; in real- world applications, the target view pose is typically determined directly via user input, making such optimization unnecessary.
A.3 UNPOSED SETTING
The umposed setting presents the most challenging scenario. Unlike the posed-targe setting, where the target view pose Py known and can guide view synthesis, the unposed setting assumes no pose information is available even during training, leaving explicit pose-based viewpoint control impossible.
To overcome this limitation, the early method, RUST (Sajjadi et al., 2023). employs strategy similar to evaluation-time alignment, but adapted for training. Specifically, RUST introduces an implicit alignment mechanism by allowing the model to observe the ground truth target image T and learn to estimate its pose in a self-supervised manner, depicted in Figure I of the main paper. Following this, our UP-LVSM framework introduces the Latent Plücker Learner, which estimates latent Plücker coordinates from the target view T and the scene latent S. This design enables the model to infer the viewpoint from which to render, facilitating implicit alignment between the synthesized and ground truth target views for effective supervision. The concurrent work, Rayzer (Jiang al., 2025), adopts a similar strategy by inferring the spatial relationship from multiple input and target views to predict each view's s corresponding Plücker maps for view synthesis.
15

---

While such alignment techniques are effective during training and evaluation, they are unsuitable for real-world deployment, where the ground truth target view is unavailable. Unlike the posed and posed-target settings, where the target pose can be explicitly determined via numan-specified camera sequences, the unposed setting relies on a learned, implicit pose space. This fundamentally limits direct, interpretable control and thereby weakens its practical applicability. To address this, our work prioritizes explicit camera control over implicit solutions (Sajaadi et al., 2023) or relative ones (Jiang al., 2025), and proposes an effective strategy, which we detail in Appendix D.
B MORE RELATED WORK
Feed-forward Novel View Synthesis Recent advancements in novel view synthesis using dense multi-view inputs have made significant progress (Mildenhall et al., 2020; Barron al., 2022; Kerbl et al., 2023; Yu et al., 2024), but their reliance on explicit geometric cues limits applicability to un- structured observations. In contrast, generalizable methods aim to bypass computationally expensive per-scene optimization, typically by combining neural networks with 3D representations (Yu et al., 2021; Wang et al., 2021a; Du et al., 2023; Charatan et al., 2024; Chen et al., 2024; Xu et al., 2024; Zhang et al., 2024; Ye et al., 2025; Huang & Mikolajczyk, 2025). Another paradigm (Dosovitskiy et al., 2020; Rombach et al., 2021; Sajjadi et al., 2022; 2023; Suhail et al., 2022: Jin et al., 2025; Jiang al., 2025) explores geometry-frre solutions using feed-forward neural networks, with the recent methods (Jin et al., 2025; Jiang et al., 2025) achieving impressive results without explicit 3D bias.
Multi-view Imagery Dataset Learning based novel view synthesis approaches typically rely on large-scale datasets consisting of multi-view images and their corresponding camera parameters for training. Early datasets focused on object-level data (Chang et al., 2015; Reizenstein et al., 2021; Collins et al., 2022; Downs et al., 2022), while recent efforts (Yu et al., 2023; Deitke et al., 2023) have significantly expanded data scales. Meanwhile, several scene-level datasets (Dai et al., 2017; Chang et al., 2017; Li & Snavely, 2018: Yao et al., 2020; Li et al., 2021; Liu et al., 2021; Roberts et al., 2021; Yeshwanth et al., 2023; Ling et al., 2024; Tung et al., 2024) have been proposed to facilitate scene-level view synthesis. Among them, the RealEstate dataset (Zhou et al., 2018) has garnered significant attention due to its early release, open-source nature, and massive size, becoming widely used training set and benchmark for recent generalizable view synthesis methods (Yu et al., 2021; Charatan et al., 2024; Chen al., 2024; Ye et al., 2025; Jin al., 2025).
Pose-free View Synthesis Despite efforts to reduce dependence on input camera poses during inference (Fan et al., 2023; Smart et al., 2024; Ye et al., 2025; Zhang et al., 2025), generalizable novel view synthesis methods typically depend on posed data for training supervision, with few tackling the elimination of pose annotations. Early work (Sajjadi et al., 2023) pioneered the really unposed setting, bypassing pose dependence even during training. However, their solution struggles with high-fidelity rendering, and the latent pose representation remains difficult to interpret, making direct camera pose control challenging. In contrast, our data- framework harnesses scalability and the Latent Plücker Learner design, achieving rendering quality comparable to methods requiring pose input or supervision (Ye et al., 2025; Jin et al., 2025).
C MORE DISCUSSIONS ABOUT CONCURRENT WORK
As summarized in Table there two concurrent works also aiming for the unposed setting (Rayzer (Jiang et al., 2025) and SPFSplat (Huang & Mikolajezyk, 2025)).
Rayzer While our design emphasizes camera control, the critical trend uncovered by our investiga- tion is also confirmed by Rayzer (Jiang et al., 2025), a concurrent work with their focus on multiple sparse views (N > 5 instead of our N = 2), achieving promising results at the unposed setting, which bypasses 3D supervision. Most of our main paper experiments exclude this approach due to difference in problem settings and the current lack of an official implementation. Despite this, in Table 11, we compare the results reported in its paper (Jiang et al., 2025) with our UP-LVSM performance on RealEstate 10K. It is important note that the Rayzer is trained with N 2 5 input views, which differs from our N = 2.
SPFSplat Another concurrent work, SPFSplat (Huang & Mikolajczyk, 2025), pushes the bias- driven approaches to the unposed setting by extending NoPoSplat (Ye al., 2025) with self-
16

---

supervised pose estimator. The training and evaluation settings of SPFSplat are consistent with ours, and we include its reported results in Table 11.
Predictive Value of Our Findings Notably, the predictive value of our core findings confirmed by the performance of these two methods. Rayzer, with its data-centric approach, significantly surpasses SPFSplat's bias-driven method in NVS metrics. We attribute this to SPFSplat's reliance on predefined 3D representation (3DGS) and tailored, handcrafted rendering functions "creates bottleneck at scale", as elaborated in Section 3. Furthermore. SPFSplat's 's performance (25.84 dB) bias-driven unposed method, it can
aligns with the trend predicted by our scalability findings: outperform bias-driven posed-targe target methods (e.g., NoPoSplat, 25.46 dB) but will be less competitive than the data-centric posed-target methods (e.g., PT-LVSM, 26.00 dB).
Method
Nasiber of Tepor Niews
PSNR
Large Dee
SSIM T
LPIPS
Modian Ovedap
SSIMT
LPIPS
PENR
Small Deadlp
55IM
PSNR
Full Evi. 5SIMT
LPIPS
SPE (Haass Rayart Dang ul., 3025)
2015)
29.38
0.999
0,111
25.70
0.553
0.151
23.18
0,200
25.84
0.852
0.151
N0A
N/A
N0A
NIA
NVA
N5A
N/A
N03
27
0.561
0.146
UP-LVSM (Dus)
29.51
0,901
8:998
26.93
0.82
0.132
34.54
0.7%6
0.174
29,0
0.891
0.104
Table II: Quantitative Comparisons on RealEstate 10K (Zhou et al., 2018). Following Ye et al. (2025),
we conduct evaluations across different overlap levels. Our UP-LVSM consistently outperforms the concurrent approaches.
D CAMERA CONTROL
Input Camera Sequence
Mapped in Latent Space
Motivation While UP-LVSM employs the Latent Plücker Learner to eliminate pose an- notation dependence for improved scalability, the implicit nature of its estimated latent cam- era poses hinders the explicit control of the rendering view. This might restrict its broad applicability. In this section, we demonstrate that it is easy to extend UP-LVSM for camera- controllable rendering in real-world scenarios by additionally learning mapping from the SE (3) space to the learned latent space. Re- garding camera intrinsics, we follow NoPoS- plat (Ye et al., 2025) in assuming a known set of intrinsics for simplicity, while it is also fea- sible to extend the Latent Plücker Learner to accommodate learnable intrinsic.
4
Figure 12: The linear transformation effectively maps input camera sequence into latent space, fa- cilitating explicit camera control.
Model
PSNR1
SSIM1
LPIPS|
UP-INSM (w. Latent Pltccee Leamer)
28.
0.891
0.104
LV-L (w. Linear Mapper)
28.41
0.886
0.110
Table 12: NVS Performance with Mapped Poses.
Implementation Specifically, we introduce a linear pose mapper by (A E R b RT). which maps a real-world camera pose vector Č concat( (xt, q) E IR. into its corresponding latent representation = concat(x, q) = AČ + b E RT. This latent camera pose is then used to generate the associated Plücker representation P. We fine-tune UP-LVSM with this linear mapper using a small subset of posed data (1202 scenes in ReallEstate 10K (Zhou et al., 2018), 1.8% of the training dataset). As described in Appendix A. we also apply camera pose normalization to reduce pose ambiguity. After fine-tuning, human- speeifiie camera sequences can be directly mapped to latent Plücker representations for view synthesis. This fine-tuning introduces negligible impact on rendering quality, as evidenced in Figure 13 and Table 12.
Discussion Earlier work (Sajjadi et al., 2023) also attempts to learn a latent pose, but typically fails to provide explicit camera control for view synthesis. primarily due to the uninterpretable nature of the learned pose latent. contrast, our Latent Plücker Learner effectively encourages the model to learn a meaningful manifold as latent space, enabling controllability. This is evidenced by our investigation in Figure of the main paper, where we visualize the learned latent Plücker space to validate that the model captures meaningful 3D pose space using only 2D supervision. We also visualize the linearly mapped poses in 12, providing evidence that our design effectively supports transformation between explicit SE (3) cameras and the latent ones.
The concurrent work, Rayzer (Jiang et al., 2025), also addresses camera control, but in a relative rather than explicit manner: it first estimates the camera poses of input views and then allows user-
17

---

w/o. Mapper
v. Mapper
Mapper
Mapper
Et
Figure 13: Ablation Study of Camera Mapper. The camera mapper fine-tuning introduces negligible
impact rendering quality.
specified interpolation between these poses. While effective, this strategy offers less flexibility than our approach. Additionally, it relies on multiple input views (N 2 5) to infer spatial relationship, while our method only requires two views (N 2).
E MORE DISCUSSION ABOUT SCALABILITT
In this section, we further explore the scalability analysis presented in Section the main paper. By conducting experiments the synthetic dataset, we address key question: Why does the availability of camera poses limil scalability?
Background As discussed in Section 3.2, theoretically, camera poses provide additional information that acts strong 3D cues. Thus, methods with access camera poses (e.g., LVSM (Jin et al., 2025)) should have a higher performance ceiling- one that significantly outperforms unposed methods like UP-LVSM. However, the curves in Figure 5 reveal unexpected trend: unposed method, UP-LVSM, achieves superior performance as the dataset scales. We attribute this unexpected trend to noise in pose annotations, Pose annotations in real-world datasets (Zhou al., 2018: Yao et al., 2020; Yeshwanth et al.. 2023; Ling et al., 2024) are typically generated by tools (Wu al., 2011 I; Schonberger & Frahm, 2016)-whiee which rely on geometric inductive biases-as and these tools often introduce noise and inconsistencies. Relying on such noisy poses during training constitutes indirect form of 3D knowledge dependence, creating critical scalability bottleneck.
(SfM)
0.27
PSNR
26
SSIM
0..81
0.18
0.09
UP-LYSM
18
0.57
le-2
0.00
Noise Scale
Noise Scale
Noise Scale
Figure 14: Performance of Methods Trained with Noisy Poses. Different levels of Gaussian noise (o² = 0.001, 0.11,0.1) were added to the rotation (in quaternion form) and translation components of the poses in training data. While UP-LVSM remains agnostic noisy poses, LVSM experiences significant degradation with increasing noise levels, exhibiting sensitivity even to small amounts of noise (0.001).
To validate this, we investigate the impact of pose noise on LVSM performance, with experimental results presented in Figure 14. These results show that even small amount of noise significantly affects LVSM's performance. This provides partial insight into the trend observed in Figure of the main paper, where unposed methods (e.g., UP-LVSM) eventually outperform
Investigation
18

---

those that depend poses (e.g., LVSM), even when the latter have access additional information. Moreover, we additionally train LVSM and UP-LVSM on the Objaverse dataset (Deitke et al., 2023), where the synthetic data will comprise pose noise. As demonstrated in Table 13, given ground truth camera poses, pose-dependent methods like LVSM begin to achieve superior performance as expected, providing strong evidence support claims. Moreover, under ground truth poses, LVSM and UP-LVSM exhibit similar scalability, indicating that it is unreliable pose annotations from SfM that limit scalability and lead lower performance ceiling.
frefree
parformanc
SSIMT /LPIPS)) 32K
APSNR ALPIP
128K
Avg. Gain
LVSM UP-LYSM
x
24.58 2158//881//00 17717 28.90/ 0.8T 106
29.63 1663/O00 898 0.006
$0.22 0.506 00087
1,77 /0,09 00428
19.66/0.12/0.008
3..8//.773/0.275
Table 13: Qualitative Results on Objaverse (Deitke et al., 2023).
F
MORE RESULTS
Input
Input 2
Render
Render
Render 3
Render
Render 5
Render 6
Figure 15: More Indoor Results of UP-LVSM.
Input
Input 2
Render
Render 2
Render 3
Render Render 5
Render 6
11
Figure 16: More Outdoor Results of UP-LVSM.
19

---

Input
Input 2
Render
Render
Render 3
Render
Render
Render 6
Figure 17: More Challenging Results of UP-LVSM (Little Overlapped Inputs).
Input
Input 2
Render
Render
Render 3
Render
Render 5
Render 6
Figure 18: More Challenging Results of UP-LVSM (Complicated Lighting Effects).
G MORE DISCUSSION ABOUT EXTRAPOLATION
All the view synthesis results in the main paper or the supplementary video are mainly interpolation between inputs views. This is because currently, most existing generalizable novel view synthesis NVS, but perform much worse for extrapolation, extrapolation is actually guessing what the whole scenes look like from partial observation. thereby indeed requiring generative modeling techniques (Wewer et al., 2024). Our methods suffer from the similar problem. Following extrapolation evaluation principles in previous work (Wewer
(NVS) methods are good at interpolation-stie
20

---

et al., 2024), we measure the extrapolation performance of existing methods. Table 14 compares the interpolation and extrapolation performance, demonstrating fundamental limitations of these generalizable methods.
Evaluation
Metric
PixeINeRF
PixelSplat
MVSplat
IVSM
NoPoSplat
Ours (PT-LVSM)
Ours (UP.IXSM)
PSNR
20.33
25.51
26.45
27.60
25.46
26.00
28.82
Interpolation
SSIM
0,572
0.867
0,874
0.874
0.854
0.825
0.891
LPIPS
0.549
0.126
0.123
0.137
0.135
0.104
PSNR
19.96
21.19
20.00
23.80
22.42
19.18
23.82
Extrapolation
SSIM
0.572
0.799
0.787
0.795
0.786
0.611
0.760
LPIPS
0,568
0.196
0,205
0.168
0,201
0.282
0.185
Table 14: Quantitative Comparisons. Existing methods perform much worse for extrapolation.
H MORE DISCUSSION ABOUT LIMITATION
Scalability Analysis In the main paper (Section 3.3), validate the key trend that reducing depen- dence on 3D knowledge enhances scalability on three real-world datasets, i.e., RealEstate 10K (Zhou al., 2018). DL3DV (Ling et al., 2024), and ACID (Liu et al., 2021). Of these, the largest is RealEstate which comprises 66K scenes. An open question remains: if data scale continues to grow beyond 66K scenes, can UP-LVSM sustain this performance trend? In other words, at what point will the performance curves in Figure 5 begin to saturate? Although we have observed this when validating the same trend on the synthetic Objaverse dataset (Deitke et al., 2023) (see Appendix E), we cannot conduct this experiment on larger real- woold datasets primarily because open-source NVS data remains limited. But we believe our UP-LVSM, benefiting from its pose- free training, can leverage large-scale video datasets to effectively extend training quantities, not limited NVS datasets, with primary challenge lying in that these datasets may lack strong camera motion and contain dynamic scenes, which would necessitate extensive data cleaning.
Methodology From a methodological standpoint, our UP-LVSM is still based on DINOv2 (Oquab et al., 2023) and inherits its limitation a relatively large patch size of 14 (compared LVSM's patch size of 8). The large patch size hinders fine-granularity image synthesis, leading to blurring artifacts in richly textured areas. As demonstrated in LVSM (Jin et al., 2025), smaller patch sizes lead to more competitive performance the cost of increased training time and higher CUDA memory usage. Striking a balance between performance and training cost, particularly through improvements to the network architecture, is important avenue for future exploration. Furthermore, we observe that increasing the amount of training data increases the risk of gradient explosion. While we mitigate this issue by adopting QKNorm (Henry et al., 2020) as LVSM, addressing this issue more effectively, particularly when scaling to larger datasets, will be crucial in future work.
I IMPLEMENTATION DETAILS
In this section, we provide implementation details for the methods compared in the main paper, including network architectures and training hyperparameters.
1.1 PT-LVSM & UP-LVSM
Following prior works (Sajjadi et al., 2022: Zhang et al., 2024; Jin al., 2025), the core archi- tecture of PT-LVSM is composed entirely of Transformer layers (Vaswani et al., 2017). Unlike previous implementations that train the encoder from scratch, we incorporate pretrained DINOv2 encoder (Oquab et al., 2023) to enhance training stability, particularly in the early stages, due to the absence of input pose annotations. The Transformer component adopts a decoder-only architecture, as in LVSM (Jin et al., 2025), comprising 24 layers. Each multi-heas attention layer includes 12 heads, each with 64-dimensional feature embeddings. The entire model, including both the Transformers and the DINOv2 encoder, is jointly optimized with learning rate of 0.0004.
Training is conducted on the full RealEstate 10K dataset (Zhou et al., 2018) using 8 NVIDIA A100 GPUs, with a batch size of 16 per GPU. Training for 100K steps takes approximately 60 hours. The loss function used is L MSF(T, 7) + XPerceptual(T, T), where À = 0.5, and Perceptual
21

---

denotes the perceptual loss introduced in (Johnson et al., 2016). For numerical stability, we follow LVSM (Jin et al., 2025) in employing QKNom (Henry et al., 2020) to mitigate the risk of gradient explosion.
The architecture of UP-LVSM differs slightly due to its encoder-decoder structure and latent Plücker representation. The encoder comprises DINOv2 backbone followed by 6 Transformer layers. The decoder consists of 14 Transformer layers. Additionally, the Latent Plücker Learner uses a DINOv2 encoder followed by a 4-layer Transformer. Note that our number of layers is equal LVSM and PT-LVSM for fair comparisons under the same level of parameter amount. All other training settings are consistent with those of PT-LVSM. After pretraining, UP-LVSM is fine-tuned to support camera- control-ont rendering using linear mapper. This fine-tuning is performed on the little subset of the RealEstate 10K dataset (1202 scenes, 1.8% of the full set) with a learning rate of 0.0001, requiring approximately 4 hours for 8K steps. During this stage, the ground truth target image is no longer provided; instead, the latent Plücker generated via the linear mapper from the ground truth larget pose, rather than from the Latent Plücker Learner.
For both PT-LVSM and UP-LVSM, we adopt camera pose normalization mentioned in Appendix A.1 to designate the pose of the first input view as canonical. However, this conflicts with the permutation- invariant nature of the Transformer, where the first input view should be recognized as special, but the Transformer inherently treats all inputs equally. To this end, assign special significance to the first view by adding a linearly projected canonical Plücker onto its DINOv2 image tokens. The ablation study demonstrates model cannot converge when trained without this trick, validating its effectiveness.
Lastly, due to the DINOv2 encoder's requirement that input dimensions be divisible by the patch size of 14, we rescale RealEstttee000 images to a resolution of 224 x 224, rather than the more commonly used 256 x 256. Following the approach in LVSM, we first train at low resolution (e.g.. 224 x 224), and then fine-tune on higher resolutions such as 518 x 518 to better adapt the model to high-resolution rendering. However, for the experiments reported the main paper, we standardize all evaluations to the 224 x 224 setting, including all baseline comparisons.
1.2 BASELINES
For all baseline methods evaluated in the main paper- (Yu al., 2021), Pixel- Splat (Charatan et al., 2024), MVSplat (Chen et al., 2024), LVSM (Jin et al., 2025), and NoPoSplat (Ye al., 2025)-we use the original training configurations provided in their respective official repos- itories. Since PixeINeRF does not provide official configurations for the RealEstate 10K dataset, we adapt its official code work with this dataset and successfully reproduce the performance reported in (Ye et al., 2025; Charatan et al., 2024). All other methods include officcal support for the RealEstate 10K dataset, requiring no modification aside from rescaling the input images to 224 x 224 (consistent with our setup described above).
1.3 OBJECT-LEVEL. TRAINING
Method
Perceptual Weight À Background
Rendering Views
Joint Training
Performance (PSNR1)
0.5
White
24, Sparse
No
Not Converged.
(b)
0.2
White
24. Sparse
No
Not Converged.
(c)
0.5
Gray
24, Sparse
No
Not Converged.
UP-LVSM
(d)
0.2
Gray
24. Sparse
No
23.38
(e)
0.2
Gray
128. Sequantial
No
24.95
0.2
Gray
128, Sequantial
Yes
24.37
(g)
0.2
Gray
24, Sparse
Yes
Not Converged.
Table 15: We use small amount (8K) of object-level data to investigate different training strategy, including varying background color for alpha compositing, different strategies of view rendering, and whether trained jointly with 8K scene-level data from RealEEstate1OK.
For object-level experiments (Tables 4 and 13) the Objaverse dataset (Deitke et al., 2023), we follow prior works (Jin et al., 2025) to train our method and the baselines under a different setting, as demonstrated in Table B (d). We first prepared 136K objects from Objaverse, with each rendered by Blender from 24 randomly sampled views. During training, we sample 4 input views and target views at each step, in contrast to the scene-level training's input views and 6 target views. To avoid
22

---

instability, perceptual loss weight of 0.2 instead of 0.5. The experimental results shown in Table B, where we discuss the influence different strategies on training stability.
J INVESTIGATION DETAILS
This section provides additional details for the analysis presented Section 4.4 the main paper, including attention weight analysis and latent Plücker space analysis.
J.1 ATTENTION WEIGHT ANALYSIS
We elaborate on Figure 8 of the main paper by visualizing patch-wise attention weights to illustrate that our model performs spatial reasoning and captures cross-view correspondences. Below, we describe the process in detail.
Consider an input image resolution of 224. Following the DINOv2 architecture, which uses patch size of 14, each input image I E IR Bx xx21x22xxx is converted into feature tokens D E XB 16x 768 where B is the batch size and N the number of input views. These tokens are R BBBx266NX76 and passed through the Transformer layers in the encoder.
then flattened to D'
We examine the attention weights W E BBBx256xxx23 x256N from the final Transformer layer, where each element represents the attention between pairs of input patches across all views. In the where N = and B = 1, the bottom-left 256 x 256 block of W, denoted as W E IR 252 256 corresponds the cross-view attention between the input views. Specifically, each element ji, indicates the attention weight from the i-th patch of the first view to the j-th patch of the second view.
While we use the viridis colormap to visualize the attention weights W in Figure 8 of the main paper, also visualize the DINOv2 token similarity verify that the model learns cross-view correspondence during training, rather than relying solely on the pretrained DINOv2 encoder's inherent capabilities. Specifically, given DINOv2 tokens Dú, D2 2556x768 768 from two views, we compute their cosine similarity along the feature dimension to obtain S 2566x256 For clearer S s-min(S) visualization, S is normalized as S' max(S) (minS)))ii(S)) min(S)
J.2 LATENT PLÜCKER ANALYSIS
This section provides a detailed explanation of the analysis presented in Figure of the main paper. Camera poses are represented 7-dimensional vectors, with the first dimensions encoding translation and the last 4 representing rotation as a quaternion. Given N pairs of camera poses Clateut E Nx7 and Creal R Nx7, concatenate them to form Call R2Nx7 We then apply I-SNE (Van der Maaten & Hinton, 2008) to project this 7-dimensional space into two dimensions, yielding Caatent: Creat R after splitting. Figure these 2D points visualized accordingly, with corresponding pairs colored identically.
K THE USE OF LARGE LANGUAGE MODELS (LLMs)
We used Large Language Models (LLMs) as writing assistants for this work. They are used only for text polishing, grammar checking, and sentence-level rephrasing. The core method development in this work does not involve LLMs as any important, original, or non-standard components.
23


Reviews:
[{"review_id": "0owutawuK3", "reviewer": "Reviewer_6twy", "summary": "The authors posit the principle “the less you depend, the more you learn”—arguing that reducing reliance on explicit 3D priors (like NeRF/3DGS representations or SfM-derived camera poses) improves scalability and generalization as training data scales. \n\nThis message extends upon LVSM and removes the training- and test- time pose requirements to make the framework even more data centric. The authors introduce a Latent Plücker Learner to infer camera poses in a self-supervised manner.\n\nThe paper presents empirical analysis across datasets (RealEstate10K, DL3DV, ACID, Objaverse) and compares UP-LVSM to bias-driven and pose-dependent methods (MVSplat, NoPoSplat, LVSM). Results show that UP-LVSM scales better with data and even surpasses 3D-supervised models in rendering fidelity and generalization.", "questions": "- in Tab. 8, it seems to me that DINOv2, though not explicitly trained on 3D tasks, seem to provide more accurate correspondence estimation than UP-LVSM which is trained on the 3D NVS task. I guess I find it a bit hard to understand. \n\n- in Tab. 5 and Fig. 7, how is the target pose provided to different methods? Did all the methods use the same ground-truth target pose? Or they consume poses estimated by UP-LVSM?\n\n- Line 1105 mentions that \"However, for the experiments reported in the main paper, we standardize all evaluations to the 224 ×224 setting, including all baseline comparisons.\" I wonder if this is fair to baselines if their provided checkpoint was trained only on 256x256.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The paper is easy to follow and the authors promise to release code for reproducibility.\n\n- The paper evaluates across diverse datasets and metrics, providing ablations, scalability curves, and qualitative examples. The performance gains are consistent and very decent (e.g., +1 PSNR improvements over LVSM on large-scale data).\n\n- Removing the requirement of training- and test- time pose annotations have huge potential of scaling to much larger-scale datasets; hence this work (along with RayZer) opens up a lot of new possibilities.", "weaknesses": "- UP-LVSM seems to have separately trained models on the RealEstate10k scene data and the Objaverse object data. It remains unclear to me if the latent plucker learning component can be trained on a mix of scene and Objaverse datasets. From the scalability perspective, it makes sense to have a method capable of ingesting all available data sources. \n\n- Tab. 4 is presented in a confusing way; it's unclear what baseline the performance gain is evaluated against. Moreover, I think it makes more sense to also include the absolute PSNR/SSIM/LPIPS values, rather than just providing the relative changes.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GsUIHEXkmY", "reviewer": "Reviewer_WSHE", "summary": "- The paper analyzes existing Novel View Synthesis (NVS) methods and discovers a core trend: methods that depend less on explicit 3D knowledge (poses, handcrafted 3D representations) benefit more from data scaling and eventually outperform 3D-biased approaches.\n\n- Based on this finding, the authors propose UP-LVSM, a feed-forward, fully data-centric NVS framework that learns implicit 3D structure directly from large-scale 2D images—without camera pose supervision or predefined 3D representations.\n\n- The method introduces a Latent Plücker Learner to infer camera geometry implicitly, enabling state-of-the-art performance in novel view synthesis from sparse, unposed images.", "questions": "Mentioned in the weaknesses", "limitations": "", "rating": 2, "confidence": 4, "soundness": 1, "presentation": 3, "contribution": 2, "strengths": "- This paper demonstrates that removing explicit 3D priors allows performance to scale significantly with data, outperforming pose-supervised 3D knowledge-driven methods.\n\n- This paper has a pose-free & explicit-3D-free pipeline, a fully feed-forward transformer architecture that works without SfM poses, NeRF/3DGS priors, or handcrafted 3D structures—simplifying training and deployment.\n\n- Extensive experiments show state-of-the-art results and confirm the central hypothesis, providing both conceptual insight and practical contribution to scalable NVS.", "weaknesses": "- I have watched the supplementary video. (1) The zoom-in and zoom-out distance is too short. (2) Also, compared to existing models, the method seems to only improve visual quality.\n\n- For datasets where GT is provided, the resolution appears to be very low. I am curious how the method performs on higher-resolution datasets. (If this dataset is the best available choice, I expect the authors to justify that in the rebuttal.)\n\n- The process of simply combining a Transformer with DINOv2 raises concerns regarding novelty. The authors should demonstrate where the novelty lies in this work. What is the novelty contribution of the Latent Plücker Learner?\n\n- The ablation study section is not intuitive. Merely describing components with text makes it difficult to understand what was included or excluded in each ablation setting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pdfi1IQkKe", "reviewer": "Reviewer_n5sM", "summary": "This paper investigates the trade-off between 3D bias-driven and data-centric design philosophies for Novel View Synthesis (NVS). The central hypothesis is \"the less you depend, the more you learn\": methods with weaker 3D dependencies exhibit superior scalability, accelerating in performance as training data increases.\n\nThe paper tests this hypothesis by conducting a systematic scalability analysis comparing existing methods (MVSplat, LVSM, NoPoSplat) on training subsets of varying sizes. They find that data-centric methods show greater performance gains with more data.\n\nBuilding on this insight, the paper proposes UP-LVSM, a novel, data-centric framework that operates in a fully unposed setting. This method learns to synthesize novel views from sparse images without any camera pose information or explicit 3D representations during training. The core technical contribution is the Latent Plücker Learner, a component that learns a latent pose space in a self-supervised manner, enabling viewpoint conditioning without ground-truth poses.\n\nExperiments show that UP-LVSM, when trained on a large dataset (66K scenes), achieves state-of-the-art performance, even outperforming methods that require ground-truth pose annotations. The paper argues that this validates their hypothesis and demonstrates that noisy 3D knowledge (such as SfM poses) can be a \"performance bottleneck\" at scale.", "questions": "Do you have a sense of the upper bound? For instance, how does UP-LVSM (trained on 66K scenes) compare to a per-scene optimized 3DGS (a different kind of \"upper bound\")? \n\nIn Table 8, the 3D correspondence probing shows UP-LVSM (31.9) performing slightly worse than the off-the-shelf DINOv2 (36.8) at 0-15°. This is counterintuitive, as your model's encoder is fine-tuned on this 3D-aware task. Do you have an explanation for why the specialized model would be worse than the general-purpose one?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1. The paper's core finding—that at scale, it's better to learn 3D from 2D data than to rely on noisy, explicit 3D knowledge (like SfM) —is a significant and impactful claim, which is well-supported by the UP-LVSM results.\n\n2. The analysis in Section 3, which isolates the effects of 3D inductive bias and pose annotation dependence, is a valuable contribution in its own right. The use of dataset subsets to show performance trends (Figures 2, 4, and 5)  is very well done.\n\n3. The proposed UP-LVSM achieves SOTA performance on RealEstate10K, impressively outperforming LVSM (28.82 vs 27.60 PSNR)  despite LVSM having access to ground-truth input poses. This is a very strong result that validates the paper's hypothesis.\n\n4. The Latent Plücker Learner is a novel and well-designed component for learning a latent pose space without supervision. The design thoughtfully considers and addresses the risk of information leakage from the target view.\n\n5. The appendix is not an afterthought but contains crucial, high-quality analysis. Appendix E shows that LVSM are highly sensitive to pose noise, whereas UP-LVSMs are immune. Appendix D solves a key practical limitation of unposed methods.\n\n6. The paper is also exceptionally well written, structured and the figures and tables are all very informative.", "weaknesses": "1. The paper's claim to remove \"3D inductive biases\" and operate without \"any 3D knowledge\"  is inaccurate. The Plücker ray embedding, which is foundational to the Latent Plücker Learner, is a strong 3D geometric prior. It encodes a line in 3D space. The claim should be more precise, e.g., \"without explicit 3D scene representations (like meshes or 3DGS) or camera pose annotations. \n\n2. A major practical drawback of unposed methods is the inability to control the camera for rendering. The paper presents a very simple and effective solution (fine-tuning a linear mapper) in Appendix D. For me personally, this is a crucial insight for making the method practical, yet it is absent from the main paper, potentially leaving readers with the impression that UP-LVSM is not controllable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "K5zspbzVrh", "reviewer": "Reviewer_KQEM", "summary": "The authors study generalizable novel view synthesis (NVS) from sparse, unposed 2D images and show that methods with less explicit 3D knowledge improve faster with larger data; they propose a framework that eliminates 3D inductive bias and pose annotations and learns implicit 3D awareness from 2D images, reporting photorealistic and 3D-consistent novel views comparable to methods that use posed inputs", "questions": "1. At what dataset size and diversity does the “less 3D knowledge” model overtake 3D-driven counterparts? Is there a measurable crossover point per dataset type?\n\n2. Which camera motions, occlusion patterns, or scene topologies cause the model to produce inconsistent geometry or view synthesis artifacts?\n\n3. How does the method perform on metric 3D/geometry benchmarks (e.g., depth/pose consistency) compared with explicit 3D methods?\n\n4. Does training on large web-scale data produce robust synthesis on specialized domains or is fine-tuning with some 3D bias still necessary?\n\n5. Can hybrid approaches (weak 3D priors, self-supervised pose signals) such as HawkI reduce data needs while retaining benefits of the minimalist design?\n\n6. What are the training compute, memory, and inference-time costs relative to NeRF-style or other 3D-aware baselines?\n\n7. Which minimal inductive elements (if any) materially help (positional encoding, multi-view consistency losses, rendering modules) and which are redundant?\n\n8. Does reliance on large web data introduce unwanted biases in scene content, lighting, or demographic representation that affect downstream uses?\n\n9. If we train on large in-the-wild data and evaluate on narrow-domain datasets to measure generalization and need for adaptation, how does it work?\n\n10. It will be good to add experiments with light-weight 3D cues (approximate poses, sparse depth) to quantify trade-offs between inductive bias and data size.\n\n11. It will be good to publish FLOPs, GPU hours, and inference latency to clarify practical deployment feasibility", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Clear empirical trend discovery, identifies and quantifies the data-scaling trend: models that depend less on explicit 3D knowledge improve faster as data scales, motivating a data-centric alternative to heavy 3D priors.\n\n2. The framework removes both explicit 3D architectural bias and pose supervision, which, if robust, simplifies pipelines and broadens applicability to unposed datasets.\n\n3. Practical impact in terms of enabling NVS without pose labels lowers annotation cost and allows training on large, in-the-wild photo collections.\n\n4. Empirical results claim parity with posed-input methods, suggesting the approach is not only elegant but effective on benchmarks.\n\n5. The paper reframes the trade-off between inductive bias and data scale, providing a clear hypothesis for future work and dataset collection strategies", "weaknesses": "1. The method’s advantages hinge on data scaling; performance in low-data regimes or niche domains may degrade relative to 3D-aware methods.\n\n2. Without explicit 3D representations or poses, ensuring geometric consistency across large camera motions or severe occlusions can be fragile; failure modes and long-range consistency are likely under-explored.\n\n3. Reported parity with posed methods may depend on specific datasets or evaluation metrics; results on highly diverse, real-world scenes or metric 3D accuracy may be weaker.\n\n4. Removing explicit 3D structure reduces model interpretability and makes diagnosing geometric errors or dataset biases harder.\n\n5. Achieving strong performance via data scaling may require substantial compute and careful curricula; the practical resource requirements may be high.\n\n6. Learning implicit 3D from 2D alone risks encoding photographic regularities that do not generalize to different capture conditions or object categories.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)