True Self-Supervised Novel View Synthesis is Transferable
Authors: Thomas Mitchel
Hyunwoo Ryu
Vincent Sitzmann
Venue: ICLR 2026 Oral (2026)
Keywords: Novel View Synthesis
Self-Supervised
Unsupervised
Representation Learning
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: The key criterion for determining whether a models is capable of NVS is transferability, and we present the first fully geometry-free and self-supervised model capable of it.

Abstract: In this paper, we identify that the key criterion for determining whether a model is truly capable of novel view synthesis (NVS) is transferability: Whether any pose representation extracted from one video sequence can be used to re-render the same camera trajectory in another. We analyze prior work on self-supervised NVS and find that their predicted poses do not transfer: The same set of poses lead
to different camera trajectories in different 3D scenes. Here, we present XFactor, the first geometry-free self-supervised model capable of true NVS. XFactor combines pair-wise pose estimation with a simple augmentation scheme of the inputs and outputs that jointly enables disentangling camera pose from scene content and facilitates geometric reasoning. Remarkably, we show that XFactor achieves transferability with unconstrained latent pose variables, without any 3D inductive biases or concepts from multi-view geometry — such as an explicit parameterization of poses as elements of SE(3). We introduce a new metric to quantify transferability, and through large-scale experiments, we demonstrate that XFactor significantly outperforms prior pose-free NVS transformers, and show that latent poses are highly correlated with real-world poses through probing experiments.

Published as a conference paper at ICLR 2026
TRUE SELF-SUPERVISED NOVEL VIEW SYNTHESIS IS TRANSFERABLE
Thomas W. Mitchel
Hyunwoo Ryu" MIT CSAIL,, PlayStation hwryuêmi edd
Vincent Sitzmann MIT CSAIL s sitzannmmit edu
Adobe, PlayStation
thomas mi mi tccheee 1 @mm i com
ABSTRACT
In this paper, we identify that the key criterion for determining whether a model is truly capable of novel view synthesis (NVS) is transferability: Whether any pose representation extracted from one video sequence can be used to re-render the same camera trajectory in another. We analyze prior work self-supervised NVS and find that their predicted poses do not transfer: The same set of poses lead different camera trajectories in different 3D scenes. Here, we present XFactor, the first geometry-free self-supervised model capable of true NVS. XFactor com- bines pair-wise pose estimation with a simple augmentation scheme of the inputs and outputs that jointly enables disentangling camera pose from scene content and facilitates geometric reasoning. Remarkably, we show that XFactor achieves transferability with unconstrained latent pose variables, without any 3D inductive biases or concepts from multi-view geometry - such as an explicit parameter- ization of poses elements of SE(3). We introduce new metric to quantify transferability, and through large-scale experiments, we demonstrate that XFactor significantly outperforms prior pose-free NVS transformers, and show that latent poses are highly correlated with real-world poses through probing experiments. Project Page: https: /www. mi tchel. computer wwtputteexxffccooo /xxactor//
INTRODUCTION
In recent years, novel view synthesis (NVS) has emerged as a canonical 3D computer vision prob- lem. Methods today are built the rich discipline of multi-view geometry, which has given rise to frructueefrommmmotee models that can preprocess large datasets of multi-view images to compute corresponding SB(3) camera poses. Given dataset of multi-view images and their camera poses, state-of-the-arr methods allow a user to specify camera pose as an SE(3) transform and render the corresponding view near photorealistically. However, the bitter lesson (Sutton, 2019) teaches to be skeptical of any inductive bias in learning systems. In this paper, we ask: Can we formulate NVS without reliance on multi-view geometry, tackling
pure machine learning problem?
To answer this question, we must first ask what novel view synthesis is without relying on the vocabulary of multi-view geometry. To this end, we identify transferability as the key property of any novel view synthesis model: the ability use set of camera poses extracted from one sequence to render the same camera trajectory in any other scene. As a corollary, the key requirement of any valid representation of camera poses is not that they can be identified with SE(3) representation, but that they render the same camera trajectory across scenes.
Equipped with this insight, we tackle self-supervised novel view synthesis pure machine learn- ing problem. We find that existing methods (Jiang et al., 2025; Sajjadi et al., 2023) do not infer transferable camera poses, and are instead prone interpolating context frames. This is not true novel view synthesis, as it does not allow the user to define which view they want to render in an arbitrary scene. Here, we present XFactor, the first self-supervised model capable of true NVS. XFactor combines pair-wise pose estimation with a simple augmentation scheme of the inputs and outputs that jointly enables disentangling camera pose from scene content and facilitates geometric reasoning. This is motivated by two key insights: 1) preventing the model from interpolating by
"Equally contributing authors.

---

Published as a conference paper at ICLR 2026
bootstrapping from two-view NVS model that extrapolates by design, 2) reifying transferability into a training objective compatible with real-world video by augmenting sequences of frames in manner that minimizes pixel content overlap while preserving camera motion, such applying two inverse masks to the same sequence. XFactor achieves transferability with unconstrained latent pose variables, without any 3D inductive biases or concepts from multi-view geometry such as an explicit parameterization of poses as elements of SE(3). We then fine-tune the two-view XFac- tor model into a multi-view model that we show enables transferable, high-quality NVS: Given sequence of frames and choosing one as the reference, we can generate latent trajectory by using the encoder to estimate the pose between the reference and each frame: Then, using any other video sequence context, our renderer will reproduce that same camera trajectory in the new scene.
Through extensive experiments, we show that our method is the first fully geometry- and self- supervised method that achieves true NVS across diverse, large-scale real-world datasets at both the scene and object level - including RE10K (Zhou et al., 2018), DL33V (Ling et al., 2024), MVImgNet (Yu et al., 2023), and CO3Dv2 (Reizenstein et al., 20.1). In particular, we introduce metric for quantifying the degree to which novel views adhere to reference poses, and demonstrate that XFactor dramatically outperforms prior methods RayZer (Jiang al., 2025) and RUST (Sajjadi et al.. 2023). In a series of ablations, we analyze what design decisions matter to solve transferable novel view synthesis, and demonstrate that, counter-intuitively. forcing the model parameterize camera poses as SE( is harmful and rather, what matters is a careful design of inputs and outputs to the pose estimator and renderer. In summary, we make the following key contributions:
1. We introduce transferability as the key criterion for determining whether self-supervised model is capable of true NVS and introduce the True Pose Similarity metric to quantify it.
2. We identify that prior multi-view self-super NVS models interpolate context frames instead of reasoning about viewpoints. We address this by boot-strapping multi-view NVS on top of two-frame model which, by design, always extrapolates.
3. We propose a novel self-supervised NVS training objective which explicitly promotes transferability, and introduce representation ceamiig-innspred augmentation strategy for training with real-world video.
4. Derived from these insights, we present XFactor which to our knowledge is the first fully self suff-supere NVS model to achieve transferability and thus perform true NVS.
5. We empirically demonstrate the merits of our formulation in comprehensive large-scale experiments and ablations.
2 RELATED WORK
We review prior work on novel view synthesis with and without known camera poses.
Oracle, Semi-Oracle, and Geometric Methods. Using camera poses obtained from an external pose oracle, such as COLMAP (Schönberger & Frahm, 2016a), neural networks can be trained to predict 3D neural scene representations (Yu et al., 2021; Charatan et al., 2023) or novel views directly (Jin et al., 2024; Sajjadi et al., 2021; Sitzmann et al., 2021). We refer to these methods as "Oracle Methods". Recent work has attempted to reduce the reliance on poses at training time to tap into larger datasets. These methods work by training a pose prediction module jointly with the novel view synthesis module. However, existing methods nevertheless rely on some form of external oracle, such pre-trained optical flow correspondence methods (Smith al., 2023; Chen & Lee, 2023; Wang et al., 2023), pre-trained depth estimators (Fu et al., 2023; Brachmann al., 2024), or initialization with weights that were pre-trained a supervised structure- from-- motion task (Huang & Mikolajezyk, 2025b). Some prior work achieves impressive camera pose estimation without relying on pre-trained oracles (Kang et al., 2025; Yin & Shi, 2018; Zhou al., 2017), enabled by strong expert-crafted geometric inductive biases such as warping, correspondence matching, and depth prediction. The goal of our paper is to develop first-principles approach to novel view synthesis that does not rely on any form of conventional multi-view geometry.
We use the term 'geometry-free' to denote that the model is free of heuristic 3D representations such as Gaussian splats, volumetric rendering or Plücker embeddings.
2

---

Published as a conference paper at ICLR 2026
Unsupervised Geometry-Free Latent Pose Methods. While several works have addressed un- posed novel view synthesis via explicit 3D modeling (Bian et al., 2023; Levy et al., 2024; Ye et al., 2024; Wang et al., 2025c; Kang et al., 2025; Huang & Mikolajezyk, 2025a), only a small number of methods have attempted to solve the unposed novel view synthesis problem without relying on external 3D oracles and using little 3D inductive bias as possible, tackling NVS as a pure deep learning problem. In this case, a pose estimation module predicts some form of camera poses that are used as conditioning inputs to geometry-free renderer transformer. A key challenge is to pre- vent the pose estimator from communicating information about the target frames to the renderer. RayZer (Jiang et al., 2025) and Less3 (Wang et al., 2025a) attempt accomplish this by pa- rameterizing latent poses as rigid-body SE(3) transforms. While renders are high quality, we show that this approach has significant limitation: The same set of predicted poses renders different camera trajectories in different 3D scenes, i.e., camera poses do not transfer between scenes. As we will see, this is the effect of the renderer performing interpolation context frames rather than true NVS. Closest to our work is RUST (Sajjadi al., 2023), which attempts fully geometry -free approach to novel view synthesis - its promising results were an inspiration for the present method. RUST attempts prevent cheating via an information bottleneck: the pose estimator receives only part of the target view. However, RUST does not solve the transferability problem - our proposed method is the first method to achieve geometry-free true NVS. Finally, we note that our approach shares similarities with recent work on latent action models (Zhang et al., 2025; Gao et al.. 2025; Bruce et al., 2024; Schmidt Jiang, 2024). While these models seek to extract transition latents de- scribing a variety of ego-centric actions from adjacent video frames, we instead focus on the specific problem of recovering transferable camera pose representations.
METHOD
3.1 NOVEL VIEW SYNTHESIS AS LATENT VARIABLE MODELING
To isolate the key properties of NVS, we first formulate it as a latent variable model. Given a sequence of images I [2,. Ig.... In} } of a static scene, existing NVS methods typically begin by partitioning them into two disjoint subsets of context images Ie and target images Ir with Io U IT I. These methods can generally be decomposed into three core components: pose encoder POSEENC, a scene encoder SCENEENC, and renderer RENDER. Given a choice of reference view IR Ic relative to which poses will be expressed, the pose encoder maps the context and target images to sets of latent pose representations: The scene encoder converts the context images and corresponding latent poses to a latent scene representation:
(Ic. Irr POSBENC, (Zc. Zr) and (Ic. Zc)
SCENEENS S.
(1)
In the prevailing formulation consistent across both supervised (Jin et al., 2024) and unsupervised settings (Jiang et al., 2025; Sajjadi et al., 2023), the role of the rendering decoder is to synthesize a prediction of the target images from the target poses and latent scene representation
(S, Zr)
RENDER
Ir,
and the model trained minimize what we call the autoencoding objective
L = d, (Ir, RENDER [S, Zr1).
(3)
subject to an image distance metric dj. Satisfying this objective requires the model only to have the ability to render target frames using scene and pose representations from the same sequence.
An important auxiliary tool in this framework is the ORACLE, which is simply algorithm that ingests a sequence of frames and spits out the ground-truth camera poses as elements of SE(3):
(Ji ..,, In ORACEE, (g), 92, 9a E SE(3)".
(4)
A canonical choice is ORACLE = COLMAP (Schönberger & Frahm, 2016a). However, in this paper, instead choose ORACLE VGGT (Wang et al., 2025b) due to its robustness and ease of use.
State-of-the-arr forward oracle NVS models including LVSM (Jin et al., 2024) and pixel- Splat (Charatan et al., 2023) simply define POSRENC ORACLE and seek to learn only SCENRENC
3

---

Published as a conference paper at ICLR 2026
and RENDER. Similarly, single-scene oracle models such those based NeRF Gaussian Splat- ting seek to directly optimize the scene representation S in the form of the weights of the rendering MLP or the Gaussians themselves. In contrast, self-supervised NVS models RayZer (Jiang al., 2025) and RUST (Sajjadi et al., 2023) aim to learn all three modules end-to-end without reliance on ORACLE. However, we show empirically (Sec. 4.1) that the prevailing framework for NVS described in Equations (1-3) is in fact ill-suited for the self-supervised setting as it does not consider the fundamental property making a model capable of true NVS: transferability.
3.2 TRUE NOVEL VIEW SYNTHESIS IS TRANSFERABLE
NVS is simply the ability to render a scene from a user-controllable viewpoint: is critical that the same camera pose always results in the same viewpoint being rendered. If the model cannot do this, it is not a true NVS model, but rather, a frame interpolator. We propose that in the perspective of NVS as a latent-variable model (Sec. 3.1), controllability is equivalent transferability and can be formalized as the property that pose representations transfer between scenes.
I = I U I and _ I U I be sequences whose target frames It and I share the same camera motion, Lee ORACLE [TA] ORACLE (TT) Then, say that an arbitrary NVS model consisting of core components [POSEENC, SCENEENC, RENDER] produces transferable pose rep- resentations (or a true NVS model) if the latent representations and
POSEENC [Ig IA] (z). Z
SCENEENC [I] _g] S
(5)
and renderer satisfy
RENDER [S II.
(6)
This criterion automatically satisfies the autoencoding objective in Equation (3) when I I and captures the essence of controllable NVS: the ability to apply camera trajectories from one scene to synthesize views of another scene. We note that both conventional feed-forward and single-scene oracle NVS models with POSEENC = ORACLE are automatically transferable (and thus are capable of true NVS) with the autoencoding objective, as for any scene representation S
RENDER S ORACLE [
RENDER S ORACLE
2 I..
3.3 QUANTIFYING TRANSFERABILITY WITH TRUE POSE SIMILARITY (TPS)
We introduce a standardized metric to quantify the degree of transferability of latent pose representations which we call True Pose Similarity (TPS). Given an ORACLE and trajectory com- parison metric dSE(3)", such as Relative Rotation Accuracy (RRA), Relative Translation Accuracy (RTA), or Area Under Curve (AUC) which combines the two, we define the TPS be- tween two sequences of frames I and I of equal length to be the value of the metric between the oracle poses from each sequence:
Context Soene
Target Frames
Novel Views
Medel
ORACLE
Same? Transferahle!
(PS 1D des(s)n (ORACLE [I+1, ORACLE[I (8)
Different? Not Transieable!
To quantify the transferability of a |POSEENC, SCENEENC, RENDER] NVS model with TPS, consider two arbitrary sequences z IA U IÁA' and 7 Ig U IP. We then use the scene representation from the second sequence S and the target latent poses 2 from the first sequence as in Equation (1) render a new trajectory in the second sequence, leveraging TPS to measure whether their camera trajectories agree:
TPS(IA, RENDER [B
( 2z1).
We note that this quantity only measures one component transferability - that the rendered viewpoints are geometrically consistent, and not also faithful to the context sequence. For instance, this metric, unlike the definition in Sec. 3.2, can be hacked by a model RENDER(S"" Z ~ IA. so it is necessary to pair it with a perceptual measure either qualitative or quantitative faithfulness. We highlight that we only rely on ORACLE for benchmarking purposes; our proposed method does not rely on any external pre-trained or expert-crafted ORACLE for training.
verify
4

---

Published as a conference paper at ICLR 2026
Pose-Preserving Augmentation ORACLEJAUG A|=ORacLE(AGG
Stereo-Monocular NVS
Posc
Source
AUG
Rendering Loss
POSBENC
Input Pair
RENDER
AUG B
Context
Pred
Target
Figure I: XFactor combines a [PoSEENC, RENDER] stereo-monocular model with our proposed transferability objective learn transferable camera pose latents. Given pair of input images, we apply two augmentations that minimize pixel content overlap while preserving pose information, such inverse masking. The stereo POSEENC extracts the relative pose latent from the first pair. Then, given the context image from the second pair and the first's target pose, the renderer is asked reconstruct the second's target.
3.4 SOLVING TWO PRINCIPAT. PROBLEMS: INTERPOLATION AND INFORMATION LEAKAGE
In the self-supervised setting there is no guarantee of transferability and we demonstrate empirically that existing models RayZer and RUST fail to produce transferable pose representations under the TPS metric (Sec. 4. 4.1). ). In what follows we provide two key insights regarding why these models fail, and from them derive approach for learning transferable pose latent representations.
The Moccco-Monocular Model. We first note that in both RayZer and RUST, their pose encoders and renderers have access to multiple context views. We find that training such self-supervised multi-view model leads to a model that uses the latent "pose" to encode how to interpolate context views synthesize the target view. Such "pose" will not transfer a different scene, as a different scene will feature different context views. This is hence not true NVS because it does not allow the user to define which view they want to render in an arbitrary scene.
To prevent the model from learning to interpolate and instead reason about poses, we propose to bootstrap a self-supervised multi-view NVS model off of stereo-monocular model that must al- ways extrapolate. Specifically, we consider the case where there is only single context and target image, respectively - e.g. I (T1: ]2] with Ic [T,} and LT = (12). Thus, the POSRENC be- comes two-view stereo model, SCENEENC can be absorbed by RENDER, and RENDER is monoc- ular:
POSEENC (T, I2] = Z2 and
RENDER [T, Zz] = T2.
(10)
By providing RENDER with only single image for reconstruction we eliminate the interpolation path and guide optimization toward learning transferable pose representations. We note that this approach shares similarities with CroCo (Weinzaepfel et al., 2023), a reprrssntatioon-ccannig method which leverages monocular renderer promote the learning of depth cues.
The Transferability Objective. While we show that the ferable pose representations (Sec. 4.3), it still allows for POSEENC to encode information about target pixels, rather than a purely geometric description of the relative pose. This again provides an easier "off ramp" for RENDER, which does have perform full NVS but can cheat by decoding pixel information smuggled into the target pose latent.
model produces trans-
We propose to discourage the entanglement of pixel information by explicitly defining the training objective as transferability:: Given two pairs of frames I = (T I54) and I {T 7 which are known to share the same relative camera pose we ask that the relative pose latent extracted from the first sequence must be able to render the target image from the second,
L d, (T RENDER [T POSHENC [TA, 1z"1))
(11)
which call the transferability objective. However, despite the obvious benefits of imposing transferability as the training objective, less clear how to get such pairs in practice, especially when training with real-world data.
5

---

Published as a conference paper at ICLR 2026
To this end our third and final key insight is that given any sequence of frames I, any two Ttame-wiss augmentations AUG and AUG that preserve the ground camera pose, i.e.,
ORACLE [AuG[Z]] = ORACLE [AuG[z]] = ORACLE [Z],
(12)
can be used to produce two new sequences which share identical camera motion but very little pixel information. Combining this insight with the transferability objective gives rise to a novel training procedure. In practice, given imput pair we implement this strategy by randomly generating two equal-area masks whose union covers the whole image, and apply these in combination with color- jitter and blur to generate new pairs. Then, following the transferability objective in Equation (11), POSEENC extracts the relative pose latent from the first pair with which RENDER is asked to render the target image in the second pair given the first image as context.
We note that prior self-supervised methods RayZer and RUST also seek to prevent information leak- age through different strategies. RayZer takes more explicit approach by bottlenecking the pose latents via parameterization as elements of SE(3). However, as we show empirically in both bench- mark comparisons and ablations, an explicit SE(3) parameterization not only fails to provide any degree of transferability in the multi-view setting, but in fact degrades it compared to an uncon- strained stereo- mereol- baseline. contrast, RUST takes an approach that is more similar to our own wherein the target pose latent is estimated from the scene representation and a partial view of the target image, and the renderer asked to reconstruct the full target. However, RUST still suffers from interpolative bias via multi-view training and an augmentation strategy that does not eliminate much of the pixel content overlap.
3.5 XFACTOR: A MODEL FOR TRUE SELF-SUPERVISED NOVEL VIEW SYNTHESIS
Combining the [POSEENC, RENDER] stereo-monocular model with the transferability objective gives rise to XFactor (Figure 1) short for Transferable(X] Latent Factorization(Facton As we demonstrate empirically in (Sec. 4.1), XFactor produces a fully transferable latent pose representa- tion and to our knowledge the first fully self-supervised model to achieve true NVS the sense of Equations (5 - 6). We also note that XFactor does so without any geometric or 3D inductive biases whatsoever, demonstrating that such design choices are not a necessary condition for transfcrable latent poses.
Both POSEENC and RENDER are implemented as multi-view VITs. We define the image distance metric d, used to compute the transfer objective linear combination of the L' norm the difference between the ground truth and predicted target image pixels and the LPIPS loss (Zhang al., 2018), with a weight of 0.5 the latter. During training, augmentation masks are generated per batch example by splitting the patchified image plane into quadrants and randomly partitioning them into two groups of two. This not only allows for masks which either equally partition the image into left/right or upper/lower halves, but also diagonalized partitions. There also exists a small chance (5% in our implementation) that an image pair will not be masked, in which case transfer objective for that example reduces to the intra-sequence autoencoding objective and gives the model an opportunity to reason about the whole images. Augmentations are not applied during inference. Complete architectural and implementation details are included in Appendix A.
Multi- View XFactor. Given trained XFactor stereo-monocular model extend it to multi- view model by fine-tuning [PoSEENC, RENDER] in a secondary training stage. Here each multi- frame sequence
Ic J {4r} is split into two disjoints sets consisting of context images and a single target image, with the latter chosen randomly. The reference image In Ie represents the view relative to which all poses will be expressed, and chosen to be the frame with the minimum maximal baseline between all other frames (i.e. the "middlest"). We continue to use pose-pree pose-preserving augmentations that are, however, now applied to all frames. For each sequence, POSEENC is applied pair-wise to predict the relative pose latents between the reference frame and all others. Then, RENDER is asked to render the target image of the second sequence with the second's 's context frames and poses and the target pose from the first.
4 EXPERIMENTS
We provide empirical evidence that XFactor produces a transferable camera pose representation (4.1) which well-predicts oracle SE(3) poses when probed (4.2). Last, show through abla-

---

Published as a conference paper at ICLR 2026
BOST
MONY
Teyet
Shone
20122 BaCc.
MEST
Buc.. Stene RAe
KneT Korta Yho Ley.
BIST Basti Nhare tave
BEST Bache Tho Taye
RayZer (Jiung al., 2025)
(Sujadi
Metrie
RE10K
DL3DV
MVImgNet
CO3D22
RE10K
DI.3DV
MVImgget
CO3D<2
RE10K
DL3DV
MVImgNet
CO3Dv2
Ace
10'
98.6
95.6
88.1
88.5
76.9
44.9
9.57
31.2
87.2
62.7
12.4
4,2
(7)
30'
99.7
98.8
98.2
97.7
95.1
81.4
43.0
76.1
92.9
48.6
9.5
(1)
20"
30'
90.
K3.5
87.9
93.9
89.4
95.2
73.8
61..9
52.1
20.3
38.8
24.4
22.1
37.7
29.1
16.0
46.3
32.1
47.5
31.8
27.6
48.6
83.4
22.2
AUC
10"
34.6
34.8
28.2
14.0
2.3
1.3
3.3
4.3
5.2
3.1
0.9
20'
55.2
57.2
53.4
31.2
7.6
5.9
2.7
13.8
10.8
HD
4.5
26.6
8.4
24.7
Table 1: The Transferability Test. We compare the transferability of XFactor's, RayZer's, and RUST's pose representations across four datasets. We evaluate using TPS with RRA, RTA, and AUC different error thresholds. For all metries except FID, higher is better. Visualizations of transfer renderings and camera trajectories extracted with ORACLE are shown for each method above. The target trajectory visualized in red, XFactor in green, RayZer in blue, and RUST in gold.
tions that combining the stereo-monocular model with our transferability objective is ideal for both achieving transferability and producing a pose representation relative to a variety of alternative de- sign decisions (4.3). In addition, we also report results on the benchmark of auto-encoding video sequences established by Jiang al. (2025) Appendix B.
Comparisons. We compare XFactor against two strong existing self-supervised NVS models: RayZer (Jiang al., 2025) the current self-supervised state-of-the-art, and RUST (Sajjadi et al., 2023) which also estimates poses from partial views and does not make use of any 3D inductive bias. At this time, neither the authors RayZer nor RUST have published code. We implement both models following the respective papers and shared our RayZer implementation with the au- thors, who confirmed poses between full and partial views. Our implementation differs slightly from that of the authors: we do not leverage a set-latent scene representation. instead absorbing the scene encoder into a multi-view POSEENC and RENDER; POSEENC the set of all context views as well as the par- tial target view. Thus, comparisons against RUST can be seen as a comparison with training a multi-view model end-to-end with full-to-partial objective instead of transferability.
faithful. For RUST, we view their principal contribution as predicting
Datasets and Training. We train all models on a large-scale, aggregate dataset consisting of real- world videos at both and object levels. Specifically, dataset consists of RE108 ZE10K (Zhou et al.,

---

Published as a conference paper at ICLR 2026
XFacoo (Ours)
RayZer (hing al., 2025)
RUST (Sujidi et al., 2023)
MVImngNet
MVlnngNet
MVImgNet
Ace
10
99.3
96.9
95.3
09.1
05.4
97.4
(7)
20
99.8
99.1
99.9
30
99.8
100
99.4
98.4
99.9
99.4
99.7
98.5
99.4
97.1
96.8
96.3
T.Ace
10
73.95
87.5
93.7
67.9
57.1
06.1
81.3
51.0
49.1
57.1
74.
44.3
(1)
202
87.5
96.1
98.4
$2.9
78.6
85.0
91.5
73.3
72.8
78.6
91.1
68.1
30"
91.9
99.6
88.5
AUC
10'
46.8
60.1
66.4
42.6
29.6
32.2
46.4
25.3
24.6
27.9
31.5
186
78.9
58.5
307
72.9
85.4
67.5
60.8
66.2
79.4
59.2
69.2
49.3
Table Pose Probe Accuracy. We report probe accuracy trained to predict ground-truth SE(3) poses from the latents of each model in terms of RRA, RTA, and AUC. We show several examples of XFactor's poses (green) relative to ORACLE ground-trutt truth (red). Zoom in see details.
2018), DL3DV (Ling al., 2024), MVImgNet (Yu al., 2023), and CO3Dv2 (Reizenstein al., 2021). Frames are first center-cropped and then resized to 256 x 256 pixels. Complete training and implementation details can found in Appendix
4.1 TRANSFERABILITY
Our principal evaluation concerns transferability. Here, we compare multi-view XFactor, RayZer, and RUST on the evaluation splits of our aggregate dataset: For each dataset, we randomly draw 4000 pairs of sequences, select five equally spaced target frames, and compute the TPS in Equa- tion (9) with respect to RRA, RTA, and AUC at 10 intervals. We also compute the Frechet Inception Distance (FID) between the statistics of the sequences of input and rendered target frames gen- eral measure of transferred rendering quality.
The results, averaged over all sequences per dataset, and qualitative comparisons shown in Ta- ble 1. XFactor significantly outperforms the other methods, reporting an AUC @ 20 over five times that of RayZer and RUST. Notably, despite sometimes producing reasonable looking renderings, both RayZer and RUST completely fail the transferability test and are not capable of true NVS. This likely due the susceptibility of their design toward learning interpolation latents due to end-to- end multi-view training with autoencoding, Of the two, RUST's performance is slightly better. We attribute this to strategy of estimating pose latents between full and partial views it brings its objective to a place between transferability and autoencoding.
4.2 POSE PROBE
Next, we evaluate the degree which each model's latent pose representation encodes information about the ORACLE camera poses. To do so, freeze the POSEENC from each model and train a three-layer MLP to predict the ground-truth SE(3) camera poses extracted by ORACLE from the estimated pose latents. We evaluate the quality of the probe-extracted trajectories relative to the ground-truth in terms of RRA, RTA, and AUC with the results shown in Table 2. Visualizations of poses extracted from XFactor's representations shown above.
Overall, XFactor's 's pose latents provide a superior characterization of the oracle poses, with high AUC values 10 and 20 and outperforming the other methods significantly. It follows that
8

---

Published as a conference paper at ICLR 2026
XFactor (Ours)
Bottleneck
Unconstrained
Metrie
RE10K
DL-DV
MVImgNet
0O39x2
RE10K
DILDD
MVImgNett
CO3Dv2
RE10K
DLADY
MVImgNet
CO30v
30" (1)
96.5
98.2
97.7
96.7
98.0
97.9
T 207 (1)
75.1
76.6
84.1
33.05
70.4
72.6
85.3
35,5
64.3
70.1
81.0
31,99
AUC 20"
47.2
44.8
49.3
14.5
40.6
41.4
50.8
15.8
36.8
39.4
47.0
14.0
33.9
R.20" (1)
99.5
98.5
99.7
97.5
96.9
96.7
97.6
T:20" (1)
79.4
77.4
83.0
74.4
85.4
75.4
ATCC 20" (t)
54.8
61.2
71.5
50.3
47.2
51.9
71.0
48.6
40.8
51.2
70.8
48.4
SB(3)
Plücker (Jlng 2025)
Additional View: Decoder
Additional View: Encoder Decoder
Metric
RE10K
DI.3DV
MVlingNet
CO3Dv2
RE10K
DH.3D MVIingNer
CO3Dv2
REIOK
DI.3DV
MVIingNet
CO3Dv2
Trander
R 2.20" (13
99.8
94.2
0.826
95.6
99.7
94.22
86.1
97.7
99.6
93.9
43.5
96.2
1 1'20""
65.3
58.6
31.7
62.0
52.2
53.1
35.3
19.7
14.2
10.4
AUC
12.6
14,9
32
FFI (1)
3.53
36.1
7.37
6.11
5.77
30.5
16.2
9.86
5.0
36.6
6.4
9.1
Probe
3000 (1)
99.0
98.2
99.1
96.2
98.6
96.1
98.8
95.8
95.4
95.3
92.2
"1200" (1)
77.9
$5.0
95.4
72.8
74.7
78.4
94.3
70.6
73.9
88.4
93.9
74.8
AUCC 20" (t)
50.6
56.3
67.7
45.2
48.0
47.4
64.2
42.6
47.4
58.1
65.9
48.6
Table 3: Ablations. We ablate potential alternative design decisions using
XFac-
for as a starting point. Models are compared in terms of transferability and pose probe efficacy.
stereo-monocular model combined with our transferability objective also doubles as an effective method for self-supervised representation learning of 3D camera pose information. However, unlike the transferability test, neither RayZer nor RUST completely fail and in fact both learn a reasonably informative representation. This suggests that while transferability can improve geometric reason- ing, evidence of the latter does not automatically lead to the former.
4.3 ABLATIONS
Here we ablate the influence of the fundamental components of XFactor's design - the stereo- monocular model and transferability objective. To do so, we train and evaluate several alterna- tive models each representing different potential design decisions, and compare against stereo- monocular XFactor terms of transferability and pose probe accuracy.
To ablate the influence of a monocular model relative to multi-view, we train two additional models with the transferability objective: A stereo POSEENC which uses single additional con- text view in RENDER (Additional View: Decoder), essentially our proposed multi-view XFactor trained end-to-end: A full multi-view model wherein both POSEENC and RENDER see the addi- tional context view (Additional View: Encoder + Decoder). To evaluate the effectiveness of the transferability objective we train three stereo-monocular models with the standard autoencoding objective: One without any additional modification (Unconstrained);; One with 16-dimensional pose latents, representing a bottleneck relative to the 256-dimensional latents used XFactor (Bottle- neck); One which predicts SE(3) poses and camera intrinsies and uses Plücker embeddings in the decoder following Jiang et al. (2025) (SE(3) & Plücker).
The results are shown in Table 3. While XFactor overall performs best out of all models, that transitioning to multi-view training, first by adding only an additional context view to RENDER, and then by adding a context view POSEENC, progressively degrades, then completely destroys transferability. In contrast, the bottlenecked stereo-monocular model performs competitively with XFactor in terms of transferability, though XFactor's ' latents provide a comprehensively stronger characterization of real-world pose. However, we note that bottlenecking strategy may not always be desirable and can limit descriptiveness, for instance, if one seeks a representation that also en- codes changes in lighting or other evolving phenomena in scene. In contrast, the transferability objective improves transfcrability without an explicit design constraint. Countcrintuitively we find that asking the dverco-monocutt model to predict explicit SE(3) poses and camera parameters in fact significantly degrades transferability relative to both XFactor and the unconstrained baseline.
5 DISCUSSION
Limitations. While model has claim to being the first geometry-free, fully-self supervised method to achieve true NVS. several limitations are outstanding. First, the restriction of POSEENC to stereo model precludes ultra-wide baseline pose estimation in single forward pass. In prin-

---

Published as a conference paper at ICLR 2026
ciple, multi-view POSEENC robustly estimate latent poses across arbitrary baselines long as it is possible to chain together a trajectory between frames that share overlap. In fact, such a model is highly effective in the supervised regime (Wang et al., 2025b), however, applying it in self-supervised setting without introducing interpolative bias remains an open problem. Second, the rendering quality of transferred frames can exhibit blurring and warping artifacts which increase in frequency as the target poses diverge from those of the context. As illustrated in Fig, 3, determin- istic NVS models supervised with ground-truth poses exhibit similar issues. Therefore, we posit this stems from the fact that XFactor is deterministic, rather than generative, model and that the artifacts are result of the model trying to resolve uncertainty without currently being equipped with the proper tools to do so. We believe that integrating recent advances in camera-controlln generative models (Song et al., 2025; Bai al., 2025) into our approach could effectively address this limitation.
Conclusion. We have presented new characterization of NVS that does not rely notions from conventional multi-view geometry, instead formulating it as a pure machine-learning task in the form of latent variable model. We identified transfcrability NVS. Based on our analysis, we introduced XFactor, the first gcometry- capable of true novel view synthesis. We provided large-scale experiments real-world datasets that demonstrate XFactor's efficacy, and validate key design decisions with careful ablation studies. We hope that our analysis will encourage the community to seek new formulations of classic 3D vision problems based on key principles of machine learning.
the key input-output behavior of
self-supervised model
ACKNOWLEDGMENTS
Vincent Sitzmann was supported by the National Science Foundation under Grant No. 2211259, by the Singapore DSTA under DST000ECT203000223 (New Representations for Vision, 3D Self-Supervised Learning for Label-Efficient Vision), by the Intelligence Advanced Research Projects Activity (IARPA) via Department of Interior/Interior Business Center (DOI/IBC) under 140D0423C0075, by the Amazon Science Hub, by the MITT-Goggle Program for Computing Inno- vation, by Sony Interactive Entertainment, and by 2025 MIT ORCD Seed Fund Compute Grant.
REFERENCES
Jianhong Bai, Menghan Xia, Xiao Fu, Xintao Wang. Lianrui Mu, Jinwen Cao, Zuozhu Liu, Haoji Hu, Xiang Bai, Pengfei Wan, et al. Recammaster: Camera-controlled generative rendering from single video. arXiv preprint arXhv:2503.11647 2025.
Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Variance- Varinne-iiarrinnc--ovvaannn regularizztionnnn for self-supervised learning, 2022. URL http::/ps///aat....oooa//a////// 04966.
Wenjing Bian, Zirui Wang, Kejie Li, Jiawang Bian, and Victor Adrian Prisacariu. Nope-nerf: Opti- mising neural radiance field with pose prior. 2023.
Eric Brachmann, Jamie Wynn, Shuai Chen, Tommaso Cavallari, Áron Monszpart, Daniyar Tur- mukhambetov, and Victor Adrian Prisacariu. Scene coordinate reconstruction: Posing of image collections via incremental learning of a relocalizer. In ECCV, 2024.
Jake Bruce, Michael Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, Yusuf Aytar, Sarah Bech- tle, Feryal Behbahani, Stephanie Chan, Nicolas Heess, Lucy Gonzalez, Simon Osindero, Sher- jil Ozair, Scott Reed, Jingwei Zhang, Konrad Zolna, Jeff Clune, Nando de Freitas, Satin- der Singh, and Tim Rocktäschel. Genie: Generative interactive environments, 2024. URL https://arxis orgg alss 22222
David Charatan, Sizhe Li, Andrea Tagliasacchi, and Vincent Sitzmann. pixelsplat: 3d gaussian splats from image pairs for scalable generalizable 3d reconstruction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
10

---

Published as a conference paper at ICLR 2026
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations, 2020. URL https: haapp:///axxiii /aaaxaxvvvv//// org/ abs/ 2002. 05709.
Yu Chen and Gim Hee Lee. Dbarf: Deep bundle-adjusting generalizable neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 44-34, June 2023.
Yang Fu, Sifei Liu, Amey Kulkarni, Jan Kautz, Alexei A. Efros, and Xiaolong Wang. Colmap-free 3d gaussian splatting. 2023.
Shenyuan Gao, Siyuan Zhou, Yilun Du, Jun Zhang, and Chuang Gan. Adaworld: Learning adaptable world models with latent actions. CoRR, abs/2503. hass/203.18938,, 18938, March 2025. URL https://don org/10.... 48550 arXiv 2503.
Ranran Huang and Krystian Mikolajczyk. No pose at all: Self-supervised pose-free 3d gaussian splatting from sparse views. In Proceedings of the IEEE/CVF International Conference Com- puter Vision, pp. 27947-27957 2025a.
Ranran Huang and Krystian Mikolajczyk. No pose at all: Self-supervised pose-free 3d gaussian splatting from sparse views. arXiv preprint 11XI 2025b.
Hanwen Jiang, Hao Tan, Peng Wang, Haian Jin, Yue Zhao, Sai Bi, Kai Zhang, Fujun Luan, Kalyan Sunkavalli, Qixing Huang, and Georgios Pavlakos. Rayzer: A self-supervised large view synthe- sis model. In Proceedings of the International Conference on Computer Vision (ICCV), 2025.
Haian Jin, Hanwen Jiang, Hao Tan, Kai Zhang, Sai Bi, Tianyuan Zhang, Fujun Luan, Noah Snavely, and Zexiang Xu. Lysm: large view synthesis model with minimal 3d inductive bias. arXiv preprint arXiv:2410. 17221222.. 2024.
Gyeongjin Kang, Jisang Yoo, Jihyeon Park. Scungtae Nam, Hyconsoo Im, Sangheon Shin, Sangpil Kim, and Eunbyung Park. Selfsplat: Pose-free and 3d prior-free generalizable 3d gaussian splat- ting. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 22012- 22022, 2025.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv:1412.6990 2014.
Axel Levy, Mark Matthews, Matan Sela, Gordon Wetzstein, and Dmitry Lagun. Melon: Nerf with unposed images in (3). In 2024 International Conference 3D Vision (3DV), pp. 354-364. IEEE Computer Society, 2024.
Lu Ling, Yichen Sheng, Zhi Tu, Wentian Zhao, Cheng Xin, Kun Wan, Lantao Yu, Qianyu Guo, Zixun Yu, Yawen Lu, et al. D13dv-10k: large-scale scene dataset for deep learning-basee 3d vision. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 22160-22169, 2024.
Jeremy Reizenstein, Roman Shapovalov, Philipp Henzler, Luca Sbordone, Patrick Labatut, and David Novotny, Common objects in 3d: Large-scale learning and evaluation of real-life 3d cate- gory reconstruction. In International Conference on Computer Vision, 2021.
Mehdi SM Sajjadi, Henning Meyer, Etienne Pot, Urs Bergmann, Klaus Greff, Noha Radwan, Suhani Vora, Mario Lucic, Daniel Duckworth, Alexey Dosovitskiy, et al. Scene representation trans- former: Geometry-free novel view synthesis through set-latent scene representations. arXiv preprint arXiv:2111.13155 2021.
Mehdi SM Sajjadi, Aravindh Mahendran, Thomas Kipf, Etienne Pot. Daniel Duckworth, Mario Lučić, and Klaus Greff. Rust: Latent neural scene representations from unposed imagery. Pro- ceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition, pp. 17297- 17306, 2023.
Dominik Schmidt and Minqi Jiang. Learning to act without actions. 2024. URL https: // //arxiv. wttp://oorr/bb////1/113 10812.
11

---

Published as a conference paper at ICLR 2026
Johannes Lutz Schönberger and Jan-Michael Frahm. Structure-from from-Motion Revisited. In Proceed- ings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016a.
Johannes Lutz Schönberger and Jan-Michael Frahm. Structure- Structure-from from-rrooiiii motion revisited. In Confer- ence on Computer Vision and Pattern Recognition (CVPR), 2016b.
Johannes Lutz Schönberger, Enliang Zheng, Mare Pollefeys, and Jan-Michate Frahm. Pixelwise view selection for unstructured multi-vien view stereo. In European Conference on Computer Vision (ECCV), 2016.
Vincent Sitzmann, Semon Rezchikov, Bill Freeman, Josh Tenenbaum, and Fredo Durand. Light field networks: Neural scene representations with single-evaluation rendering. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
Cameron Smith, Yilun Du, Ayush Tewari, and Vincent Sitzmann. Flowcam: Training generaliz- able 3d radiance fields without camera poses via pixel-aligned scene flow. Advances in Neural Information Processing Systems (NeurIPS), 2023.
Kiwhan Song, Boyuan Chen, Max Simchowitz, Yilun Du, Russ Tedrake, and Vincent Sitzmann. History-guided video diffusion. arXiv preprint 4rXiv:2502.06664, 2025.
Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: En- hanced transformer with rotary position embedding, 2023. URL https: StRL::ts:://aat /ttsss//aaxvvv.oa org/abs/ 2104. 64.
Richard Sutton. The bitter lesson. Incomplete Ideas (blog). 13(1):38, 2019.
Haoru Wang, Kai Ye, Yangyan Li, Wenzheng Chen, and Baoquan Chen. The less you depend, the more you learn: Synthesizing novel views from sparse, unposed images without any 3d knowl- edge. arXiv preprint arXiv:2506.0988 2025a.
Jianyuan Wang, Nikita Karaev, Christian Rupprecht, and David Novotny. Visual geometry grounded deep structure from motion. arXiv preprint arXiv:2312.0456. 2023.
Jianyuan Wang, Minghao Chen, Nikita Karaev, Andrea Vedaldi, Christian Rupprecht, and David Novotny. Vggt: Visual geometry grounded transformer. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2025b.
Ruoyu Wang, Yi Ma, and Shenghua Gao. Recollection from pensieve: Novel view synthesis via learning from uncalibrated videos. arXiv preprint arXiv:2505. 13440, 2025c.
Philippe Weinzaepfel, Thomas Lucas, Vincent Leroy, Yohann Cabon, Vaibhav Arora, Romain Brégier, Gabriela Csurka, Leonid Antsfeld, Boris Chidlovskii, and Jérôme Revaud. CroCo v2: Improved Cross-view Completion Pre-training for Stereo Matching and Optical Flow. In ICCV, 2023.
Botao Ye, Sifei Liu, Haofei Xu, Xueting Li, Mare Pollefeys, Ming-Hsuan Yang, and Songyou Peng. No pose, no problem: Surprisingly simple 3d gaussian splats from sparse unposed images. arXiv preprint arXiv:2410.24207 2024.
Zhichao Yin and Jianping Shi. GeoNet: Unsupervised learning of dense depth, optical flow and camera pose. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2018
Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. PixeINeRF: Neural radiance fields from one or few images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
Xianggang Yu, Mutian Xu, Yidan Zhang, Haolin Liu, Chongjie Ye, Yushuang Wu, Zizheng Yan, Chenming Zhu, Zhangyang Xiong, Tianyou Liang, Guanying Chen, Shuguang Cui, and Xi- aoguang Han. Mvimgnet: large-scale dataset multi-view images, 2023. URL https: / //arxiv.org/abb/2253 06042.
12

---

Published as a conference paper at ICLR 2026
Chuheng Zhang, Tim Pearce, Pushi Zhang, Kaixin Wang, Xiaoyu Chen, Wei Shen, Li Zhao, and Jiang Bian. What do latent action models actually learn? arXiv preprint arXiv:2506.15691, 2025.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features perceptual metric. In CVPR, 2018.
Tinghui Zhou, Matthew Brown, Noah Snavely, and David G Lowe. Unsupervised learning of depth and ego-motion from video. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnification: Learning view synthesis using multiplane images. ACM Trans. Graph. (Proc. SIGGRAPH), 37, 2018. URL wwtpt:ttss///iiiiieee org/ abs 10000 09817.
A XFACTOR: ARCHITECTURE AND IMPLEMENTATION DETAILS
The XFactor POSEENC and RENDER modules are implemented as multi- view ViTs with RoPE positional embeddings (Su et al., 2023). Following VGGT (Wang et al., 2025b), we fuse global and per-image attention inside each layer. While initially trained in the stereo-monocular setting, both PoSEENC and RENDER are capable of handling arbitrary number of views independent of the weights, though we only take advantage of this capability for RENDER when extending to multi- view. During training, both POSEENC and RENDER are passed binary attention mask encoding the randomly generated disjoint partitions (AUG, AUG) which allows for encoding and rendering with respect to the transferability objective be computed in a single forward pass.
POSEENC consists of local-global attention layers, followed by pose head in the form of an MLP. single global token initialized and copied across the context and target view, representing the context and target pose latents. The attention layers are equivariant under swapping of the context and target image. Symmetry broken by the pose head, which is designed such that the reference pose is always mapped to the zero vector.
RENDER implemented similarly, consisting of local-global attention layers and an MLP pixel prediction head. The input pose latents are broadcasted across the token dimension, with the context pose latents fused to the pacification of the target image and the result is concatenated along the token dimension with the broadcasted target latents to form internal two-view representation. After applying the attention layers, the features corresponding to the broadcasted target latents are extracted from the position of second image and passed to the pixel prediction head.
B EVALUATION
Comparisons and Training. In our comparisons, initialize all of XFactor's, RayZer's, and RUST's POSEENC, SCENEENC (used only in RayZer), and RENDER modules with eight trans- former layers, 1024 features, 16 heads, and a patch size of 16, resulting in 16, 24, and 16 total layers for each model, respectively. Both XFactor and RUST use a latent pose dimension of 256. We stan- dardize comparisons such that all methods render with 5 context views. Multi-view XFactor and RUST each take the reference view as one of the context views and along with a single additional target view. For RayZer, we use an additional target views as is done in (Jiang et al., 2025).
We train with two separate baselines, with one set used in both training storeo-monocular XFac- tor and ablations, and the other used for both training multi-view XFactor, RayZer, RUST and all comparison evaluations. For the former, pairs of images are formed by randomly sampling frames from each dataset up to maximum baseline consisting of 100, 12, 12, and 20 frames for RE10K, DL3DV, MVImgNet, and CO3Dv2, respectively. These baselines were heuristically selected based on dataset difficulty and to ensure at least a small amount of overlap between pairs of frames. For fine-tuning, and training RayZer and RUST, we simply double the stereo- monocular baseline.
We train stereo- monocular XFactor, RayZer, and RUST all with the AdamW optimizer (Kingma & Ba, 2014), using batch size of 256, weight decay of 5.0 10-3, 3, and learning rate of 4.0 x 10-4 for 100,000 iterations, decaying to 1.0 x 10 on a cosine schedule. To extend XFacoor to multi-view fine-tune for additional 100,000 iterations.
13

---

Published as a conference paper at ICLR 2026
XFactor (Ours)
Zer Uiang et al., 2025)
RUST (Sujadi et al., 2023)
Metric
RE1OK
DL.DV
MVimggNt
CO30v2
RE10K
DL.DV
MVImgget
COSDV2
RE10K
DL.DV
MVImgget
CO30v2
PSNR <t>
26.1
22.9
20.3
21:4
22.6
18.9
17.7
19.1
19.3
SSIM (1)
0.859
0.766
0.733
0.821
0,809
0.676
0.622
0.728
0.71
0.571
0.593
0,662
LPIPS (1)
0.114
0,157
0.194
0.1677
0.135
0.185
0.358
0.205
0.220
0.286
0.3561
0.290
Table 4: Autoencoding Reconstruction Quality.
XFactor (Aug. Inferemce)
MVImgget
26.1
24.2
25.5
LPIPS (1)
0.107
0.151
0.196
0.172
2.55
22.1
3.43
4.84
Table 5: Augmentations at Inference. We evaluate transferred rendering quality in terms of stan- dard perceptual metrics by applying our pose-preserving augmentations inference with multi-view XFactor.
We train stereo-monocular XFactor 4 NVIDIA H200 GPUs, taking approximately 15 hours to reach 100,000 iterations. Multi-view XFactor is trained on NVIDIA H200 GPUs, taking approxi- mately 48 hours reach 100,000 iterations.
The pose probe is a three-layer MLP with feature dimension of 256. The probe is trained for 10,000 iterations with batch size of 64. The experiments differ slightly between the comparisons in Sec. 4.2 and ablations in Sec. 4.3. In the comparisons, the probe is used to predict five poses trajectory, and a scale-invariant loss is applied between the predicted and ORACLE-extracted trajectories. In the ablations, the probe is used to predict the relative pose between two frames and the loss is computed directly between the predicted pose and that of the ORACLE.
Autoencoding Reconstruction. Here we report results on autoencoding reconstruction for each model. This is simply the ability to render target frames from sequence using scene and pose representations from the same sequence. While this work argues that this task is fundamentally not equivalent true NVS and purely measure of model's ability act an interpolator, is the principal evaluation benchmark used in RayZer and also provides a quantitative measure of reconstruction quality so we include it for completeness.
Autoencoding reconstruction results are shown in Table 4 in terms of standard perceptual metrics including PSNR. SSIM, and LPIPS averaged across sequences from each dataset. this setting XFactor and RayZer achieve good reconstruction quality. We note that our results here differ slightly from those in the original paper. This is largely because we trained models with a unified loss of T with LPIPS, whereas the original RayZer uses T with custom VGG perceptual loss. This change improves LPIPS but lowers PSNR/SSIM, since we explicitly penalize LPIPS but not La error. Additional discrepancies stem from training RayZer on our large-scale, mixed scene- and object-level dataset and using only 5 context views for fair comparisons across methods; the original RayZer trains separate models per dataset and uses more context views for several settings. These differences make our evaluation more demanding and naturally produce performance shifts.
Perceptual Measure of Transfer Rendering. To get fuller picture of transfer rending qual- ity, we evaluate multi-view XFactor by applying our pose-presse pnse-preserving augmentations inference. Specifically, given a single sequence we apply our augmentations to generate two new sequences, extract the pose latents from each, and render the each sequence's target views with the others pose latents.
The results are shown in Table 5 where compare the transfer renderings against the ground truth target frames in terms of standard perceptual metrics. The results are very close to those achieved in autoencoding experiments in Table 4, demonstrating that XFactor can transfer with high fidelity when the distribution of target poses is close to those of the context used for rendering. However, this approach cannot be used to provide a fair comparison with RayZer and RUST, since both models do not partially masked inputs during training and thus cannot be expected to well-handle them
14

---

Published as a conference paper at ICLR 2026
XFactor (Ours)
SimCLR (Chen et al.. 2020)
VICRey (Dundes et al.. 2022)
Metrie
RE10K
BL.DV
MVlnngNet
003Dv2
RE10K
DI.DVV
MVImggNet
CO3I22
RE10K
DLA3DY
MVImgNet
CO30v2
20" (1)
99.5
08.5
99.7
97.5
92.2
90.0
37.5
59.1
99.2
90.0
59.2
202 (T)
79.4
89.3
96.1
77.4
12.5
76
1.5
4.5
2.6
0.0
0.1
1.5
AUC 20"
54.8
61.2
71.5
50,3
388
23
0.5
0.8
0.8
0.0
0.0
34
Table 6: Comparison with Supervide Objectives: Pose Probe Accuracy. Probe accuracy for representations produced by training with our transferabilityy objective (results copied from Ta- ble 3), a contrastive SimCLR (Chen et al., 2020) objective, and a mutual-information based VI- CReg (Bardes et al., 2022) objective.
XFactor (Ours)
Temporal Shift +11
Temporal Shift +-2
Metrie
REIOK
DI.DDV
Ne
CO3Dv2
RE1OK
DILDD
MVImgNer
CO3Dv2
RE10K
DL.DDV
MVImgNet
CO3Dv2
Transfe
R 2.200 (T)
99.9
98.36
96.5
98.2
99.9
96.4
92.1
97.65
99.8
96.1
91.0
97.5
T 207 (T)
75.1
76.6
84.1
33.05
72.4
66.4
70.8
25.8
71.3
63.0
66.8
23.8
AUC
47.2
44.8
49.3
14.6
41.6
36.2
36.1
11.1
32.7
29.2
9.38
Frobe
8.200 (1)
99.5
98.5
99.7
97.5
99.2
96.9
95.4
96.0
97.4
94.4
T.207 (T)
79.4
89.2
96.1
77.4
78.4
92.2
67.4
67.6
74.6
91.3
65.7
ALIC 20" (t)
54.8
61.2
71.5
50.1
44.6
47.0
55.7
37.5
38.8
40.3
55.2
36.6
Table 7: Robustness of Transferability Objective. We progressively corrupt our stereo-monocular transferability objective by forming pairs of sequences by sometimes applying a temporal offset to pairs of frames, instead of via our proposed augmentation strategy. XFactor results are copied from Table 3.
inference. Continuing improve TPS and to comprehensively evaluate transferability remain important outstanding problems.
Comparison Against Self-Supervised Learning Objectives. We compare training POSEENC with our transferability objective to popular existing self-supervised objectives. To do so, two train versions of our POSEENC module without the transferability objective: In the first, we train with a contrastive SimCLR (Chen al., 2020) and in the second we train with the mutual- information based VICReg (Bardes et al., 2022) objective. In both cases, we feed two augmenta- tions POSEENC produce pairs of positively corresponding pose latents. For the SimCLR-style objective, for each positive pair in the batch, we use all other pose latents as negative examples to compute the InfoNCE with temperature of T 0.1. For the VICReg objective, the variance and covariance are computed over the batch. We do not train a RENDER module because we do not use the transferability objective. These models are otherwise trained in the same manner as is done with all other ablations.
To compare pose representations, we train a pose probe and report the probe accuracy in Table 6. We find that the SimCLR and VICReg objectives perform far worse than our transferability objective (and also all other design ablations in Table 3), almost completely failing to extract meaningful pose information. This suggests that rendering may be a critical facilitator of the emergence of geometric reasoning. Despite these results, we believe our method can still be improved by incorporating ideas from previous work self-supervised learning, and will pursue this in future work.
Robustness of Transferability Objective. We perform additional experiments to evaluate the ro- bustness of the transferability objective. We observe that for any pair of frames {I, I,} with tem- poral indices {i,}}.
f. it is likely that the relative viewpoint change between {4, I2} will be similar to
the change between frames (1++1, II+1]. though not exactly equal. Similarly, the viewpoint change between 2 [ may also be similar, but less so. Thus, we train two new versions of our stereo-monee monteo--onocuaar model by progressively corrupting the transfcrability objective with pairs of im- ages whose relative poses are similar, but imperfect. In the first experiment, for each example in the batch, there is an equal chance that pairs of sequences will be generated by our proposed augmenta- tion scheme or by shifting the input frames by temporal offset of one frame - e.g. I {T, 1 and 1 {1+11, I5+1]. }. In the second, we increase the degree of corruption, and there exists an equal chance pairs will be generated by either our augmentation scheme, a temporal shift of 1, or a temporal shift of 2. These models are trained in the same manner all other ablations.
15

---

Published as a conference paper at ICLR 2026
(a) Weak Distortion
(b) Heavy Distortion
Corrupted Reference
Corrupted Reference
VGGT COLMAP
VGGT
Qualitative Robustness Analysis of TPS Oracles. We evaluate the robustness of VGGT (Wang et al., 2025b) and COLMAP (Schönberger et al., 2016; Schönberger & Frahm, 2016b) oracles against visual corruptions. (a) VGGT is effectively invariant minor distortions, while COLMAP quickly diverges. (b) VGGT maintains robustness even under significant random distor- tion, whereas COLMAP fails to register poses entirely.
Figure 2:
VGGT Oracle
COLMAP Oradle
Metric
RE10K
DL3DY
MVImgNet
CO3Dv2
REIOK
DL3DY
MVImgNet
CO3Dv2
Ace
10"
100.0
100.0
100.0
99.2
86.3
96.4
88.9
91.2
20
100.0
100.0
100.0
99.6
94.3
99.0
95.9
97.4
(T)
30
100.0
T. Ace
10"
87.7
97.0
96.9
82.5
38.7
71.5
58.6
44.3
(t)
20"
93.6
99.5
99.7
91.1
57.1
85.0
76.1
64.3
30
95.7
99.9
99.9
93.7
69.7
90.0
84,1
74.1
AUC
10"
70.4
80.5
74.5
61.1
20.2
46.9
35.7
24.3
(1)
20
80.8
89.4
74.4
51.7
30
85.4
92.8
91.1
80.4
43.6
71.5
61.2
49.7
Success Rate (%)
100.0
100.0
100.0
100.0
28.9
52.3
21.4
32.0
Table 8: Quantitative Robustness Analysis of TPS Oracles. We evaluate the robustness of the VGGT (Wang et al., 2025b) and COLMAP (Schönberger et al., 2016; Schönberger & Frahm, 2016b) oracles against visual corruptions by mea- suring the TPS between reference and corrupted videos. A weak distortion, visu- alized in Fig. 2 (a), is applied to the video. We find that the VGGT oracle is sig- nificantly more robust across all datasets and metrics. In contrast, the COLMAP oracle suffers from an excessive rejection ratio, diminishing the utility of the eval- uation samples.
The results shown in Table 7. In terms of transferability, our objective proves remarkably robust. TPS falls off only slightly when corruption is first introduced and remains relatively stable as cor- ruption increases while still outperforming all other ablative design decisions (Table 3) terms of AUC @ 20". However, probe accuracy degrades to greater extent, suggesting imperfect viewpoint correspondence between sequences hampers the emergence of geometric reasoning.
Robustness of TPS Oracles. Any pose estimator can be used as a TPS oracle. but the evaluation quality may vary considerably based on this choice. To assess this variation, we compare the robust- ness of the VGGT oracle against the traditional SfM baseline, COLMAP (Schönberger et al., 2016; Schönberger & Frahm, 2016b) by using each method to compute the TPS between given video and a corrupted version of the same video. We find that VGGT consistently yields a more robust TPS evaluation, as demonstrated qualitatively in Fig. 2 and quantitatively in Table 8. Additionally, we evaluate the COLMAP oracle on the actual transferred videos, with results provided in Table 9. In particular, the baseline RayZer (Jiang al., 2025) suffers from excessively high rejection rate, making it impossible to reliably measure its transferability.
Reconstruction Artifacts. As shown in the visualizations in Tab. and the videos on our web- site?, XFactor's reconstruction is imperfect when rendering wise-baseline, out-of-sight, or out-of-
httpps:/ www. mi mitchel chc.chhel.ccomptte///faccooo
16

---

Published as a conference paper at ICLR 2026
In-disriibution, Interpolation Targets (INSM)
(b) Out-af-distribationn, Extrapolation Targets (LVVSM)
Figure 3: Supervised Methods Also Struggle. Substantial reconstruction artifacts are also observed in supervised models when rendering difficult target views. The first and last frames used as context views. (a) The supervised baseline, LVSM (Jin et al., 2024), excels at reconstructing easier target views. (b) However, LVSM struggles cleanly render difficult target views, showing blurry or warped artifacts in the reconstruction.
XFFcctor (Ours)
RayZer (Jiang al., 2025)
Metric
RE10K
DL3DY
MVImgNet
CO3Dv2
RE10K
DL3DY
MVImgNet C03Dv2
Success Rate
52.1
59.1
17.7
38.5
9.4
16.1
0.1
7.7
Table 9: Fragility of COLMAP on Transferred Videos We find that COLMAP is highly fragile when applied to transferred videos. In particular, samples from the baseline method, RayZer (Jiang et al., 2025), exhibit significantly higher rejection ratio, preventing meaningful measurement of transferability with TPS.
distribution viewpoints. For instance, renderings of these difficult viewpoints often exhibit blurry or warped artifacts. However, demonstrate that the supervised baseline, LVSM (Jin et al., 2024), suffers from similar issues, as illustrated Fig.
17


Reviews:
[{"review_id": "5IqcB14gIm", "reviewer": "Reviewer_QYSv", "summary": "This paper tackles self-supervised novel view synthesis by defining pose latent transferability as the key criterion for success. The proposed model, **XFactor**, achieves this through a geometry-free, self-supervised approach that combines pairwise pose estimation with curated data augmentations. By training on real-world videos without explicit geometric biases, XFactor learns transferable pose latents. To measure this capability, the authors introduce a new metric, True Pose Similarity (TPS). Experiments show that XFactor outperforms prior NVS transformers like RayZer and RUST on large-scale benchmarks. Furthermore, analysis confirms that its learned pose latents meaningfully correlate with actual camera poses, validating the approach.", "questions": "1. Are there plans for open-sourcing code and pretrained models to facilitate reproducibility？\n\n2. Minor issue: The paper defers critical details of the training procedure to the appendix. For instance, the exact loss function is not defined in the main methodology. The description of the dual-masking augmentation is also incomplete, as it omits the probability of the no-masking case, making it unclear how frequently the transfer objective is applied during training.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. This paper is well-written and has clear results/figures to present the results.  Datasets and baselines are well-described.\n\n2. This work defines \"transferability\" as a discriminative property for novel view synthesis (NVS) and introduces a new metric (TPS) for its quantification. It argues that view reconstruction alone is insufficient for genuine NVS without this property.\n3. The proposed model, XFactor, utilizes a pose-preserving, dual-masking data augmentation to disentangle the scene representation from the camera pose. This method achieves high transferability without relying on explicit geometric priors.\n4. Empirical results show that XFactor substantially outperforms baselines across multiple datasets. The paper supports its claims with a comprehensive evaluation, including pose-probing to validate the learned latent space and detailed ablation studies that provide insights into the model's design.", "weaknesses": "- The novel views generated by XFactor, as seen on the project's homepage, suffer from noticeable blurring and distortion, even when the camera trajectory is accurately recovered. The paper would be stronger if it proposed a method to address these rendering imperfections. While the authors mention that blurring and warping occur for distant poses, a more in-depth analysis of these artifacts would add important nuance to the empirical results and better guide subsequent research.\n- The paper's experimental validation would be more convincing if it included a comparison against other self-supervised learning objectives, such as those based on contrastive learning or mutual information. Additionally, the evaluation of the ViT-based architecture is incomplete without benchmarking it against more recent paradigms, including video diffusion models like ReCamMaster, which have shown strong performance on related tasks.\n- The current experiments are limited to well-controlled datasets, leaving the performance of XFactor in more challenging, unconstrained real-world environments an open question. The paper does not provide insight into how the method would handle scenes with dynamic objects or significant photometric variations, such as changes in lighting or the presence of motion blur, which are common in real-world video.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vhWeTwU8ga", "reviewer": "Reviewer_ZciQ", "summary": "The paper argues that the correct criterion for self-supervised NVS is transferability: a pose representation extracted from one sequence should produce the same camera trajectory when used to render another scene. The authors show that recent pose-free/self-supervised methods (RayZer, RUST) learn interpolation latents that fail this test. The authors propose XFactor, a geometry-free model using pairwise pose estimation, stereo-monocular bootstrapping, and pose-preserving augmentations (e.g., inverse masks) to disentangle pose from content. The authors also propose a new metric named True Pose Similarity (TPS) quantifies transferability using oracle poses.\n\nExperiments on datasets like RealEstate10K, DL3DV, MVImgNet, and CO3Dv2 demonstrate XFactor's superior transferability and pose probing accuracy, with ablations validating design choices.", "questions": "1. How might TPS change if the authors evaluated it with COLMAP alongside VGGT. How sensitive would TPS be to small perturbations (translation/rotation jitter) in the oracle poses?\n\n2. Could a generative decoder plausibly reduce the blurring/warping seen at large baselines without degrading pose control — and if not, what trade-offs are expected between realism (e.g., FID/LPIPS) and pose fidelity (TPS)? Which probabilistic design choices seem most likely to preserve pose conditioning?\n\n3. What are resource costs: expected latency and peak GPU memory for rendering one target view on common GPUs, and how latency and memory scale as the number of context views increases?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "The paper is insightful, which reframes NVS with the *transferability*. This is a fresh perspective, and the proposed TPS metric is a practical tool for evaluation the transferability of an NVS model.\nQuality of this paper is strong, with large-scale experiments on diverse real-world datasets supporting claims of transferability; ablations thoughtfully dissect components like stereo-monocular training and augmentations. Paper presentation is quite clear, from problem analysis to method derivation, making complex ideas accessible.", "weaknesses": "1. The proposed TPS relies on an oracle (VGGT used in current experiments). VGGT is reasonable, but VGGT vs more conventional SfM (e.g., COLMAP) can differ—especially on difficult sequences (textureless regions, non-static content). The paper does not report robustness of TPS to oracle errors (noise, scale drift, missing frames), nor does it show if conclusions change with a different oracle.\n\n2. The authors note blurring/warping for large baseline transfers and suggest this stems from determinism. This is an important limitation for practical use and should be explored more: does a generative renderer (diffusion model) fix artifacts? What are tradeoffs with photorealism vs pose fidelity?\n\n3. The practical cost at inference (latency/memory) for rendering many target poses or long sequences is not reported.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "h9tHZg6ETP", "reviewer": "Reviewer_VrW1", "summary": "This paper addresses the problem of self-supervised novel view synthesis (NVS). The authors make a crucial and insightful observation: the key criterion for a \"true\" NVS model is transferability, i.e., a representation of a camera pose extracted from one video sequence should be usable to render the exact same camera trajectory in a completely different scene. They argue that prior self-supervised, geometry-free methods fail this test, as their learned \"pose\" representations are entangled with scene content, leading them to act as sophisticated frame interpolators rather than genuine view synthesizers.\n\nTo address this, the paper introduces XFactor, a novel self-supervised NVS model designed explicitly to learn transferable pose representations. The authors quantify transferability by proposing TPS (True Pose Similarity), and prevent the model from learning to interpolate by making XFactor bootstrapped from a stereo-monocular model and trained with a pose-preserving data augmentation strategy. Experimental results demonstrate the effectiveness.", "questions": "1. The reported performance of the RayZer baseline in Table 4 appears to be a significant degradation when compared to the results presented in the original RayZer paper (Figure 1). I recognize and appreciate that the authors had to re-implement this baseline due to the lack of public code, and commend them for seeking confirmation from the original authors that their implementation is \"faithful\" (lines 312-314). However, the substantial performance gap is confusing. Are there key differences in the training dataset, evaluation protocol, or hyperparameter settings compared to those used in the original RayZer publication?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper asks a fundamental question in the field of self-supervised NVS. The proposed \"transferability\" is a powerful and original reframing of the problem. The paper is well-written and well-motivated. The design of the proposed solutions is elegant and principled, with its effectiveness validated by extensive experiments. Through the lens of transferability, the paper provides valuable insights to the community of self-supervised NVS.", "weaknesses": "I appreciate the paper's main idea and find its core contribution to be insightful and significant. While the work is very strong, I have some concerns, primarily regarding the evaluation methodology for what the paper defines as a \"true NVS model\".\n\n\n### 1. The Challenge of Evaluating a True NVS Model\n\nMy main concern is how to fairly and comprehensively evaluate NVS performance under the new, important definition of \"true NVS model\". While I agree that standard autoencoding metrics like those in Table 4 are biased, I am not convinced that the proposed True Pose Similarity (TPS) metric (Eq. 9), used in Tables 1-3, provides a complete picture. My reasons are as follows:\n\n**1.1. Assessing Perceptual Quality in the Transfer Setting:** The TPS metric evaluates the *geometric consistency* of a transferred camera trajectory but does not directly measure the *perceptual quality* of the synthesized images. The paper itself compellingly argues that autoencoding reconstruction (seq-to-seq rendering on the same scene) is \"not equivalent to true NVS\" (lines 641-642) and is merely a measure of interpolation ability. This creates a critical gap in the evaluation: we can see from the TPS scores in Table 1 & 2 that XFactor is geometrically consistent, but we have no quantitative perceptual measure of the actual view synthesis quality, which should be a key point for NVS methods. This leaves the assessment of the model's core capability incomplete.\n\n**1.2. Fragility of the Oracle-based Metric:** The TPS metric's reliance on a post-hoc oracle (VGGT) makes it potentially fragile. As the paper notes (lines 199-202), the metric could be \"hacked\". More practically, it can fail in challenging extrapolation scenarios. For example, consider transferring a forward-motion trajectory from a driving scene (A) to a scene featuring an ego-centric rotation of an object (B). A model might produce meaningless, blurry artifacts for $\\text{Render}(\\mathcal{S}^B, \\mathcal{Z}_T^A)$, as it is a difficult extrapolation synthesis task (line 474-477). In such a case, an oracle like VGGT would likely fail to extract any meaningful pose from these artifacts, leading to a poor TPS score. This low score, however, would be an artifact of the evaluation tool's failure, not necessarily a failure of the model's transferability logic.\n\n**1.3. Inherent Noise in Oracle Tools:** It is well-documented that structure-from-motion oracles like COLMAP, and the feed-forward methods like VGGT, are not perfectly reliable and can produce noisy or incorrect poses. This issue has been analyzed in prior work such as RayZer [1] (arxiv version, Figure 4) and Less3Depend [2] (arxiv version, Appendix F). This unreliability introduces noise into the evaluation, making it difficult to disentangle the model's errors from the oracle's errors.\n\nGiven these points, I wonder if a well-defined perceptual measurement, independent of TPS, is necessary to complete the evaluation. For example, leveraging the **Pose-Preserving Augmentation** framework (Eq. 11) at **test time**. By applying two different augmentations to the same source sequence, the model could render a target frame from the second augmented sequence using the pose from the first. The resulting image could then be directly compared to the ground-truth target using standard perceptual metrics (PSNR, LPIPS, etc.). This would provide a direct, quantitative measure of rendering quality under the context of true transferability, which might complement the TPS metric.\n\nI believe a discussion of these evaluation limitations in the main paper or appendix would be highly beneficial. On this point, I am open to hearing the authors' clarification and am highly willing to raise my rating if this concern is addressed.\n\n### 2. Potentially Related Work\n\nThe paper astutely identifies the ill-posed nature of the autoencoding objective for NVS. This issue has also been noted by [3], which also addresses this problem, but proposes an alternative solution that involves leveraging explicit 3D representations for fine-tuning. Including a discussion that contrasts XFactor's geometry-free, transferability-based approach with this alternative 3D-aware solution would further strengthen the paper's contribution and contextualize it within the latest research landscape.\n\n\n---\n\n\n[1] RayZer: A Self-supervised Large View Synthesis Model (ICCV'25)\n\n[2] The Less You Depend, The More You Learn: Synthesizing Novel Views from Sparse, Unposed Images without Any 3D Knowledge (arxiv)\n\n[3] Recollection from Pensieve: Novel View Synthesis via Learning from Uncalibrated Videos (arxiv)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3tyXjlLwRA", "reviewer": "Reviewer_9vfk", "summary": "This paper aims to define \"true\" self-supervised novel view synthesis (NVS) by introducing transferability as the key criterion. The paper defines transferability as the ability for a pose representation extracted from one video to render the same camera trajectory in a different scene. The paper further argues that prior generalizable, self-supervised NVS methods fail this test, as they learn to interpolate context frames rather than reason geometrically.\n\nTo solve this challenge, the paper presents XFactor, a model claimed to be \"geometry-free\" and capable of this \"true\" NVS. XFactor's design is based on two main ideas: 1) bootstrapping from a stereo-monocular (two-view) model to force extrapolation rather than interpolation, and 2) a new transferability objective that uses pose-preserving augmentations (e.g., inverse masking ) to disentangle pose from scene content. The authors also introduce a new metric, True Pose Similarity (TPS), to quantify this transferability. Experiments show XFactor significantly outperforms re-implementations of prior work (RayZer, RUST) on this new metric.", "questions": "Given that your criterion of \"transferability\" does not apply to single-scene NVS (e.g., NeRF), will you concede that your paper is not about \"true NVS\" in general, but rather about the more specific sub-problem of generalizable NVS from unposed video?\n\nYou state your goal is a \"user-controllable viewpoint\". How can a user control the 256-dim latent pose vector of XFactor without first having a source video with the exact camera motion they desire? Isn't this a \"pose-copying\" system, not a \"pose-controlling\" one?\n\nYour training objective simulates two sequences with identical poses. How brittle is the learned representation to this perfect \"pose-matching\" assumption? Have you analyzed the performance degradation when the source and context trajectories are not identical, but merely similar?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1. The paper identifies that previous generalizable self-supervised models (like RayZer and RUST) are prone to interpolating context frames when trained with a simple autoencoding objective, which is a valuable insight.\n\n2. The transerability objective, enabled through the pose-preserving augmentation scheme, is a novel and original formulation for a self-supervised task. It provides a new way to force the model to learn a representation that disentangles the camera motion from the appearance. \n\n3. The use of a stereo-monocular model to force extrapolation and avoid the interpolation failure mode is a clever, though costly, design choice.\n\n4. The paper's internal logic is sound. The experiments rigorously support the paper's central, albeit narrow, claim: that the XFactor model, trained with its specific transferability objective, is superior at the task of pose-trajectory transfer, as measured by the proposed TPS metric. The ablations (Table 3) are particularly strong, effectively demonstrating that the novel objective is the key ingredient for this success, and that adding multi-view information from scratch is detrimental to this specific goal. The counterintuitive finding that an explicit $SE(3)$ bottleneck is harmful is genuinely interesting.\n\n5. The paper is exceptionally well-written, clear, and logically structured. The authors do an excellent job of motivating their specific problem formulation, defining their terms (transferability, TPS), and explaining their method (XFactor). Figure 1 provides a very clear illustration of the novel training objective.", "weaknesses": "1. The paper's central premise—that \"transferability\" is the key criterion for \"true NVS\" —is an overstatement. The paper defines \"true NVS\" in a way that conveniently excludes the most dominant NVS paradigms, such as single-scene optimization (NeRF, 3D Gaussian Splatting), which the paper dismisses as \"Oracle Methods\". The work's scope is limited to generalizable, unposed video models, and the abstract and introduction must be revised accordingly.\n\n2. The paper equates \"transferability\" with \"user-controllable\" NVS. The proposed XFactor model does not achieve this, though. A user cannot specify a novel viewpoint; they can only copy a viewpoint from another existing video sequence. The learned 256-dimensional latent pose is a black box. This is a \"pose-copying\" system, not a \"pose-controlling\" one, which is a significant gap between the stated problem and the delivered solution.\n\n3. The claim of being \"geometry-free\" is a bit misleading in my opinion. The entire transferability objective is built on a strong, expert-provided geometric prior: that augmentations like masking are pose-preserving, a fact the authors even formalize as $ORACLE[AUG[\\mathcal{I}]] = ORACLE[\\mathcal{I}]$. This is not a \"pure machine learning problem\"; it is a clever re-encoding of a geometric bias into the training data.\n\n4. The paper compares XFactor to re-implemented baselines (RayZer, RUST)  on its own novel metric (TPS), a metric for which XFactor is explicitly optimized. These baselines, unsurprisingly, fail spectacularly. The paper's own ablation (Table 3, \"Unconstrained\" baseline) shows that a standard stereo-monocular model with an autoencoding objective doesn't do much worse. This shows the win is not from a superior architecture but mainly from a training objective that is perfectly aligned with the evaluation metric, making the comparison to prior art feel circular and a bit unfair.\n\n5. The model's reliance on a stereo-monocular POSEENC to avoid interpolation is a major tradeoff. It sacrifices the core strength of multi-view 3D reconstruction: robust integration of information across many frames. This design choice may solve the interpolation proble,m but it creates a model that is likely less robust and has a poorer understanding of 3D geometry than a true multi-view system.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)