Cameras as Relative Positional Encoding
Authors: Ruilong Li
Brent Yi
Junchen Liu
Hang Gao
Yi Ma
Angjoo Kanazawa
Venue: NeurIPS 2025 poster ()
Keywords: multi-view transformer; positional encoding; projective transformation
Primary area: applications
TL;DR: Camera conditioning as relative projective transformation in multi-view transformer

Abstract: Transformers are increasingly prevalent for multiview computer vision tasks, where geometric relationships between viewpoints are critical for 3D perception. To leverage these relationships, multiview transformers must use camera geometry to ground visual tokens in 3D space. In this work, we compare techniques for conditioning transformers on cameras: token-level raymap encodings, attention-level relative pose encodings, and a new relative encoding we propose—Projective Positional Encoding (PRoPE)—that captures complete camera frustums, both intrinsics and extrinsics, as a relative positional encoding. Our experiments begin by showing how relative conditioning methods improve performance in feedforward novel view synthesis, with further gains from PRoPE. This holds across settings: scenes with both shared and varying intrinsics, when combining token- and attention-level conditioning, and for generalization to inputs with out-of-distribution sequence lengths and camera intrinsics. We then verify that these benefits persist for different tasks, stereo depth estimation and discriminative spatial cognition, as well as larger model sizes.

Cameras as Relative Positional Encoding
Ruilong Li
Brent Yi!" Junchen Liu'*
Hang Gao
Yi Ma Angjoo Kanazawa
UC Berkeley
NVIDIA
HKK
Abstract
Transformers are increasingly prevalent for multiview computer vision tasks, where geometric relationships between viewpoints critical for 3D perception. To leverage these relationships, multiview transformers must use camera geometry ground visual tokens 3D space. In this work, we compare techniques for conditioning transformers on cameras: token-level raymap encodings, attention- level relative pose encodings, and a new relative encoding- Projectvee Positional Encoding (PRoPH) that captures complete camera frustums, both intrinsics and extrinsics, as relative positional encoding. Our experiments begin by showing how relative conditioning methods improve performance in feedforward novel view synthesis, with further gains from PRoPE. This holds across settings: scenes with both shared and varying intrinsics, when combining token- and attention-level conditioning, and for generalization to inputs with out-of-distribution sequence lengths and camera intrinsics. We then verify that these benefits persist for different tasks, stereo depth estimation and discriminative spatial cognition, as well as larger model sizes. Code is available our project webpage
Introduction
Images of our world exist in the context the viewpoints they were captured from. The geometry of these viewpoints-ing inttissiicniii and extrinsic parameters that give pixel coordinates their physical meaning-grovee visual observations in 3D space. This spatial grounding is increasingly important in deep learning, especially as advances in 3D vision and embodied intelligence make multiview tasks more ubiquitous.
To solve multiview tasks with transformers, models must bind viewpoint information to patch tokens from each input image. This binding requires special care: just as naive positional encoding techniques for ID sequences hinder performance for learning in language models []], naive encodings
of camera geometry may also suboptimal for multiview vision models [2-4]. Advances both settings can be summarized as transitions from absolute [5 to relative (6] encodings.
In this work, we study the problem of conditioning vision transformers on the camera geometry of input images. We survey existing techniques for addressing this, which include (i) absolute encodings in the form of pixel-aligned, token-level raymaps- raymaps-thhse these are the most common in recent state-of-the art models 7+100 and (ii) attention- level relative encodings based on SE(3) pose relationships 3 4. We then present new camera conditioning technique, Projective Positional Encoding (PRoPE), that is designed to capture the complete geometry of cameras as relative positional encoding. PRoPE models viewing frustum relationships that describe both intrinsics and extrinsics, while remaining easy to incorporate with standard transformer architectures and fused attention kernels [I]
Our experiments are on three tasks, which span six datasets. We begin with series of studies compar- ing camera conditioning techniques for feedforward novel view synthesis using RealEstate 10K [12 and Objaverse [13]. Our results highlight the advantages relative encodings- particuullay PRoPE compared to absolute ones. We then verify that these benefits extend to other settings: we demonstrate improvements when integrating PRoPE into UniMatch [14 for stereo depth estimation across three
Equal contribution
https://www htap::://www.liruioorg wwwpnng.n///roo sa/prope/
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Absolute Positional Encoding (APE)
Relative Positional Encoding (RPE)
LLM
cats
cats
1
2
Distance
Multiview Ceometry
,,
(K,, )
(Ks, )
Transformation
Figure I: Cameras as relative positional encoding. Language models and multiview transformers must both bind "positional" information to input tokens, in terms of sequence position for language and camera parameters for multiview computer vision. We present study on camera conditioning that includes absolute positional encodings (raymaps), relative pose encodings [ 4 and new method captures relative projective relationships between more complete camera, frustums.
benchmarks, for discriminative spatial cognition task using DL.3DV novel view synthesis models
15 and when scaling to larger
The contributions of this paper are as follows:
(1) Survey. We survey both absolute raymap and relative SE(3) conditioning techniques for camera geometry multiview transformers.
(2) Method. We propose PRoPE (Projective Positional Encoding), a new relative positional encoding technique that injects both camera intrinsics and extrinsics into transformer's self atffetttenton blocks.
(3) Evaluation. We present a series of novel view synthesis (NVS) experiments that compare camera conditioning techniques empirically. Our results highlight the advantages of relative pose encoding methods like CAPE [4] and GTA [3]. while demonstrating further improvements from PRoPE across range of settings: scenes with shared intrinsics, scenes with varying intrinsies, for hybrid conditioning that combines both token-level and attention-level representations, and in generalization for -iitt--ddittiittiinn test inputs.
(4) Task generalization. We show that the benefits of cameras as relative positional encoding generalize (i) to stereo depth estimation when integrated into UniMatch, (ii) to discriminative spatial cognition, and (iii) when scaling to larger model sizes.
2 Related Work
Absolute and relative positional encodings. Transformer architectures are permutation-invarian they therefore require explicit position encoding to understand token order in sequential inputs (5]. Position encoding in sequence models has been an active area of research [6] While early works (20,27 focused on absolute positional encoding (APE), recent methods have increasingly adopted relative positional encoding (RPE), particularly RoPE (28), as a standard across domains,
including natural language processing (29 31 and computer vision [32]44] 28 (35) Relative
encodings aim to improve models by defining positions as relative offsets between token pairs. These offsets injected into the pairwise interactions of standard dot product attention [5]:
QK
Attn Q, K,V) softmax
V.
(1)
Vd
where Q, K. E RT'd Position offsets can be injected using the pairwise nature of the QK E RT matrix, either via additive biases (. 36 37 or SO(2)-based rotation (]. RPE offers important advantages over APE, including translation invariance, improved relationship mod- cling, and generalization to long sequences [38] 39.. In this work, we study both absolute and relative encodings for conditioning transformers camera geometry instead of ID position.
Multiview transformers. Many computer vision tasks are multiview: they take multiple images and known camera geometry for each image as input. Examples exist in 3D reconstruction and view synthesis 44) pose estimation [45]. depth prediction [14 46 47), 3D scene understanding [48
49), robotics 50 and world models [51]. Many recent works leverage the improved scaling
54 These models slice input images into patches, and use each patch an independent visual token for the transformer. In this work, we use the model designs proposed by LVSM [8] and UniMatch [14 as a starting point for studying critical design decision how transformers conditioned camera geometry.
properties 222
52 53 vision transformers for solving these tasks 7

---

Camera conditioning in transformers. The dominant approach for conditioning multiview trans- formers on cameras currently raymaps 45 7 10: per-pixel 6D embeddings that contain either ray origins and directions |4 7 or Plücker coordinates
Concatenating these parameters to pixels enables conditioning on both camera intrinsics and extrinsics at the token level. Raymaps, however, require defining a frame of reference [56] 7 [7]. which is problematic because the choice of world coordinate system is arbitrary and can hinder generalization. While this problem can be partially addressed by normalizing poses [58
55 45
57 g prior works have shown that a more fundamental
fix is possible through relative parameters 2 59 Notably, attention-level encodings that capture relative SE(3) poses don't ton't require defining a consistent global frame, are compatible with fused attention kernlls (IT), and have been shown to improve novel view synthesis performance [ 4 In the following section, survey both absolute raymap and relative SE(3) methods for encoding camera geometry for transformers. We then propose a new relative encoding method, PRoPF, that encodes relationships between more complete camera frustums.
Conditioning Transformers on Cameras
3.1 Preliminaries
We study transformers that take N images from known cameras as input:
((I, K; TT" )
(2)
where each I; E H an image, K; 33x3 is the camera intrinsics, and TTE (R,". t(=) SE(3) is the transformation for computing camera coordinates from world coordi- nates. The latter two terms encode the viewing frustum that corresponds each image: intrinsics capture the shape and Reld-of-view of the frustum, while extrinsics capture position and orientation. Both intrinsies and extrinsics are encapsulated in the "world-to-iman projection matrix P; E R3x4,
P: K; 03x1 T
For notational convenience, these 3 x projection matrices can be made invertible by lifting x with the standard basis vector (0, 0, 0, 1) This transformation maps 3D world coordinates to projective image space defined by the frustum of camera i. It can be used compute 2D image coordinates from world coordinates:
P
P.Xwordd
where x; e 3 and Xworld I+ are homogeneous coordinates in the image and world respectively. The inverse relationship can used compute ray directions in 3D space from 2D image coordinates. For homogeneous image coordinate x.. (u, v, L)T
[ad u0
(5)
where a E IR is a scalar magnitude and d E S2 is unit-norm ray direction.
3.2 Pixel-aligned Camera Encoding
Token-level, pixel-aligned raymaps the dominant method for encoding geometry in multiview transformers 7
42 10 [9. Networks that employ raymaps concatenate images I; E H x3 with per-pixel raymaps M, E R W x along the channel dimension, which expands inputs to R xx There two main approaches for computing these raymaps, which refer as "naive" and Plücker.
Naive raymaps. Naive raymaps are computed as per-pixel origin and direction vectors:
a i,Nivaa
O; for (u, [1, W [1,
R
(6)
d.
0 - (R
T
(7)
where each ray direction d 2 computed using P. by following Equation 5

---

Plücker raymaps. Plücker raymaps 45 8 raymaps with moment term:
can be implemented by replacing the origin term in naive
0, x d. d.
M
dhe Plucker
R for (u, v) E [1. x []
(8)
This moment term makes the ray representation invariant to the choice of origin along the ray.
Properties. Raymaps offer a simple approach for conditioning on both camera intrinsics and extrinsics. An important drawback, however, is that they are absolute: similar to early position encoding techniques for ID sequences [5], raymaps are expressed in global terms. They are sensitive to the arbitrary choice of reference frame, which can hinder generalization.
3.3 Relative SE(3) Encoding
To remove the need for global reference frame, recent works have introduced relative encodings for SE(3) camera poses. Two existing approaches fall under this category: CaPE [4] and GTA (3). Given images i and ig, both aim to condition networks on Ta (T%, iz using modified self-attention blocks. This captures dense relationships-hhvv each camera pose situated relative to every other camera pose-and makes networks invariant to how the world frame is defined.
Notation. To formalize the operations required for attention-level geometry encoding, we denote the batched matrix-vector product @, Kronecker product 8, and identity matrices I E R We use i for image/camera indices and for patch/token indices. i(l) is the index the image that patch t belongs to. Rows of the Q, K, E Rxxd matrices are subscripted Qt K2, Vg E R Batched matrix-vector products are defined as:
E IR Nxd, xd2 E ERR Nxd2 = (A @ B) Rxd.
(9)
B) Σ A Anzk Brk
(10)
We use D 1xxxd to denote batches of block-diagonal matrices, where individual matrices in D Di...., Dr] are subscripted D, E Rddd
CaPE []. CaPE injects relative SE(3) pose by transforming the Q and K matrices before they passed to self-attention. CaPE can be formalized using per-token block-diagonal matrices D CapE which is computed by diagonally repeating the camera extrinsics:
CaPE
La/1 6 T(())
(11)
Like RoPE (.) transformations then applied to the Q and K matrices before self-attention. This can be encapsulated into an augmented self-attention block:
Attn (Q,K, V) Attn( (D CaPE)
QD - - K,V..
(12)
The effect of this is that each Q,, K₁ IR dot product in QK E BXT is replaced with:
Q D
CaPE
(D
(aae)---ke IR,
(13)
where Daae (
CaPE)
Ia/4
Tit
) thus conditioning outputs relative pose.
GTA B. GTA proposes a formulation for per-token transformations with a similar high-level goal as CaPE. GTA's attention variant transforms the Q and K matrices the same way, while proposing to also transform the V matrix:
Attn (Q, (Q,K, = D A @ @Q (D @ K, (D V).
(14)
This has the added benefit of injecting relative transformations into the attention operator's value aggregation. The attention layer output for each token t becomes
[Aun (Q, K, V,),,, = Σ o CTA (FAEDA (
(15)
f2
where 0t, softmax score computed from the transformed dot product (Equation GTA's experiments 3 compare several SE(3)-based formulations for D with and without the value matrix transformation. The best-performing methods include the value transform, and both SE(3) for camera pose and RoPE [I] for 2D patch position. Our experiments include GTA using these terms.

---

3,4 Projective Position Encoding (PRoPE)
We introduce new relative positional encoding method in our study, which we call PRoPE. The core observation of PRoPE is that the SE(3) poses considered by existing relative encoding techniques only a partial representation of camera geometry. Instead of relating each camera i and camera i2 with only their poses T and T PRoPE uses the projective relationship between full frustums:
P.. P
(16)
This 4 x 4 matrix can be interpreted as transformation between the local projective spaces defined by each camera: it encodes the complete geometric relationship between camera views. As we will see in Equation 20 it also retains the key global invariance property of SE(3)-based relative encodings.
To implement PRoPE, we define
of DeRoPE Rdxdd matrices and use GTA-style attention (Equation [4) to inject them into transformer blocks. We design these matrices to (1) encode frustum relationships between cameras- this uses the projective relationship in Equation [6 and (2) encode
relative patch positions within cameras-thin follows GTA 3 and uses RoPE terms. These goals are achieved with complementary submatrices, each with shape x d.
2"
D Proj
D PRoPE
(17)
DROPE
D Proj
Lafs (((t)
(18)
DROPE
RoPPaay
ER
(19)
RoPEa/4(20)
In these definitions, RoPEa/4(-) patch coordinates for token f.
constructs x rotary embeddings for (Iz, Me), which are the
3.5 Properties of PRoPE
PRoPE several important properties, which become more evident when we expand the projective transformation:
P- 11
K 0
T (T
K
0
(20)
0 1
Global frame invariance. Redefining the world frame equivalent to right-multiplying both Tee SE(3) terms, which is algebraically eliminated in Equation 20
Reduction to relative SE(3) attention. For cameras with identity intrinsics, Equation 20 reduces to the relative (3) transformations utilized in CAPE and GTA. These methods can be interpreted as case PRoPE where the intrinsic matrices are set identity.
Reduction to RoPE (28). Equation 20 evaluates to identity for patches from the same image. For these token pairs, D PROPE contains only the RoPE terms used by single-image vision transformers.
Experiments
The goal of our experiments to understand how camera conditioning PRoPE-impact the performance of multiview transformers. To accomplish this, we present experi- ments comparing encoding strategies under several task and evaluation conditions.
4.1
Experiment Setup
We include metrics for several camera conditioning Cechniques--aare -Nave and Plücker raymap encodings, CAPE 4.. GTA (3) and PRoPE. In our experiments, GTA refers to the SE(3)+S0(2) variation studied
by B where SO(2) refers to RoPE on patch positions. As discussed Section 3.4 PRoPE adopts the self-attention mechanism and RoPE combination proposed by GTA. The primary difference between PRoPE and GTA is therefore the use of relative projective relationships instead of relative SE(3) relationships between cameras.
Our core experiments evaluate camera conditioning techniques using feedforward novel view synthe- sis (NVS). NVS is ideal benchmarking task because it requires fine-grained geometric reasoning:
5

---

RealEstate 10K [2
Objaverse 13
Method
PSNRT
LPIPS]
SSIMT
PSNR1
LPIPS
SSIM1
Plücker Raymap
20.48
0.209
0.622
21.44
0.159
0.851
Naive Raymap
20.54
0.210
0.623
21.59
0.153
0.856
CAPE
4
21.11
0.234
0.656
19.68
0.220
0.827
GTA
22.51
0.164
0.707
23.70
0.104
0.879
PRoPE
22.80
0.146
0.725
23.70
0.104
0.879
Table 1: Novel view synthesis comparison, with constant intrinsies in each scene. We compare different camera conditioning approaches applied to the LVSM [8] framework.
RealEstate 10K [2
Objaverse LPIPS
Method
PSNRT
LPIPS!
SSIM
PSNR1
SSIMT
Plücker Raymap
19.89
0.327
0.608
21.43
0.177
0.852
Naive Raymap
20.56
0.301
0.629
21.00
0.191
0.845
CAPE 4
15.94
0.497
0.699
16.78
0.322
0.760
GTA 3
15.77
0,512
0,641
18.00
0.257
0.775
PRoPE
21.42
0.247
0.678
22.98
0.138
0.871
Table Novel view synthesis, with varying intrinsics in each scene. We compare the LVSM 8 model trained with different camera conditioning strategies, intrinsics-augmenter dataset variants.
models are trained to render scenes from target viewpoints, given only calibrated reference images and target camera parameters. We do this by reimplementing and training variants of LVSM [8), state-of-the-ar art novel view synthesis method that originally encoded camera geometry with Plücker raymaps. We train and evaluate separately on the RealEstate1OF [12] and Objaverse [[3] datasets. Scenes in RealEstate 10K are captured with constant intrinsics, but cameras vary between scenes. Objaverse renders use the same camera intrinsics across the entire dataset.
We take several steps to ensure fairness in evaluations. Models are trained in the same codebase, with matched hyperparameters and training steps. All main experiments use identical input, output, and overall model sizes (~25M parameters); we also validate larger models in Section 4.7 More details are provided Appendix A.I.I
4.2 Relative vs Absolute Positional Encodings
Novel view synthesis results are presented in Table and discussed below.
Consistent with prior work 4 3. we observe that relative encodings for camera geometry consistently outperform absolute ones. CAPE, GTA, and PRoPE all yield improvements over widely used raymap encodings, with PRoPE (followed closely by GTA) producing the best results.
Relative encodings outperform absolute
Projective positional encoding improves view synthesis quality. PRoPE consistently outperforms other encoding methods across metrics on RealEstate [12], despite the dataset's limited intrinsics variation. This confirms that capturing more complete camera information (both intrinsic and extrinsic) in our relative encoding is beneficial. We also find no loss performance when the train and test images all have constant intrinsics: GTA 3 and PRoPE produce identical metrics for the Objaverse dataset, which verifies that PRoPE reduces to GTA when camera intrinsics are unimportant.
4.3 Attention-Level Intrinsics Conditioning
Real-world data often involves different cameras and focal autonomous cars or zoom lenses on point-and-shoo cameras. To understand how effective PRoPE is at encoding intrinsics information, we evaluate each conditioning method versions the RealEstate 10K [22 and Objaverse [3] datasets. We augment RealEstate 10K by applying zoom factor sampled uniformly in [1, to each image. For Objaverse, we switch from constant field of views to uniformly sampled ones between 35 and 50 degrees. In contrast to Section 4.2] this means that cameras within scenes vary in both extrinsics and intrinsics. We present quantitative results in Table and qualitative results in Figure A.I
multiview rigs on
aatrinsics-augmeent
PRoPE enables intrinsic-aware multiview understanding. We observe that PRoPE outperforms all alternative camera conditioning techniques for both the RealEstate 10K and Objaverse datasets. Existing attention- basod methods, which let networks condition relative pose, perform poorly

---

PROPE
Raymap
Raymap
CamRay
CamRay
Figure 2: Hybrid camera encoding. Left: token-level conditioning using only raymaps, which capture both intrinsies and extrinsics. Right: hybrid encoding where attention-level PRoPE captures relationships between camera frustums, and token-level CamRay encodes local ray directions.
RealEstate 10K
12
Objaverse LPIPS!
13
Method
PSNRT
LPIPS]
SSIM
PSNR
SSIM
GTA
15.77
0.512
0.641
16.98
0.261
0.762
GTA+CamRay 21.41
0.238
0.673
22.69
0.123
0.869
PRoPE
21.42
0.247
0.678
22.82
0.119
0.872
PRoPE+CumRay 21.78
0.211
0.692
22.98
0.114
0.874
Table 3: Novel view synthesis with hybrid camera encodings, with varying intrinsics in each scene. Intrinsics can be conditioned by concatenating local frame camera rays to the network input.
without knowledge of intrinsics. While token-level raymaps carry sufficient camera information, they perform worse overall than PRoPE's relative conditioning formulation.
4.4 Hybrid Encoding Strategies
Token- and attention-level camera encodings require modifications to different parts of the transformer architecture. They are therefore compatible with each other: both conditioning styles can be used simultaneously. To compare PRoPE against stronger baseline while simultaneously exploring these "hybrid" conditioning strategies (Figure[) we train LVSM variations that couple relative encodings with local, camera-frame raymaps:
M i,CaRRay
d. K 1 ERR
(21)
We refer to this raymap CamRay. CamRay shares many properties with existing raymaps (Sec- tion 3.2) encodes intrinsics, is pixel-aligned, and can be concatenated to input images- but not tied to absolute coordinate system. It can therefore be used in conjunction with relative pose and camera encoding techniques without sacrificing global frame invariance. As we observe in Section 4.6 this provides empirical advantages over Plücker raymaps.
CamRay can be interpreted as a token-level encoding of camera intrinsics. We therefore evaluate using the intrinsics- augmented NVS datasets described in Section 4.3 Results are reported in Table and discussed below.
PRoPE effectively encodes camera geometry. On both RealEstate 10K and Objaverse, we observe that PRoPE comparable with or outperforms CTA+CamRay. This true even though PRoPE simpler: it is only applied attention-level, while GTA+CamRay includes both attention-level and teken-level terms.
Token-level and attention-level conditioning techniques are complementary. GTA and PRoPE both benefit from the extra CamRay input. GTA benefits significantly more; this can be explained by the absence of intrinsics information in the standard
GTA formulation.
4.5 Out-of-distribution Robustness
One hypothesis for why relative camera encodings outperform absolute ones is improved generaliza- tion characteristics; this is similar to why RoPE (28 can improve performance for language modeling. To test this, we benchmark conditioning methods using test-time settings that introduce distribution shifts sequence length and intrinsics (Figure
Setting 1: Longer Input Sequences at Text Time. Inspired by test-time context length extrapolation 66 61], our first setting deploys NVS models trained with fixed number of input views (2 our experiments) to significantly more views (up to 16) at test time. This is particularly important
7

---

Longer Sequence Lengths
Longer Sequence Lengths
PRoPE
Unseen Intrinsics
Training Views
Testing Views
Unseen Intrinsics
GTA
Raymap
CAPE
Training FoV
Testing FoV
Figure 3: Out-of-distributioo tasks. Left: We evaluate camera conditioning methods on both longer sequence lengths and unseen camera intrinsics. Right: PRoPE improves results for both unseen sequence lengths and unseen intrinsics.
Ruseor
CAPE
OTA
FRAPE
moot Cartlay
0.5
0
0.70
190C
0.5
16
(a) Testing with (2, 4, 8, 16) input views.
0.66
0.64
(b) Testing with 2, 3]x zooming-in.
Figure 4: Evaluation RealEstate10K [12]. Relative encoding methods demonstrate superior
robustness on handling (a) varying numbers of input views and (b) different focal lengths at test time.
Plucker
CAPE
GTA
PRoPE
PRoPE+CamRae
Testing w/ 2 views
GT
Testing w/ 16 views
Figure 5: Results of Longer Input Sequences at Test Time. PRoPE demonstrates superior robust- ness when tested with varying numbers of input views, despite being trained with only 2 views. Plucker
CAPE
GTA
PRoPE PRoPE+CamRay
GT
Testing w/ zoom
Figure 6: Results of Varying Focal Length at Test Time. PRoPE explicitly models relative intrinsics find this makes it more robust zoomed-in test views.
in real-world scenarios where the number of observations can vary substantially across different applications, and sometimes dynamically increase.
Setting 2: Out-of-distribmtion Intrinsics at Test Time. Our second setting evaluates a model's ability to handle varying focal lengths at test time. This is crucial as focal length can vary significantly across

---

Dataset
Model
Abs Rel
Sq Rel
RMSE
RMSE log
UniMatch
0.123
0.175t
0.678
0.203
RGBD 62
UniMutch PRoPE
0.105
0.2031
0.573
0.181
UniMatch
0.131
0.098
0.397
0.169
SUNSD 33
UniMutch PRoPE
0.117
0.075
0.343
0.152
UniMatch
0.065
0.085
0.575
0.126
Scenesl1
UniMutch PRoPE
0.049
0.063
0.474
0.104
Table 4: Performance Improvement on Stereo Depth Estimation Task with UniMatch [4]. TThe
"Sq Rel" metric is less reliable on the RGBD dataset due the imperfect depth and camera pose [64]
DuiMonch
UniMatch
RGED
Sceardll
SUN3D
Figure 7: Qualitative Results on Stereo Depth Estimation Task. Attention-level camera condition- ing UniMatch 14
leads to significant estimation improvements.
Method
5 views
views
17 views
Plücker
69.1%
76.9%
74.6%
PRoPE-Plücker
81.1%
90.5%
91.8%
PRoPE+CumRay
86.1 %
93.0%
94.3%
Table 5: Spatial cognition results. We report the accuracy of detecting inconsistent image-camera pairs the DL.DD [15) dataset under varying numbers of input views. Both CamRay and PRoPE significantly help with performance, without introducing additional model parameters. An illustration of this task can be found in Figure A.3
different cameras and zoom levels, and impractical to train separate model for every possible focal length. We test our models with focal lengths ranging from 1 x to 5x the training focal length, simulating scenarios where more zoomed-in images are seen during deployment.
Relative encodings improve generalization; PRoPE outperforms alternatives. Evaluated results on the RealEstate 10K [12] dataset summarized in Figure with visuals provided in Figure Sndd We make three main observations. First, while Plücker Raymap encodes more complete camera information than CAPE and GTA, it consistently underperforms intrinsics information is critical. Second, PRoPE improves performance and robustness in both out- of-distribution settings, particularly when handling out-of-distribution focal lengths. This indicates that explicitly modeling the relative projective relationship between cameras is more effective than modeling only the relative SE(3) relationship, done in GTA and CAPE, Finally, found adding CamRay to PRoPE actually hurts performance for intrinsics extrapolation; this suggests that PRoPE uniquely useful for intrinsies generalization.
all settings-eve even when
4.6
Task Generalization
for multiview transformers have focused experiments novel view synthesis. To better understand how these conclusions generalize, we evaluate PRoPE two new tasks: stereo depth estimation using UniMatch [4 and a spatial cognition task designed around DL33D [5).
Like our results so far, prior studies on relative pose encodings S
Stereo Depth Estimation. We set up this task using UniMatch 14
pretrained multiview transformer
that has seen wide adoption in downstream applications 665688 68 UniMatch originally trained on

---

1x Compute SSIM
100x Compute
Method
PSNRI
LPIPS
PSNRT
SSIM1
LPIPS|
Plücker Raymap
20.48
0.622
0.209
25.64
0.809
0.084
PRoPE
22.80
0.725
0.146
26.56
0.832
0.071
Table 6: Scaling LVSM with increased compute. We compare L.VMM 8 models trained with
Plücker raymaps versus PRoPE on RealEstate 12
at two compute scales.
three different tasks; we focus the stereo depth estimation task, which assumes known relative camera poses between input views. We incorporate camera information into UniMatch's 's cross-view attention mechanism using PRoPE, modifying only ~50 lines of the official code. All models follow the exact same training protocols as described in the original paper.
Spatial Cognition. Next, we design spatial cognition task inspired by [69]. In this task, network is given multiple images of the same scene, each paired with camera information. The problem is designed so that it cannot be solved by analyzing the camera information alone, the images alone, without reasoning about the multiview relationships among all inputs. One of the image-camera pairs intentionally corrupted by assigning it an incorrect camera pose sampled from other frames. The network is then required to identify the incorrect image-camers pair based on geometric consistency. See Appendix A.1.3 for implementation details, and Figure A.3 for input-output examples.
PRoPE's benefits generalize across tasks. For depth estimation, we provide quantitative results in Table 4 and qualitative results in Figure 7 For spatial cognition, accuracy metries provided in Table We find that PRoPE significantly improves multiview understanding across both tasks. In our spatial cognition task, we observe that performance with PRoPE continues improve as the number of views increase during testing, whereas the Plücker raymaps do not exhibit the same trend. We also observe improvements when replacing Plücker with CamRay, which indicates that the absolute extrinsics information hinders the model's 's ability to generalize.
4.7
Scaling PRoPE
In our final set of experiments, evaluate how the advantages of relative camera encoding extend to larger models with more computational resources. We conducted two experiments for this, which are discussed below.
PRoPE's benefits persist when scaling LVSM. We scale our LVSM training pipeline with approxi- mately 100x more computational resources (details in Appendix I.I.I.. We train two variants of this larger LVSM model: one that follows the original LVSM paper g and uses Plücker rays and one that incorporates PRoPE. Results are reported in Table [ encoding continues to improve model quality-by smaller but still significant margin- on larger model variants trained with more resources.
where we observe that the relative PRoPE
PRoPE improves CAT3D results. With assistance from the original authors, we add PRoPE to and retrain CAT3D 77) a large multiview diffusion model conditioned on naive raymaps. We report metrics from this model in Table A.2 PRoPE produces consistent improvements across metrics, while introducing zero additional model parameters and negligible computational overhead.
Conclusion and Future Work
In this work, we highlight how representing cameras as relative positional encodings--pariiculary in a way that captures both camera intrinsics and extrinsics-improves multiview transformers across settings and tasks. Possibilities for future work include improving numerical stability when directly multiplying projective matrices with Q/K/V vectors; it may be possible, for example, for ill-conditioned matrices emerge from telephoto focal lengths. It would also interesting to extend PRoPE to distorted camera models, for example by applying PRoPE to per-patch projective approx- imations. Furthermore, incorporating "multi-frequency" [19] I encoding for camera parameters remains nontrivial due to the non-commutativity of projective transforms, challenge shared by
relative SE(3) attention methods 3 4
10

---

References
[1] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu, Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024.
[2] Alcksandr Safin, Daniel Duckworth, and Mchdi SM Sajjadi. Repast: Relative pose attention seene representation transformer. arXiv preprint (.Xxi:23340009/0 2023.
[3] Takeru Miyato, Bernhard Jaeger, Max Welling. and Andreas Geiger. Gta: A geometry aware attention mechanism for multi-view transformers. In International Conference Learning Representations (ICLR), 2024.
[4] Xin Kong, Shikun Liu, Xiaoyang Lyu, Marwan Taher, Xiaojuan Qi. and Andrew Davison. Eschernet: generative model for scalable view synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9503-9513, 2024.
[5] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances neural information processing systems, 30, 2017.
[6] Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representations. arXiv preprint /803:v:800..00 1803,02/55, 2018.
[7] Ruiqi Gao, Aleksander Holyuski, Philipp Henzler, Arthur Brussee, Ricardo Martin-Brualla, Pratul Srini- vasan, Jonathan T Barron, and Ben Poole. Cat3d: Create anything 3d with multi-view diffusion models. arXiv preprint arXiv:2405.10314. 2024.
[8] Haian Jin, Hanwen Jiang, Hao Tan, Kai Zhang, Sai Bi, Tianyuan Zhang, Fujun Luan, Noah Snavely, and Zexiang Xu. Lvsm: A large view synthesis model with minimal 3d inductive bias. arXiv preprint arXiv:2410. 17242, 2024.
[9] Jensen Zhou, Hang Gao, Vikram Voleti, Aaryaman Vasishta, Chun-Han Han Yao, Mark Boss, Philip Torr, Christian Rupprecht, and Varun Jampani. Stable virtual camera: Generative view synthesis with diffusion models. Xrviv preprint urXiv:2503.14483 2025.
[10] Ethan Weber. Norman Müller. Yash Kant, Vasu Agrawal, Michael Zollhöfer, Angjoo Kanazawa, and Christian Richardt. Fillerbuster: Multi-view scene completion for casual captures, 2025. arXiv:2502.05175.
[11] Tri Dao. Flashattention-22 2: Faster attention with better parallelism and work partitioning, arXiv preprint arXIv:2307.08661, 2023.
[12] Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnification: learning view synthesis using multiplane images. ACM Transactions on Graphics (TOG), 37(4):1-12. 2018.
[13] Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3d objects. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13142-13153, 2023.
[14] Haofei Xu, Jing Zhang, Jianfei Cai, Hamid Rezatofighi, Fisher Yu, Dacheng Tao, and Andreas Geiger. Uni- fying flow, stereo and depth estimation. IEFE Transactions on Pattern Analysis and Machine Intelligence, 2023.
[15] Ling, Yichen Sheng, Zhi Tu, Wentian Zhao, Cheng Xin, Kun Wan, Lantao Yu, Qianyu Guo, Zixun Yu, Yawen Lu, et al. D13dv-10k: A large-scale scene datasct for deep leamiig-bassed based 3d vision. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22160-22169, 2024.
[16] Amirhossein Kazemnejad, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Payel Das, and Siva Reddy. The impact of positional encoding on length generalization in transformers. Advances in Neural Information Processing Systems, 36:24892-249775 249928, 2023.
[17] Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yarn: Efficient context window extension of large language models. arXiv preprint arXiv:2309.0007 2023.
[18] Olga Golovneva, Tianlu Wang, Jason Weston, and Sainbayar Sukhbaatar. Contextual position encoding: Learning to count what's important, arXiv preprint 4rXiv:2405.88119 2024.
11

---

[19] Connor Schenck, Isaac Reid, Mithun George Jacob, Alex Bewley. Joshua Ainslie, David Rendleman, Deepali Jain, Mohit Sharma, Avinava Dubey, Ayzaan Wahid, et al. Learning the ropes: Better 2d and 3d position encodings with string, arXiv preprint arXiv:2502.02502 2025.
[20] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018.
[21] Jacob Devlin, Ming- Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidi- rectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume (long and short papers), pages 4171-4186, 2019.
[22] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations. 2021.
[23] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9650-9660) 2021.
[24] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763, PmLR. 2021.
[25] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer 2022.
vision and pattern recognition, pages
[26] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF international conference computer vision. pages 1995-11986, 2023.
[27] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Speneer Whitehead, Alexander C Berg. Wan- Lo, et al. Segment anything. In Proceedings of the JEEE/CVF international conference on computer vision, pages 4015 4026, 2023.
[28] Byeongho Heo, Song Park, Dongyoon Han, and Sangdoo Yun. Rotary position embedding for vision transformer. In European Conference Computer Vision, pages 289-2089-55 305. Springer. 2024.
[29] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of machine learning research, 21(140):1 67, 2020.
[30] Hugo Touvroa, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal. Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprini 23X.v:2302.13971. 2023.
[31] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability llms via reinforcement learning. arXiv preprint arXiv:2501. 12948, 2025.
[32] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892-3491 2023.
[33] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, al. Scaling rectified flow transformen, for high-resolution image synthesis. In Forty-first international conference on machine learning, 2024.
[34] Black Forest Labs. Flux. httpps///gttub..cc//bbbeeeeeeeet
2024.
[35] Genmo Team. Mochi 1. http////ga///ggho//////////////o 2024.
[36] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, et al. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1 66, 2020.
12

---

[37] Ofir Press, Noah A. Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables imput length extrapolation. In ICLR, 2022.
[38] Ofir Press, Noah A Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. arXiv preprint arXiv:2108. /2409, 2021.
[39] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
[40] Mohammed Suhail, Carlos Esteves, Leonid Sigal, and Ameesh Makadia. Light field neural rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8269 8279, 2022.
[41] Mehdi SM Sajjadi, Heaning Meyer, Etienne Pot. Urs Bergmann, Klaus Greff, Noha Radwan, Suhani Vora, Mario Lučić, Daniel Duckworth, Alexey Dosovitskiy, ct al. Scene representation transformer: Geometry- free novel view synthesis through set-latent seene representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6229-6238, 2022.
[42] Rundi Wu, Ruiqi Gao, Ben Poole, Alex Trevithick, Changxi Zheng. Jonathan T. Barron, and Aleksander Holynski. CAT4D: Create Anything in 4D with Multi-Vies Video Diffusion Models. arXiv:2411. 2024.
[43] Hanxue Liang. Jiawei Ren, Ashkan Mirzaei, Antonio Torralba. Ziwei Liu, Igor Gilitschenski, Sanja Fidler. Cengiz Oztireli, Huan Ling, Zan Gojeie, et al. ferwafd bullet-time reconstruction of dynamic scenes from monocular videos. arXiv preprint arXiv:2412.03526, 2024.
[44] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99- 106, 2021.
[45] Jason Y Zhang, Amy Lin, Moneish Kumar, Tzu-Hsuan Yang. Deva Ramanan, and Shubham Tulsiani. Cameras as rays: Pose estimation via ray diffuusion. arXiv preprint arXiv:2402.14817, 2024.
[46] Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10371-103811 2024.
[47] Bingxin Ke, Anton Ohukhov, Shengyu Huang, Nando Metzger, Rodrigo Caye Daudt, and Konrad Schindler. Repurposing diffusion -baed image generators for monocular depth estimation. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 9492-9502, 2024.
[48] Yining Hong, Chunru Lin, Yilun Du, Zhenfang Chen, Joshua Tenenbaum, and Chuang Gan. 3d concept learning and reasoning from multi-view images. Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 9202-9212, 2023.
[49] Chenming Zhu, Tai Wang, Wenwei Zhang, Jiangmiao Pang, and Xihui Liu. Llava-3d: A simple yet effective pathway to empowering Imms with 3d-awareness. arXiv preprint arXiv:2409. 18/25, 2024.
[50] Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Perceiver-actor: A multi-task transformer for robotic manipulation. In Conference on Robot Learning, pages 785 799. PMLR, 2023.
[51] Ryan Po. Yotam Nitzan, Richard Zhang, Berlin Chen. Tri Dao. Eli Shechtman, Gordon Wetzstein, and Xun Huang. Long-context state-space video world models. arXiv preprint 2505:2505..0171, 2025.
[52] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pages
2021.
[53] William Pechles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4195-4205 2023.
[54] Amir Bar, Gaoyue Zhou, Danny Tran, Trevor Darrell, and Yann LeCun. Navigation world models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 15791-15801, 2025.
[55] Julius Plucker. Xvii. on a new geometry of space. Philosophical Transactions of the Royal Society of London, (155):725-791 1865.
13

---

[56] Ben Mildenhall, Pratul P Srinivasan, Rodrigo Ortiz-Cayon, Nima Khademi Kalantari, Ravi Ramamoorthi. Ren Ng. and Abhishek Kar. Local light field fusion: Practical view synthesis with prescriptive sampling guidelines. ACM Transactions on Graphicx (ToG), 38(4):1-14, 2019.
[57] Vitor Guizilini. Muhammad Zubair Irshad, Dian Chen, Greg Shakhnarovich, and Rares Ambrus. Zero-shot novel view and depth synthesis with multi-view geometric diffusion. arXiv preprint arXiv:2501.
2025.
[58] Jonáš Kulhánek, Erik Derner, Torsten Sattler, and Robert Babuška. Viewformer: Nerf-free neural rendering from few images using transformers. In European Conference on Computer Vision, pages 198- 216.. Springer, 2022.
[59] Brent Yi, Vickie Ye, Maya Zheng, Yunqi Li, Lea Müller, Georgios Pavlakos, Yi Ma. Jitendra Malik, and Angjoo Kanazawa. Estimating body and hand motion in an ego-sensed world. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 7072-7084, 2025.
[60] Reiner Pope. Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. Efficiently scaling transformer inference. Proceedings of Machine Learning and Systems, 5:606-624, 2023.
[61] Xindi Wang. Mahsa Salmani, Parsa Omidi, Xiangyu Ren, Mehdi Rezagholizadeh, and Armaghan Eshaghi. Beyond the limits: survey of techniques to extend the context length in large language models. arXiv preprint arXiv:2402.00 0224444 2024.
[62] Jürgen Sturm, Nikolas Engelhard, Felix Endres, Wolfram Burgard, and Daniel Cremers. A benehmark for the evaluation of rgb-d slam systems. In 2012 IEEE/RSJ international conference on intelligent robots and systems, pages 573-580. IEEE, 2012.
[63] Jianxiong Xiao, Andrew Owens, and Antonio Torralba. Sun3d: A database of big spaces reconstructed using sfm and object labels. In Proceedings of the IEEE international conference on computer vision, pages 1625-1632, 2013.
[64] Benjamin Ummenhofer, Huizhong Zhou, Jonas Uhrig, Nikolaus Mayer, Eddy llg, Alexey Dosovitskiy, and Thomas Brox. Demon: Depth and motion network for learning monocular stereo. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5038-5047, 2017.
[65] Yuedong Chen, Haofei Xu, Chuanxia Zheng, Bohan Zhuang, Mare Pollefeys, Andreas Geiger, Tat-Jen Cham, and Jianfei Cai. Mysplat: Efficient 3d gaussian splatting from sparse multi-view images. In European Conference on Computer Vision, pages 370-386. Springer, 2024.
[66] Yucdong Chen, Chuanxia Zheng, Haofei Xu, Bohan Zhuang, Andrea Vedaldi, Tat-Jen Cham, and Jianfei Cai. Mvsplat360: Feed-forward 360 scene synthesis from sparse views. Advances in Neural Information Processing Systems, 37:107664-107086, 2025.
[67] Cameron Smith, David Charatan, Ayush Tewari, and Vincent Sitzmann. Flowmap: High-quality camera poses, intrinsics, and depth via gradient descent. arXiv preprint arXiv:2404. 15259, 2024.
[68] Muyao Niu, Xiaodong Cun, Xintao Wang, Yong Zhang. Ying Shan, and Yinqiang Zheng. Mofa-video: Controllable image animation via generative motion field adaptions in frozen tnage-to-video diffussin model. In European Conference Computer Vision, pages 111-128. Springer, 2024.
[69] Tyler Bonnen, Stephanic Fu, Yutong Bai, Thomas O'Connell, Yoni Friedman, Nancy Kanwisher, Josh Tenenbaum. and Alexei Efros. Evaluating multiview object consistency in humans and image models. Advances in Neural Information Processing Systems, 37:43533-44353 2025.
[70] Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Zip-nerf: Anti- aliased grid-based neural radiance fields. In Proceedings the IEEE/CVF International Conference on Computer Vision, pages 19697-19708 2023.
[71] Kyle Sargent, Zizhang Li, Tanmay Shah, Charles Hermmann Hong-Xing Yu, Yunzhi Zhang, Eric Ryan Chan, Dmitry Lagun, Fei-Fei, Deqing Sun, et al. Zeronvs: Zero-shot 360-degree view synthesis from a single image. arXiv preprint arXiv:2310.1 2023.
[72] Rundi Wu, Ben Mildenhall, Philipp Henzler. Keunhong Park, Ruiqi Gao. Daniel Watson, Pratul Srini- vasan, Dor Verbin, Jonathan T Barron, Ben Poole, et al. Reconfusion: 3d reconstruction with diffussin priors. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2024.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We state our contribution as theoretical grounded analysis and new approach for camera conditioning in transformer that is more robust than previous works. This has been well discussed in the method section and proved in the experiment section.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
In section 5.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model cell-speciiiaatinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
15

---

Answer: [NA]
Justification: Our paper does involve plenty of math in section 3 but is not proving anything.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We outline the method section with detailed math. We will also provide Pseudo code in supplemental material of our algorithm.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
16

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification:
Code will be released soon.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips //ips.cc//pill wttp:ccppppcccceeeeeeeeee..c.ett httpa///guddeee/eeeeabiiissssss /Cc////oddiiisssseeeeeee www.siiiccciiisiccoooocoaiiiiiii for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: Training details will be provided the supplemental material. Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: The experiments expensive to run, and we an academic group can't afford multiple runs for error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
17

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
Will be provided in supplemental material.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics httppppp/eearrri...cpppubb/aaaaa
Answer: [Yes]
Justification:
We did our best to follow the code of ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- cration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our work is foundational research on the camera encoding approaches and not tied to particular applications.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
18

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: Our work is a foundational research the camera encoding approaches so does not poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All assets in the paper created by except for the dataset which is properly cited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
19

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
20

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM for what should or should not be described.
21

---

A.1 Experiment details
A.1.1 LVSM-based Model Details
We adhere to the original LVSM [8] implementation specifications, maintaining consisteney most settings. For complete details regarding these configurations, we direct readers to the original LVSM paper 8 The modifications made include:
We trained exclusively at 256x256 resolution and did not perform the additional fine-tuning at higher resolutions.
Limited by academic-level resources, we use a smaller version of the LVSM model with transformer blocks, and reduce the MIP channel dimension from 3072 1024. Our models are trained on 2x GPUs with a total batch size of 4, as opposed to 512 in the original paper. This applies all experiments expect for the in Section 4.7
For the scaling-up experiments in Section 4.7 we use LVSM model with 12 transformer blocks and keep the all other configurations including the MLP channel dimension (3072). These models are trained on 8x GPUs with a total batch size of 64.
A.1.2 UniMatch Modification Details
UniMatch [1] is model consists of a cross-altention transformer capture multi-view relationship. as well as self-attention transformer that serves as a single-view image encoder/decoder. We inject the camera information into both transformers, on the Q/K/V/O vectors, using our formulation. Note that our formula exactly falls back to RoPE [28] in the single-view scenario. It therefore naturally works with both transformer networks in UniMatch.
A.1.3 Spatial Cognition Model Details
We formulate this task as classification problem, where the number of classes corresponds to the number of input image-camera pairs, and the "ground-truth" class is the ID of the inconsistent image-camera pair. The training objective is to identify the inconsistent pair, which we optimize using the cross- cross-entropy loss. The architecture of the model is largely similar to LVSM (8), where the only change that the last linear layer is modified to output a single scalar for each token. Outputs are then averaged over all tokens for each input pair, resulting in a vector that represents the probability that each input pair the "bad" one (over softmax). We provide more data exemplars we used for training and testing in Figure A.3
A.2
Additional Results
A.2.1
Ablating PRoPE
As our proposed PRoPE includes two terms: the projective relationship between cameras (D, Proj) and the patch coordinate relationship (DR. We here ablate each of them to study their contribution. As shown in Table A. as a crucial component to the system, modeling the projective relationship between cameras alone already yields strong performance. Introducing RoPE (8] further enhances model's 's ability on understanding patch relationship, which is a minimal unit in vision transformer architecture. Notably, in PRoPE, we allocate half of the feature channels to encode each term. Ablating one term therefore means using all feature channels to encode the remaining one.
A.2.2 PRoPE v.s. GTA: More Analysis on the Affect of Intrinsic Modeling
The main difference between our proposed PRoPE and prior work GTA (3] is the introduce of camera intrinsic into the formulation. While GTA itself does not take into account camera intrinsics, as we have pointed out in Section 4.4 it compatible with our proposed CamRay which encodes the intrinsic at token level to form a complete representation for camera conditioning. In this section, we run more thorough experiments focusing comparing PRoPE and GTA in differnet settings (with/without CamRay) the camera conditioning techniques on both UniMatch for stereo depth
22

---

Encoding Method
PSNR
LPIPS
SSIM
w/o D Proj
16.04
0.505
0.509
w/o DR ROPE
21.39
0.238
0.673
PRoPE
21.78
0.211
0.692
Table A.1: Ablation Study on PRoPE. D
Proi is crucial for encoding the relative camera information, and DRPP is also helpful to capture the relative patch coordinate. Experiments are conducted on RealEstate 10K 12
with CamRay as input.
3-vicw SSIM
6-view SSIM
9-view
Method
PSNR 1
LPIPS
PSNR
LPIPS
PSNR
SSIM
LPIPS |
Zip-NeRF 70
12.77
0.271
0.705
13.61
0.284
0.663
14.30
0.312
0.633
ZeroNeRF
14.44
0.316
0.680
15.51
0.337
0.663
15.99
0.350
0.655
ReconFusion 72
15.50
0.358
0.585
16.93
0,401
0.544
18.19
0.432
0.511
CAT3D 7
16.62
0.377
0.515
17.72
0.425
0.482
18.67
0.460
0.460
CAT3D PRoPE
16.93
0.382
0.505
18.01
0.443
0.479
18.98
0.474
0.461
Table A.2: Incorporating PRoPE into CAT3D. Adding PRoPE to the CAT3D 7 multiview
diffusion model yields improvements the original model, with zero additional model parameters and negligible computational overhead.
Method
1
3
5x
7
Plücker
19.89
21.03
20.75
20.32
GTA
15.77
18.14
18.12
18.29
GTA+CamRay
21.41
22.70
22.35
21.77
PRoPE
21.42
22.95
22.75
22.40
PRoPE&CamRay
21.86
23.06
22.83
22.40
PSNR (f) across zoom-in levels.
Method
1x
3x
5x
Method
1x
7x
Plücker
0.608
0.694
0.732
0.755
Plicker
0.327
0.272
0.308
0.345
GTA
0.512
0.626
0.683
0.721
GTA
0.641
0.488
0.439
0.403
GTA+CamRay
0.673
0.741
0.764
0.780
GTA+CamRay
0.238
0.191
0.242
0.288
PRoPE
0.678
0.748
0.775
0.794
PRoPE
0.247
0.193
0.227
0.244
PRoPE+CamRay
0.693
0.751
0.776
0.794
PRoPE+CamRay
0.218
0.183
0.220
0.242
SSIM (1) across zoom-in levels.
LPIPS (4) across zoom-in levels.
Table A.3: Additional Novel View Synthesis Results on nut-of-distribution Intrinsics at Test
Time. Experiments are conducted with LVSM [8 on RealEstate [12 dataset with augmented
intrinsics (1-3x zoom-in) as described in Section 4.3
estimation in Table A.4 and LVSM for novel view synthesis in Table A.3 both with augmented intrinsics in training highlight the benefits of PRoPE intrinsic modeling.
The new results corroborate the benefits of PRoPE that were originally shown in the main paper. We find that PRoPE consistently outperforms GTA. Importantly, both PRoPE and PRoPE+CamRay outperform GTA+CamRay, despite the fact that these methods include the same information and are all invariant to the choice of global frame. This supports our finding that attention-level intrinsic conditioning (as PRoPE) important.
A.2.3 Qualitative Results for Stereo Depth Estimation
In Figure A.2 we show more qualitative results on the task of stereo depth estimation with Uni-
Match [14
23

---

Method
1.0x
1.5x
2.0x
3.0x
Method
1.0x
1.5x
2.0x
3.0x
UniMatch
0.234
0.228
0.238
0.299
UniMatch
0.357
0.376
0.408
0.613
+Plocker
-2.47%
-9.35%
-7.74%
-4.67%
+Plucker
-1.08%
-9.45%
-2.76%
-3.37%
+GIA
-28.83%
-33.48%
-32.13%
-12.55%
+GTA
-31.77%
-41.52%
41.16%
-11.88%
+GIIA+CaRRay
-29.90%
-3..25%
-32.42%
-13.6%%
+GIA-CamRay
-33.54%
-44.06%
-42.44%
-13.09%
+PRoPE
-33.75%
-3.66%
-33.8%%
26.40%%
+PRoPE
-5.13%
46..99%
-44.51%
-38.56%
PRoPPeCamRRy
-31.99%
-35.82%
-33.24%
-28.21%
PRooPe-CamRay
-41.79%
-46.75%
44.08%
-39888%
abs_rel rel (1) across zoom-in levels.
sq_rel rel (4) across zoom-in levels.
Method
1.xx
2.0x
3.0x
Method
1.0x
1.5x
2.0x
UniMatch
1.059
1.031
1.035
1.188
UniMatch
0.322
0.322
0.336
0.421
+Plcckrr
+1.03%
-3.95%
-0.53%
+1.64%
+Plucker
+0.49%
-5.46%
-2.33%
+2.00%
+GTA
-19.55%
-26.12%
-25.68%
-4.50%
+GTA
-23.29%
-30.33%
-30.03%
-3.24%
-GIA+CamRay
-20.41%
-27.44%
-25.72%
-6. -6.12%
+GTA-CamRay
-24.08%
-31.64%%
-30.23%
-4.34%
-28.64%
-28.25%
22.61%
-33566
-3.67%
26.78%
PRoPE+ComRay
-26.86%
-29.29%
-28.09%
-23.7%%
PRooPeCamRRa
-29.98%
-32.68%
-32.31%
-27.71%
imsc (L) across zoom-in levels.
imse_ rmse_log (L) across zoom-in levels.
Table A.4: Additional Stereo Depth Estimation Results on Out-of-distribution Intrinsics at Test Time. Experiments are conducted with UniMatch [14]'s official code as described in Section 4.6 but with 1/8 of the training resources (2 GPUs 50k steps).
Objaverse
RealEstatelOK
Plucker Raymap
Naive Raymap
GTA
PRoPE
PRoPE-CamRay
Figure A.I: More Qualitative Results of Novel View Synthesis on RealEstate10K [12 and Objaverse [3].
24

---

UU Mo--oMooo
UnihMarch 200P
DI Thrger
GT Rele
Figure A.2: More Qualitative Results of Stereo Depth Estimation on RGBD 622, SUN3D 63 and Scenesll 664).
25

---

Figure A.3: The Spatial Cognition Task. The model takes multi view images and corresponding camera information as input and aims to identify inconsistent image-camers pairs (yellow here). Understanding the cross-view relationships between images and cameras is crucial for this task.
26


Reviews:
[{"review_id": "lRcMORj8nj", "reviewer": "Reviewer_X9c5", "summary": "## Context\nRecently, multi-view transformer architectures became increasingly popular for 3D tasks. For each input view, the viewpoint, in terms of camera extrinsics and intrinsics, needs to be provided to the model. Previous works use different ways to provide those camera parameters to the model, summarized by the paper under review as follows:\n- token-level conditioning: \n\t- concatenate camera conditioning to the respective image tokens\n\t- methods: raymaps, Plücker raymaps\n\t- can encode extrinsics and intrinsics\n\t- encoding depends on the choice of the reference frame\n\t- sensitive to global SE(3) transformations\n- attention-level conditioning:\n\t- inject pairwise camera conditioning into the attention mechanism\n\t- methods: CAPE (from EscherNet), GTA\n\t- existing methods only excode extrinsics, but not intrinsics\n\t- encode relative poses between pairs of views and are therefore invariant to the reference frame\n\t- invariant to global SE(3) transformations\n\n## Method\nTo resolve the shortcomings of previous methods, the paper under review proposes a new encoding method called PRoPE+CamRay. PRoPE essentially extends GTA to also incorporate the intrinsics into the relative positional encoding between pairwise views. CamRay additionally provides the intrinsics to the model via token-level conditioning in form of an intrinsics-only-raymap.\n\n## Experiments\nThe most relevant experiments are conducted for the task of novel view synthesis on RealEstate10k, using LVSM as base model. The results (Fig. 3, 4, 5) show that PRoPE very slightly outperforms the best previous method GTA regarding a) performance with similar settings at test time as in training, and b) more input views at test time. In case of c) using different focal lengths at test time, it clearly outperforms GTA. The PRoPE+CamRay variant always outperforms PRoPE and has a larger margin to the best previous method GTA. \n\nFurther experiments are conducted with CAT3D, with LVSM on Objaverse, for stereo depth estimation with UniMatch on 3 datasets, and for a spatial cognition-task on DL3DV. The comparisons in those experiments are less conclusive (not all conditioning methods are evaluated). However, in all results, using PRoPE outperforms the performance of the respective baseline model.", "questions": "My key suggestions are:\n- Provide an intuitive explanation why CamRay is necessary. \n- Evaluate CamRay also in the UniMatch and CAT3D experiments (Tab. 1 and 2).\n- Evaluate previous methods (Plücker, GTA, CAPE) also in the other tasks to make the \"systematic analysis\" more extensive.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\n### Elegant method\nThe method is elegant, especially regarding how it comprises previous methods: in case of identity intrinsic matrices, PRoPE falls back to GTA/CAPE and in case of single-view inputs, PRoPE falls back to RoPE. \n\n### Writing\nThe paper is very clearly written and good to follow.\n\n### Consistent improvements\nThe paper under review presents results for a variety of tasks. In all cases, PRoPE improves the results. The more detailed experiments for NVS on Re10k (Fig. 3) show that PRoPE consistently outperforms the previous best GTA. Even though the improvement is small in some cases, it is consistent, and for the case of generalization to different intrinsics at test time, the improvement is significant. The additionally proposed CamRay method further boosts results. \n\n## Weaknesses\n### Unclear intuition for CamRay\nThe main difference of PRoPE over GTA is to incorporate the intrinsics into the relative positional encoding. However, CamRay additionally provides the intrinsics to the model via token-level conditioning. It is shown that this clearly improves the results. However, it remains unclear why this additional usage of token-level conditioning is required. Additional explanations would be helpful.\n\n### Limited evaluation of CamRay\nIt seems like CamRay was only evaluated in the experiments for NVS on Re10k (Fig. 3) and the spatial cognition experiments (Tab. 3). The experiments with CAT3D (Tab. 1) and UniMatch (Tab. 2) seem to be without CamRay. It would be helpful to also evaluate CamRay in these experimental setups, to provide further evidence of its effectiveness.\n\n### Limited systematic analysis\nA major claim of the paper is a \"systematic analysis\" of different camera conditionings (line 4-5). However, such a systematic analysis is actually only conducted in one experimental setup, namely NVS on Re10k with LVSM (Fig. 3). The \"systematic analysis\" would be stronger, if it would be conducted also on the other tasks. \n\n## Comments\nSometimes the authors write Plücker and sometimes Plucker coordinates.\n\nIn the experiments for NVS with varying focal lengths (line 214): does this consider context views with varying focal length, or rendering target views at different target focal lengths, or both?\n\nFigures are sometimes not referenced from the text, for example Fig. 1 or Fig. 2. \n\nFig. 1b): why should PRoPE generalize better to unseen extrinsics? I understand the point that invariance to the reference frame helps generalization in that PRoPE should be able to generalize to unseen absolute poses, as long as the relative poses have been seen during training. However, I do not understand why it should generalize better to unseen relative pose configurations. To illustrate what I mean, here is an LLM example:\n- The training dataset only contains sentences where the words \"the\" and \"likes\" appear at absolute positions 0 and 2.\n- During inference, when using APE, a model would struggle if those words appear at positions 10 and 12. When using RPE, this should not be a problem, as the positional encoding is invariant to the absolute position. \n- However, when those words appear at positions 0 and 12, I would expect that both APE and RPE have the same difficulties, as also this relative position is unseen. \n- Actually the text describes this better than Fig. 1b) as it says that some methods are \"sensitive to SE(3) transformations\" (e.g. line 90), which seems to be a better description than the \"Robust to unseen extrinsics\" in Fig. 1.", "was_revised": true, "final_justification": "The rebuttal adresses my questions well and I keep my \"Accept\" rating.\n\nIn particular, I do not share the criticism of reviewer 19jr. I agree that the proposed method builds on the existing method GTA. However, it extends it to also handle camera intrinsics, which is important in practice and is shown to improve results in the evaluations.\n\nThe evaluations that were provided in the rebuttal show that GTA+CamRay also performs well. However:\n1. As I understand it, original GTA does not use CamRay and I would see this combination rather as a contribution of the paper under review, and not as a reason to reject it. \n2. PRoPe and PRoPe+CamRay consistently and sometimes clearly outperform GTA+CamRay.", "rebuttal": "We thank the reviewer for recognizing the strengths of our work in terms of method, writing, and the consistency of performance improvements. We also appreciate the constructive feedback, which we respond to below.\n\n**Q: \"Unclear intuition for CamRay. [...] Additional explanations would be helpful.\"**\n\nWe will update our paper to include more discussion on CamRay.\n\nThe main takeaway of CamRay is that token-level and attention-level camera encodings can be complementary. While PRoPE incorporates camera intrinsics through relative positional encoding at the attention-level, it treats each patch (token) as a unit and does not explicitly distinguish between individual pixels within a token. CamRay, on the other hand, enriches input patches with pixel-wise ray information. This adds another layer of positional detail on top of what PRoPE offers, which we empirically find improve results. \n\nWe will also include more analysis, which we discuss next.\n\n**Q: \"Limited evaluation of CamRay\", \"Limited systematic analysis\"**\n\nWe agree that a more exhaustive set of experiments would strengthen our paper. We have run more experiments, which include: more combinations of camera conditioning techniques, for both in-distribution and out-of-distribution data, on both the NVS and depth estimation tasks. Unfortunately, time constraints prevent us from sharing results for the CAT3D or spatial cognition tasks in time for the rebuttal deadline.\n\nResult tables can be found in our response to Reviewer BUBt. We will update our paper with these new results, which corroborate the key takeaways from our initial submission: relative encoding is critical for performance, and including intrinsics information via PRoPE provides further gains. We also find that CamRay is useful as a tool for analysis. GTA+CamRay and PRoPE+CamRay both contain the same information and are both invariant to the world coordinate system. However, PRoPE+CamRay consistently results in better model performance. This can be interpreted as an ablation that validates the usefulness of PRoPE's intrinsics-aware relative encoding.\n\n**Q: Do varying focal length experiments \"consider context views with varying focal length, or rendering target views at different target focal lengths, or both?\"**\n\nBoth context and target views have augmented focal lengths. We will clarify this in the writing.\n\n**Q: \"Why [relative encodings] should generalize better to unseen relative pose configurations\"?**\n\nThank you for pointing this out. We agree with your statement: \"PRoPE should be able to generalize to unseen absolute poses, as long as the relative poses have been seen during training\". The experiments we conducted would be better described as testing \"generalization to unseen number of extrinsics/cameras\" (the analogy to RoPE generalizes better to longer sequence in LLMs). This affects the batch-level statistics* of the extrinsics but not the extrinsics themselves. We will revisit Figure 1b and our writing to make sure it is more precise.\n\n*Consider: for each camera, the distance to the closest other camera. With the same scene but more cameras, this distance will usually decrease."}, {"review_id": "t4NLa2fD8z", "reviewer": "Reviewer_GRaq", "summary": "The paper proposes a new positional encoding for multi-view transformers applied to 3D perception tasks. In this context, encoding and injecting 3D information inside the transformer architecture grounds the model in 3D and is critical for the quality of the resulting predictions. Existing approaches inject these priors either by concatenating dense raymaps to input images before the tokenization or by directly injecting relative pose information in the attention layers. Raymaps are not invariant with regard to transformations between global coordinate systems while existing methods encoding relative pose do not take into account intrinsic information which may hinder performances on testing views with different intrinsics. The authors therefore introduce PRoPE which encodes pairwise projective transformations and injects it in self attention layers. In addition, they introduce CamRays which express dense camera rays in the camera coordinate space instead of the world coordinate space.          \nApplied to models trained for novel view synthesis and depth prediction tasks, PRoPE and CamRays increase the performances of such 3D multi-view transformers and their ability to generalize to different testing intrinsics.", "questions": "Limited contribution is not necessarily a restricting factor, however I feel like there should be a more comprehensive set of experiments to clearly demonstrate the benefits of your method, pinpoint the limitations and highlight the scenarios in which your encodings are the most beneficial. For example I find the Intrinsic-aware Benchmarking experiment from the supplementary material to be more relevant than Tab.1 and Tab.2. \n\nIn the spatial cognition task and setting 1 of RealEstate10k experiments , what is the range of viewpoint change between cameras? Can you quantify how CamRay performs vs Plucker embeddings on different magnitude of viewpoint changes? \n\nAs an alternative, have you considered encoding epipolar information between pairs of images? Intrinsics information could also be included in the fundamental matrix. \n\nCould your model be easily extended to account for distortion?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 1, "strengths_and_weaknesses": "### Strengths \n1) The problems associated with existing encoding approaches are well identified. The proposed solutions are sound and tackle the identified shortcomings in an effective way as demonstrated in the experimental section. Injecting intrinsic information explicitly provides camera prior which is more effective than letting the network implicitly capture intrinsic information.    \n\n2) Similarly, CamRay consistently outperforms Plucker rays embedding by preserving invariance wrt. the global frame which allows for better generalization to a larger number of viewpoints demonstrated in the experimental section. \n\n3) The spatial cognition task is an interesting additional experiment that goes beyond existing evaluation metrics and seems to be a good proxy for evaluating the geometric scene understanding of the model.\n\n### Weaknesses \n1) ProPE is very largely inspired by GTA which limits the contributions of the paper. As RoPE is equivalent to SO(2) elements, the only notable difference with GTA is the introduction of intrinsic matrices when computing the relative transformation between pairs of cameras. \nIt has to be noted that the authors clearly stated that ProPE is an extension of existing work and formulated the contributions in that direction. \n\n2) This leads to my second concern. Tab. 1 and Tab. 2 show that applying PRoPE to multi-view diffusion models or stereo depth estimation models improve their performances. However as PRoPE is an extension of GTA, if the intrinsic variations within the dataset are limited/non-existent PRoPE is not expected to perform better than GTA ( see Fig. 3 a, GTA and PRoPE seem to behave similarly). For the same reason GTA is also expected to improve the models performance on these tasks.\nI think it would be less disingenuous to also report CAT3D+GTA and UniMatch+GTA. And if PRoPE improves upon GTA in these experiments, use it to highlight the benefit of your method.\n\n3) As stated by the authors, the main motivation for encoding intrinsic information is application to real world scenarios. However real world cameras have camera models with distortion, as such using this method would require an additional undistortion step in most cases. It would be an interesting addition to also encode and inject distortion parameters in the models.  \n\n4) I find the Figures related to the Spatial Cognition task to be more confusing than informative, it is not obvious to identify the mismatched pair at a glance. \n\n5) Fig 3 b), the x labels {1,3,5} zoom factors on the graphs do not match with the {1,2,3} zoom factors in the legend.", "was_revised": true, "final_justification": "The additional experiments show that ProPE consistently outperforms GTA across tasks and better highlights the benefits of directly integrating intrinsic information in the attention layers. The paper heavily relies on GTA and provide only intuitive incremental contributions. However, supported by a comprehensive experimental section, these incremental contributions hold value and do not justify a rejection.\nFollowing the rebutall and discussions I have increased my score accordingly.", "rebuttal": "We thank Reviewer GRaq for the thoroughness of their review, including: recognizing the soundness of proposed solutions, ability to retain invariance with CamRay, and value of the spatial cognition experiment.\n\n**Q: \"Also report CAT3D+GTA and UniMatch+GTA\", \"more comprehensive set of experiments to clearly demonstrate the benefits of [PRoPE]\".**\n\nWe thank the reviewer for these suggestions: we agree that more comprehensive experiments would strengthen our conclusions. We have therefore run more experiments using a more thorough combination of camera conditioning techniques on both UniMatch for stereo depth estimation and LVSM for novel view synthesis, both with augmented intrinsics in training to highlight the difference benefits of PRoPE. To minimize redundancy, we include these results in our response to Reviewer BUBt.\n\nThe new results corroborate the benefits of PRoPE that were originally shown in Figure 3b and Supplemental Table 1 in our submission. We find that PRoPE consistently outperforms GTA. Importantly, both PRoPE and PRoPE+CamRay outperform GTA+CamRay, despite the fact that these methods include the same information and are all invariant to the choice of global frame. This supports our finding that attention-level intrinsic conditioning (as in PRoPE) is important.\n\nWhile there is not enough time to re-run the CAT3D experiments within the rebuttal period (as we don't have access to its code and need assistants from its authors), we hope that these additional results help alleviate your valid concerns. We will include both the results and additional analysis in the updated version of our manuscript.\n\n**Q: \"Could your model be easily extended to account for distortion?\"**\n\nThis is an excellent point; we will update our paper with discussion about this limitation and possible future directions. We believe that our high-level conclusion that relative information is important still applies. As the reviewer notes, the most direct route for incorporating distortion into our method is to undistort images before passing them into the model. Other options include injecting projective approximations (e.g., via linearization) into the attention-level encoding (e.g., each token/patch have a different intrinsic K), while relying on CamRay for more fine-grained distortion handling. It would be interesting to explore these methods for datasets with known distortion parameters. We consider this out of scope for our work, but agree that it is an exciting area for future research. \n\n**Q: \"Can you quantify [...] on different magnitude of viewpoint changes?\"**\n\nWe use the standard test sets in each task which contains both small or large viewpoint changes. However it's challenging to quantify based on viewpoint magnitude although that is a good suggestion because both rotation, translation, and perspective changes need to be considered. And to our best knowledge there is no standard way of doing so.\n\n**Q: \"Have you considered encoding epipolar information between pairs of images?\"**\n\nGood question. In our initial experiments, we explored encoding the relative geometric distance between ray pairs (from corresponding pixels) in 3D space using the fundamental matrix. Under this approach, rays lying on the same epipolar plane would have zero relative distance. However, we found this encoding to be fundamentally ambiguous -- multiple distinct ray configurations can yield identical relative distance values. This limitation means that epipolar constraints alone cannot capture the complete relative geometric information needed for effective intrinsic encoding, resulting in undesired performance."}, {"review_id": "Sna86c7YEw", "reviewer": "Reviewer_19jr", "summary": "This paper introduces PRoPE, a projective positional encoding for multi-view transformers that unifies intrinsic and extrinsic camera parameters into a global-frame-invariant, attention-level representation. The method extends prior SE(3)-based encodings by modeling full projective transformations between views. A complementary token-level encoding, CamRay, is used to provide dense per-pixel intrinsics in the local frame. Experiments on novel view synthesis, stereo depth estimation, and spatial consistency show that PRoPE improves generalization, particularly when test-time focal lengths or number of input views differ from training.", "questions": "- Would GTA+CamRay or CAPE+CamRay close the gap to PRoPE? This comparison is critical for fair evaluation, as it ensures that intrinsic information is made available to the baseline models. Without this, it is difficult to attribute performance gains to PRoPE’s projective formulation rather than the presence of intrinsics. This is especially important in scenarios with varying focal lengths, where intrinsic-aware baselines may perform comparably if given the same input.\n\n- Can you ablate Q/K vs. Q/K/V/O injection? GTA uses all Q/K/V/O vectors, while methods like RoPE and CAPE apply transformations only to Q/K. It would be valuable to understand whether PRoPE requires the full injection or if a simpler variant would suffice.\n\n- Have you tested PRoPE under more challenging camera models, such as fisheye lenses or other camera model? These cases introduce nonlinear or non-projective distortions, and it is unclear whether PRoPE generalizes to them.\n\n- How numerically stable is PRoPE when projection matrices are ill-conditioned or when intrinsic parameters vary significantly? For example, extremely small focal lengths or near-degenerate calibration settings may cause instability—how does PRoPE behave in such regimes?", "limitations": "The paper mentions numerical instability and lack of multi-frequency modeling as future directions.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n\n- PRoPE cleanly generalizes attention-level geometric encodings by incorporating intrinsic parameters in a projectively correct and invariant manner.\n\n- The encoding is compatible with FlashAttention and standard transformer layers, and requires minimal architectural changes.\n\n- The model performs well under distribution shifts (e.g., varying focal length or number of views), which are realistic but often overlooked evaluation settings.\n\nWeakness\n\n\n- While mathematically sound, PRoPE is fundamentally an extension of existing methods like CAPE and GTA, augmented with intrinsics. Since intrinsics are low-dimensional and already used in token-level encodings, the conceptual contribution is relatively modest.\n\n- Crucially, the paper does not evaluate GTA+CamRay or CAPE+CamRay, which are the most relevant and fair baselines for comparison. These combinations would test whether the gains from PRoPE stem from its projective formulation or simply from the addition of CamRay-like intrinsic information. Their absence prevents a meaningful comparison and undermines the empirical evidence for PRoPE’s advantage.\n\n- The choice to inject projective encoding into all Q/K/V/O vectors (following GTA) is not ablated or justified against simpler Q/K-only alternatives in RoPE.\n\n- The method is named as PRoPE, and frequency modeling is an important desgin in RoPE. A more interesting and fundamental direction is to consider the frequency modeling in SE3 and 3D geometric space. \n\n- The key contribution of this paper is extending intrinsic encoding to prior SE3 encodings, but how generalise the intrinsic encoding is for different camera models and extreme focus length is not fully analyzed.", "was_revised": true, "final_justification": "After the rebuttal and the discussion with the authors, I believe the additional results and clarification resolved most of my concerns and greatly strengthened the paper. The improvement over baseline is still marginal, but I would like to upgrade my score accordingly.", "rebuttal": "We thank Reviewer 19jr for their comprehensive and constructive feedback. We include responses to key points below, which we will use to strengthen our paper.\n\n**\"Would GTA+CamRay or CAPE+CamRay close the gap to PRoPE?\"**\n\nThis is an excellent question. We have run a more exhaustive set of experiments to answer it; to reduce redundancy, tables can be found in our response Reviewer BUBt.\n\nImportantly, we observe that PRoPE and PRoPE+CamRay outperform GTA+CamRay for both novel view synthesis and for stereo depth estimation, despite the fact that these methods include the same information and invariance properties. This corroborates our finding that PRoPE's attention-level intrinsic conditioning is important. We will include these results in our updated paper.\n\n**\"Can you ablate Q/K vs. Q/K/V/O injection?\"**\n\nWe have run additional experiments ablating this decision on LVSM with the Re10k dataset (with intrinsics augmentation during training). Overall, they are consistent with the results in the original GTA paper: Q/K/V/O injection produces small but consistent improvement in results.. \n\n**Metric: PSNR** (higher is better)\n\n| Transform | GTA    | GTA+CamRay | PRoPE  | PRoPE+CamRay |\n| --------- | ------ | ---------- | ------ | ------------ |\n| QK        | 15.958 | 21.332     | 21.350 | 21.688       |\n| QKVO      | 15.772 | 21.414     | 21.419 | 21.856       |\n\n**Metric: SSIM** (higher is better)\n\n| Transform | GTA    | GTA+CamRay | PRoPE  | PRoPE+CamRay |\n| --------- | ------ | ---------- | ------ | ------------ |\n| QK        | 0.513  | 0.670      | 0.672  | 0.685        |\n| QKVO      | 0.512  | 0.673      | 0.678  | 0.693        |\n\n**Metric: LPIPS** (lower is better)\n\n| Transform | GTA    | GTA+CamRay | PRoPE  | PRoPE+CamRay |\n| --------- | ------ | ---------- | ------ | ------------ |\n| QK        |  0.639 | 0.250      | 0.269  | 0.231        |\n| QKVO      |  0.641 | 0.238      | 0.247  | 0.218        |\n\n**Q: \"Have you tested PRoPE under more challenging camera models, such as fisheye lenses or other camera model?\", \"How numerically stable is PRoPE when projection matrices are ill-conditioned or when intrinsic parameters vary significantly?\"**\n\nWe have not yet evaluated PRoPE for heavily distorted images, but we believe the key conclusions in our paper still apply to these cases. We discuss potential approaches for handling distorted cameras in our response to Reviewer GRaq. This would be an exciting direction for followup work. Degenerate cases: we have not observed any instability associated with ill-conditioned projection matrices, but we agree that this is a possible in adversarial data. We will emphasize it more heavily in our future work section. We have not observed any instability associated with significantly varying intrinsics, which we do evaluate it via OOD intrinsics augmentation (up to 7x zoom-in).\n\n**\"Conceptual contribution is relatively modest\"**\n\nWe agree with the reviewer that PRoPE builds heavily on existing ideas in relative positional encoding. We also believe, however, that the practical significance is still large. Plücker raymaps remain the dominant approach for conditioning multiview transformers, primarily because they unify representation for both camera intrinsics and extrinsics. Our work aims to show that there are better ways to implement these models, which is increasingly important as they are scaled. We also believe that the technical survey and comparative analysis in the paper will be a valuable resource for the 3D computer vision community."}, {"review_id": "dUnDNFSDgZ", "reviewer": "Reviewer_BUBt", "summary": "This paper introduces a new relative positional encoding scheme for 3D-conditioned tasks which includes the full projective relationship (intrinsics added to prior work including relative 3D encoding just with extrinsics). They demonstrate strong results on improving 3D-conditioned tasks, especially those with out-of-distribution intrinsics.", "questions": "- How much does including intrinsics help over just extrinsics (GTA)? Figure 3 provides good illustration but more robust and extensive comparison (including the GTA baseline in all experimental results ideally) would be better\n- Is there any comparison you can add with different camera models? E.g. with fish-eye or lens-distorted models or is your method limited to just the pinhole camera?", "limitations": "- Unclear how to extend encoding multi-frequency embedding for higher-frequency prediction, but prior work also had this limitation and not the focus of this work\n- Method requires known intrinsics which can be difficult to obtain (but still highly useful when one has them)", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n- Paper is clear and easy to follow with sound experiments and results\n- Well-defined scope and contribution (adding intrinsics to the relative 3D encoding instead of just extrinsics)\n- Simple implementation with few lines of code modification for strong results\n\nWeaknesses:\n\n- Contribution over prior work is necessary (adding intrinsics to the relative transformations) but not a very strong/novel change\n- Should use additional baseline comparisons, particularly GTA which is the most similar baseline to demonstrate how important the intrinsics encoding are, e.g. in table 1 and figure 6 for the CAT3D and UniMatch comparisons", "was_revised": true, "final_justification": "My opinion has not changed after the author responses: this paper presents a small and elegant change for per-pixel relative-SE3 conditioning that now encodes intrinsics with an intuitive formulation. I believe it warrants acceptance even without the non-pinhole camera distortion modeling because non-pinhole cameras are more rare and can be handled with pre-processing or likely a simple extension to the formulation to include the distortion parameters.", "rebuttal": "We thank Reviewer BUBt02 for their thorough review, and recognizing strengths of the work: clarity, soundness of experiments, scope and contribution, and simple implementation.\n\n**Q: \"How much does including intrinsics help over just extrinsics (GTA)?\"**\n\nWe ran additional experiments to quantify this, in particular by evaluating GTA+CamRay to measure the improvement from including intrinsics information. We observe that intrinsics are generally critical: GTA+CamRay outperforms GTA significantly. Importantly, we also see that PRoPE and PRoPE+CamRay *both* outperform GTA+CamRay. This shows that the method used to condition on intrinsics matters; the PRoPE-based approach for this conditioning is empirically advantageous than raymap based such as Plucker. Tables can be found below.\n\n**Q: \"Is your method limited to just the pinhole camera?\"**\n\nWe have not yet evaluated PRoPE on distorted images; but we believe our main conclusions will still be highly relevant hold. Potential approaches for handling distorted cameras are discussed in our response to Reviewer GRaq; we view these as promising directions for follow-up work. We will update our paper with discussion.\n\n\n---\n## More Analysis: LVSM for Novel-View-Synthesis\n\nHere we report the complete analysis of different conditioning approaches with LVSM on Re10K dataset. The setting is same as the the experiment reported in supplemental document Table 1 (Intrinsic-aware experiment with random 1-3x zoom-in in during training) . Here we additionally report results with GTA+CamRay, as well as the evaluation on OOD intrinsics with 1x -7x zoom-in in at test time.\n\n**Analysis** \n- GTA v.s. GTA+CamRay: GTA itself does not encode camera intrinsics. While the network can implicit learn the intrinsic from pixels, it is much less effective than explicitly providing the intrinsic to the model.\n- Plücker v.s. GTA+CamRay: Both of them provide {extrinsic, intrinsic} to the model. GTA+CamRay consistently outperform Plücker by ~2db on PSNR indicates attention-level conditioning is superior than pixel-level conditioning.\n- GTA v.s. PRoPE: A simple change of introducing intrinsic into the formula (one line change) boost performance by 4db-6db on PSNR.\n- GTA+CamRay v.s. PRoPE. As camera intrinsics gradually becomes OOD, the margin between attention level conditioning (PRoPE)  and pixel-aligned conditioning (GTA+CamRay) increases for 0db to 0.7db. This indicates attention level conditioning is more robust to OOD data.\n- PRoPE v.s. PRoPE+CamRay. On top of attention-level conditioning, which is applied per-token, CamRay additionally provides pixel-level conditioning thus could further help with the performance. And the fact that it only affect performance marginally indicates the effectiveness of PRoPE on encoding intrinsics.\n\n**Metric: PSNR** (higher is better)\n\n| ZoomIn | Plücker | GTA    | GTA+CamRay | PRoPE  | PRoPE+CamRay |\n| ------ | ------- | ------ | ---------- | ------ | ------------ |\n| 1.0x   | 19.889  | 15.772 | 21.414     | 21.419 | 21.856       |\n| 3.0x   | 21.031  | 18.142 | 22.703     | 22.951 | 23.061       |\n| 5.0x   | 20.749  | 18.123 | 22.345     | 22.748 | 22.833       |\n| 7.0x   | 20.322  | 18.286 | 21.772     | 22.400 | 22.401       |\n\n\n**Metric: SSIM** (higher is better)\n\n| ZoomIn | Plücker | GTA   | GTA+CamRay | PRoPE | PRoPE+CamRay |\n| ------ | ------- | ----- | ---------- | ----- | ------------ |\n| 1.0x   | 0.608   | 0.512 | 0.673      | 0.678 | 0.693        |\n| 3.0x   | 0.694   | 0.626 | 0.741      | 0.748 | 0.751        |\n| 5.0x   | 0.732   | 0.683 | 0.764      | 0.775 | 0.776        |\n| 7.0x   | 0.755   | 0.721 | 0.780      | 0.794 | 0.794        |\n\n\n**Metric: LPIPS** (lower is better)\n\n| ZoomIn | Plücker | GTA   | GTA+CamRay | PRoPE | PRoPE+CamRay |\n| ------ | ------- | ----- | ---------- | ----- | ------------ |\n| 1.0x   | 0.327   | 0.641 | 0.238      | 0.247 | 0.218        |\n| 3.0x   | 0.272   | 0.488 | 0.191      | 0.193 | 0.183        |\n| 5.0x   | 0.308   | 0.439 | 0.242      | 0.227 | 0.220        |\n| 7.0x   | 0.345   | 0.403 | 0.288      | 0.244 | 0.242        |\n\n---\n## More Analysis: UniMatch for Stereo Depth Estimation.\n\nHere we report the complete analysis of different conditioning approaches with UniMatch for stereo depth estimation. The setting is same as the the experiment reported in main paper Table 2, but due to resource constrain, these models are trained with 1/8 of the resources (2 GPUs x 50k steps) comparing to what we reported in the paper (8 GPUs x 100k steps). Here we report the **average relative error reduction (more negative is better)** across three test sets—RGBD, SUN3D, and Scenes11—under test-time zoom-in of {1.0x, 1.5x, 2.0x, 3.0x} to simulate cameras with OOD intrinsics. \n\n**Analysis**. Across all metrics and zoom-in levels, PRoPE consistently outperforms GTA and GTA+CamRay with noticeable margins, highlighting the effectiveness of encoding camera intrinsics directly at the attention level. Notably, under the most extreme OOD condition (3.0x zoom), PRoPE continues to yield substantial gains (22–38%), whereas GTA+CamRay’s improvements drop to just 4–13%. This demonstrates the superior generalization and robustness of PRoPE's design in challenging OOD scenarios. When using CamRay with GTA, it consistently improves the performance by 1%-4%. We note that CamRay does not always bring benefits to PRoPE: this indicates that PRoPE alone is already conditioning effectively on intrinsics.\n\n\n\n**Metric Improvement: abs\\_rel** (more negative is better)\n\n| ZoomIn | UniMatch | +Plucker | +GTA    | +GTA+CamRay | +PRoPE  | +PRoPE+CamRay |\n| ------ | -------- | -------- | ------- | ----------- | ------- | ------------- |\n| 1.0x   | 0.234133 | -2.47%   | -28.83% | -29.90%     | -33.75% | -31.99%       |\n| 1.5x   | 0.228133 | -9.35%   | -33.48% | -35.25%     | -36.66% | -35.82%       |\n| 2.0x   | 0.238433 | -7.74%   | -32.15% | -32.42%     | -33.85% | -33.24%       |\n| 3.0x   | 0.298500 | -4.67%   | -12.55% | -13.65%     | -26.40% | -28.21%       |\n\n\n**Metric Improvement: sq\\_rel** (more negative is better)\n\n| ZoomIn | UniMatch | +Plucker | +GTA    | +GTA+CamRay | +PRoPE  | +PRoPE+CamRay |\n| ------ | -------- | -------- | ------- | ----------- | ------- | ------------- |\n| 1.0x   | 0.357467 | -1.08%   | -31.77% | -33.54%     | -45.13% | -41.79%       |\n| 1.5x   | 0.376133 | -9.46%   | -41.52% | -44.06%     | -46.79% | -46.75%       |\n| 2.0x   | 0.407833 | -2.76%   | -41.16% | -42.34%     | -44.53% | -44.08%       |\n| 3.0x   | 0.612533 | -3.37%   | -11.88% | -13.09%     | -38.56% | -39.88%       |\n\n\n**Metric Improvement: rmse** (more negative is better)\n\n| ZoomIn | UniMatch | +Plucker | +GTA    | +GTA+CamRay | +PRoPE  | +PRoPE+CamRay |\n| ------ | -------- | -------- | ------- | ----------- | ------- | ------------- |\n| 1.0x   | 1.058767 | 1.03%    | -19.55% | -20.41%     | -28.64% | -26.86%       |\n| 1.5x   | 1.030967 | -3.95%   | -26.12% | -27.44%     | -29.79% | -29.29%       |\n| 2.0x   | 1.034633 | -0.53%   | -25.68% | -25.72%     | -28.25% | -28.09%       |\n| 3.0x   | 1.188033 | 1.64%    | -4.50%  | -6.12%      | -22.61% | -23.74%       |\n\n**Metric Improvement: rmse\\_log** (more negative is better)\n\n| ZoomIn | UniMatch | +Plucker | +GTA    | +GTA+CamRay | +PRoPE  | +PRoPE+CamRay |\n| ------ | -------- | -------- | ------- | ----------- | ------- | ------------- |\n| 1.0x   | 0.321733 | 0.49%    | -23.29% | -24.08%     | -32.01% | -29.98%       |\n| 1.5x   | 0.321867 | -5.46%   | -30.33% | -31.64%     | -33.56% | -32.68%       |\n| 2.0x   | 0.335533 | -2.33%   | -30.03% | -30.23%     | -32.67% | -32.31%       |\n| 3.0x   | 0.420733 | 2.00%    | -3.24%  | -4.34%      | -26.78% | -27.71%       |"}]

Author rebuttal:

Meta-review: The paper analyzes prior ways of conditioning multi-view transformers with camera information and proposes an extension to prior conditionings that incorporates camera intrinsics. The proposed conditionings (at attention and token level) show consistent improvements across methods.

The reviewers appreciated the analysis and proposed conditioning as a sound & elegant extension to prior work. They noted the consistent and, for some conditions, significant improvements. There were initial doubts whether the attention-level conditioning would provide a benefit over token-level conditioning and what the reason for the strong performance of the latter would be. Further concerns were raised about missing experimental conditions and the paper being a mere extension over prior work. But the extensive rebuttal provided additional results and effectively clarified all remaining critical issues. The reviewers and the AC unanimously agree that the presented approach is simple, novel, well motivated, and clearly shown to provide consistent improvements over prior work.
Decision: Accept (poster)