Title: Cameras as Relative Positional Encoding
Venue: NeurIPS 2025
Accepted: yes
Keywords: multi-view transformer; positional encoding; projective transformation
Abstract: Transformers are increasingly prevalent for multiview computer vision tasks, where geometric relationships between viewpoints are critical for 3D perception. To leverage these relationships, multiview transformers must use camera geometry to ground visual tokens in 3D space. In this work, we compare techniques for conditioning transformers on cameras: token-level raymap encodings, attention-level relative pose encodings, and a new relative encoding we propose—Projective Positional Encoding (PRoPE)—that captures complete camera frustums, both intrinsics and extrinsics, as a relative positional encoding. Our experiments begin by showing how relative conditioning methods improve performance in feedforward novel view synthesis, with further gains from PRoPE. This holds across settings: scenes with both shared and varying intrinsics, when combining token- and attention-level conditioning, and for generalization to inputs with out-of-distribution sequence lengths and camera intrinsics. We then verify that these benefits persist for different tasks, stereo depth estimation and discriminative spatial cognition, as well as larger model sizes.


Peer reviews:
comments: SUMMARY:
This paper introduces a new relative positional encoding scheme for 3D-conditioned tasks which includes the full projective relationship (intrinsics added to prior work including relative 3D encoding just with extrinsics). They demonstrate strong results on improving 3D-conditioned tasks, especially those with out-of-distribution intrinsics.

STRENGTHS AND WEAKNESSES:
Strengths:

- Paper is clear and easy to follow with sound experiments and results
- Well-defined scope and contribution (adding intrinsics to the relative 3D encoding instead of just extrinsics)
- Simple implementation with few lines of code modification for strong results

Weaknesses:

- Contribution over prior work is necessary (adding intrinsics to the relative transformations) but not a very strong/novel change
- Should use additional baseline comparisons, particularly GTA which is the most similar baseline to demonstrate how important the intrinsics encoding are, e.g. in table 1 and figure 6 for the CAT3D and UniMatch comparisons

QUESTIONS:
- How much does including intrinsics help over just extrinsics (GTA)? Figure 3 provides good illustration but more robust and extensive comparison (including the GTA baseline in all experimental results ideally) would be better
- Is there any comparison you can add with different camera models? E.g. with fish-eye or lens-distorted models or is your method limited to just the pinhole camera?

LIMITATIONS:
- Unclear how to extend encoding multi-frequency embedding for higher-frequency prediction, but prior work also had this limitation and not the focus of this work
- Method requires known intrinsics which can be difficult to obtain (but still highly useful when one has them)
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces PRoPE, a projective positional encoding for multi-view transformers that unifies intrinsic and extrinsic camera parameters into a global-frame-invariant, attention-level representation. The method extends prior SE(3)-based encodings by modeling full projective transformations between views. A complementary token-level encoding, CamRay, is used to provide dense per-pixel intrinsics in the local frame. Experiments on novel view synthesis, stereo depth estimation, and spatial consistency show that PRoPE improves generalization, particularly when test-time focal lengths or number of input views differ from training.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n- PRoPE cleanly generalizes attention-level geometric encodings by incorporating intrinsic parameters in a projectively correct and invariant manner.\n\n- The encoding is compatible with FlashAttention and standard transformer layers, and requires minimal architectural changes.\n\n- The model performs well under distribution shifts (e.g., varying focal length or number of views), which are realistic but often overlooked evaluation settings.\n\nWeakness\n\n\n- While mathematically sound, PRoPE is fundamentally an extension of existing methods like CAPE and GTA, augmented with intrinsics. Since intrinsics are low-dimensional and already used in token-level encodings, the conceptual contribution is relatively modest.\n\n- Crucially, the paper does not evaluate GTA+CamRay or CAPE+CamRay, which are the most relevant and fair baselines for comparison. These combinations would test whether the gains from PRoPE stem from its projective formulation or simply from the addition of CamRay-like intrinsic information. Their absence prevents a meaningful comparison and undermines the empirical evidence for PRoPE’s advantage.\n\n- The choice to inject projective encoding into all Q/K/V/O vectors (following GTA) is not ablated or justified against simpler Q/K-only alternatives in RoPE.\n\n- The method is named as PRoPE, and frequency modeling is an important desgin in RoPE. A more interesting and fundamental direction is to consider the frequency modeling in SE3 and 3D geometric space. \n\n- The key contribution of this paper is extending intrinsic encoding to prior SE3 encodings, but how generalise the intrinsic encoding is for different camera models and extreme focus length is not fully analyzed.\n\nQUESTIONS:\n- Would GTA+CamRay or CAPE+CamRay close the gap to PRoPE? This comparison is critical for fair evaluation, as it ensures that intrinsic information is made available to the baseline models. Without this, it is difficult to attribute performance gains to PRoPE’s projective formulation rather than the presence of intrinsics. This is especially important in scenarios with varying focal lengths, where intrinsic-aware baselines may perform comparably if given the same input.\n\n- Can you ablate Q/K vs. Q/K/V/O injection? GTA uses all Q/K/V/O vectors, while methods like RoPE and CAPE apply transformations only to Q/K. It would be valuable to understand whether PRoPE requires the full injection or if a simpler variant would suffice.\n\n- Have you tested PRoPE under more challenging camera models, such as fisheye lenses or other camera model? These cases introduce nonlinear or non-projective distortions, and it is unclear whether PRoPE generalizes to them.\n\n- How numerically stable is PRoPE when projection matrices are ill-conditioned or when intrinsic parameters vary significantly? For example, extremely small focal lengths or near-degenerate calibration settings may cause instability—how does PRoPE behave in such regimes?\n\nLIMITATIONS:\nThe paper mentions numerical instability and lack of multi-frequency modeling as future directions.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new positional encoding for multi-view transformers applied to 3D perception tasks. In this context, encoding and injecting 3D information inside the transformer architecture grounds the model in 3D and is critical for the quality of the resulting predictions. Existing approaches inject these priors either by concatenating dense raymaps to input images before the tokenization or by directly injecting relative pose information in the attention layers. Raymaps are not invariant with regard to transformations between global coordinate systems while existing methods encoding relative pose do not take into account intrinsic information which may hinder performances on testing views with different intrinsics. The authors therefore introduce PRoPE which encodes pairwise projective transformations and injects it in self attention layers. In addition, they introduce CamRays which express dense camera rays in the camera coordinate space instead of the world coordinate space.          \nApplied to models trained for novel view synthesis and depth prediction tasks, PRoPE and CamRays increase the performances of such 3D multi-view transformers and their ability to generalize to different testing intrinsics.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths \n1) The problems associated with existing encoding approaches are well identified. The proposed solutions are sound and tackle the identified shortcomings in an effective way as demonstrated in the experimental section. Injecting intrinsic information explicitly provides camera prior which is more effective than letting the network implicitly capture intrinsic information.    \n\n2) Similarly, CamRay consistently outperforms Plucker rays embedding by preserving invariance wrt. the global frame which allows for better generalization to a larger number of viewpoints demonstrated in the experimental section. \n\n3) The spatial cognition task is an interesting additional experiment that goes beyond existing evaluation metrics and seems to be a good proxy for evaluating the geometric scene understanding of the model.\n\n### Weaknesses \n1) ProPE is very largely inspired by GTA which limits the contributions of the paper. As RoPE is equivalent to SO(2) elements, the only notable difference with GTA is the introduction of intrinsic matrices when computing the relative transformation between pairs of cameras. \nIt has to be noted that the authors clearly stated that ProPE is an extension of existing work and formulated the contributions in that direction. \n\n2) This leads to my second concern. Tab. 1 and Tab. 2 show that applying PRoPE to multi-view diffusion models or stereo depth estimation models improve their performances. However as PRoPE is an extension of GTA, if the intrinsic variations within the dataset are limited/non-existent PRoPE is not expected to perform better than GTA ( see Fig. 3 a, GTA and PRoPE seem to behave similarly). For the same reason GTA is also expected to improve the models performance on these tasks.\nI think it would be less disingenuous to also report CAT3D+GTA and UniMatch+GTA. And if PRoPE improves upon GTA in these experiments, use it to highlight the benefit of your method.\n\n3) As stated by the authors, the main motivation for encoding intrinsic information is application to real world scenarios. However real world cameras have camera models with distortion, as such using this method would require an additional undistortion step in most cases. It would be an interesting addition to also encode and inject distortion parameters in the models.  \n\n4) I find the Figures related to the Spatial Cognition task to be more confusing than informative, it is not obvious to identify the mismatched pair at a glance. \n\n5) Fig 3 b), the x labels {1,3,5} zoom factors on the graphs do not match with the {1,2,3} zoom factors in the legend.\n\nQUESTIONS:\nLimited contribution is not necessarily a restricting factor, however I feel like there should be a more comprehensive set of experiments to clearly demonstrate the benefits of your method, pinpoint the limitations and highlight the scenarios in which your encodings are the most beneficial. For example I find the Intrinsic-aware Benchmarking experiment from the supplementary material to be more relevant than Tab.1 and Tab.2. \n\nIn the spatial cognition task and setting 1 of RealEstate10k experiments , what is the range of viewpoint change between cameras? Can you quantify how CamRay performs vs Plucker embeddings on different magnitude of viewpoint changes? \n\nAs an alternative, have you considered encoding epipolar information between pairs of images? Intrinsics information could also be included in the fundamental matrix. \n\nCould your model be easily extended to account for distortion?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 1, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n## Context\nRecently, multi-view transformer architectures became increasingly popular for 3D tasks. For each input view, the viewpoint, in terms of camera extrinsics and intrinsics, needs to be provided to the model. Previous works use different ways to provide those camera parameters to the model, summarized by the paper under review as follows:\n- token-level conditioning: \n\t- concatenate camera conditioning to the respective image tokens\n\t- methods: raymaps, Plücker raymaps\n\t- can encode extrinsics and intrinsics\n\t- encoding depends on the choice of the reference frame\n\t- sensitive to global SE(3) transformations\n- attention-level conditioning:\n\t- inject pairwise camera conditioning into the attention mechanism\n\t- methods: CAPE (from EscherNet), GTA\n\t- existing methods only excode extrinsics, but not intrinsics\n\t- encode relative poses between pairs of views and are therefore invariant to the reference frame\n\t- invariant to global SE(3) transformations\n\n## Method\nTo resolve the shortcomings of previous methods, the paper under review proposes a new encoding method called PRoPE+CamRay. PRoPE essentially extends GTA to also incorporate the intrinsics into the relative positional encoding between pairwise views. CamRay additionally provides the intrinsics to the model via token-level conditioning in form of an intrinsics-only-raymap.\n\n## Experiments\nThe most relevant experiments are conducted for the task of novel view synthesis on RealEstate10k, using LVSM as base model. The results (Fig. 3, 4, 5) show that PRoPE very slightly outperforms the best previous method GTA regarding a) performance with similar settings at test time as in training, and b) more input views at test time. In case of c) using different focal lengths at test time, it clearly outperforms GTA. The PRoPE+CamRay variant always outperforms PRoPE and has a larger margin to the best previous method GTA. \n\nFurther experiments are conducted with CAT3D, with LVSM on Objaverse, for stereo depth estimation with UniMatch on 3 datasets, and for a spatial cognition-task on DL3DV. The comparisons in those experiments are less conclusive (not all conditioning methods are evaluated). However, in all results, using PRoPE outperforms the performance of the respective baseline model.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n### Elegant method\nThe method is elegant, especially regarding how it comprises previous methods: in case of identity intrinsic matrices, PRoPE falls back to GTA/CAPE and in case of single-view inputs, PRoPE falls back to RoPE. \n\n### Writing\nThe paper is very clearly written and good to follow.\n\n### Consistent improvements\nThe paper under review presents results for a variety of tasks. In all cases, PRoPE improves the results. The more detailed experiments for NVS on Re10k (Fig. 3) show that PRoPE consistently outperforms the previous best GTA. Even though the improvement is small in some cases, it is consistent, and for the case of generalization to different intrinsics at test time, the improvement is significant. The additionally proposed CamRay method further boosts results. \n\n## Weaknesses\n### Unclear intuition for CamRay\nThe main difference of PRoPE over GTA is to incorporate the intrinsics into the relative positional encoding. However, CamRay additionally provides the intrinsics to the model via token-level conditioning. It is shown that this clearly improves the results. However, it remains unclear why this additional usage of token-level conditioning is required. Additional explanations would be helpful.\n\n### Limited evaluation of CamRay\nIt seems like CamRay was only evaluated in the experiments for NVS on Re10k (Fig. 3) and the spatial cognition experiments (Tab. 3). The experiments with CAT3D (Tab. 1) and UniMatch (Tab. 2) seem to be without CamRay. It would be helpful to also evaluate CamRay in these experimental setups, to provide further evidence of its effectiveness.\n\n### Limited systematic analysis\nA major claim of the paper is a \"systematic analysis\" of different camera conditionings (line 4-5). However, such a systematic analysis is actually only conducted in one experimental setup, namely NVS on Re10k with LVSM (Fig. 3). The \"systematic analysis\" would be stronger, if it would be conducted also on the other tasks. \n\n## Comments\nSometimes the authors write Plücker and sometimes Plucker coordinates.\n\nIn the experiments for NVS with varying focal lengths (line 214): does this consider context views with varying focal length, or rendering target views at different target focal lengths, or both?\n\nFigures are sometimes not referenced from the text, for example Fig. 1 or Fig. 2. \n\nFig. 1b): why should PRoPE generalize better to unseen extrinsics? I understand the point that invariance to the reference frame helps generalization in that PRoPE should be able to generalize to unseen absolute poses, as long as the relative poses have been seen during training. However, I do not understand why it should generalize better to unseen relative pose configurations. To illustrate what I mean, here is an LLM example:\n- The training dataset only contains sentences where the words \"the\" and \"likes\" appear at absolute positions 0 and 2.\n- During inference, when using APE, a model would struggle if those words appear at positions 10 and 12. When using RPE, this should not be a problem, as the positional encoding is invariant to the absolute position. \n- However, when those words appear at positions 0 and 12, I would expect that both APE and RPE have the same difficulties, as also this relative position is unseen. \n- Actually the text describes this better than Fig. 1b) as it says that some methods are \"sensitive to SE(3) transformations\" (e.g. line 90), which seems to be a better description than the \"Robust to unseen extrinsics\" in Fig. 1.\n\nQUESTIONS:\nMy key suggestions are:\n- Provide an intuitive explanation why CamRay is necessary. \n- Evaluate CamRay also in the UniMatch and CAT3D experiments (Tab. 1 and 2).\n- Evaluate previous methods (Plücker, GTA, CAPE) also in the other tasks to make the \"systematic analysis\" more extensive.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-03"}