RoRE: Rotary Ray Embedding for Generalised Multi-Modal Scene Understanding
Authors: Ryan Griffiths
Donald G. Dansereau
Venue: ICLR 2026 Poster (2026)
Keywords: transformers
attention
geometric vision
multi-modal vision
novel view synthesis
thermal
fisheye
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Transformers have emerged as powerful implicit rendering models, capable of performing geometric reasoning and producing photorealistic novel views in a single feedforward pass. A central challenge in these architectures is how to inject camera parameters into the transformer in a way that generalises across diverse sensing conditions. In this work, we present Rotary Ray Embedding (RoRE), an approach that embeds image patches directly as rays, using a learning based rotary positional embedding (RoPE). This ray-based formulation provides a unified and general representation, improving robustness to unconventional camera geometries and sensing modalities. We evaluate our approach on conventional perspective imagery, fisheye cameras, and multi-modal RGB-thermal setups, showing that a single network can flexibly integrate arbitrary numbers of cameras and modalities into a coherent scene representation. Experiments demonstrate improved generalisation and cross-modal consistency compared to existing methods, highlighting the potential for relative ray-based embeddings to build adaptable, plug-and-play vision systems. Code available at: https://roboticimaging.github.io/RoRE

Published as a conference paper at ICLR 2026
RORE: ROTARY RAY EMBEDDING FOR GENERALISED MULTI-MODAL SCENE UNDERSTANDING
Ryan Griffiths, Donald G. Dansereau
Australian Centre for Robotics, School of Aerospace, Mechanical and Mechatronic Engineering, The University of Sydney {r. griffiths, donald. wwweeddaanseeeayyeeneea edu. au
ABSTRACT
Transformers have emerged powerful implicit rendering models, capable of per- forming geometric reasoning and producing photorealistic novel views in single feedforward pass. A central challenge in these architectures is how to inject cam- era parameters into the transformer a way that generalises across diverse sensing conditions. In this work, we present Rotary Ray Embedding (RoRE), an approach that embeds image patches directly as rays, using a learning based rotary posi- tional embedding (RoPE). This ray-based formulation provides unified and gen- eral representation, improving robustness to unconventional camera geometries and sensing modalities. We evaluate our approach conventional perspective imagery, fisheye cameras, and multi-modal RGB-thermal setups, showing that a single network can flexibly integrate arbitrary numbers of cameras and modalities into a coherent scene representation. Experiments demonstrate improved general- isation and pross-modal consistency compared to existing methods, highlighting the potential for relative ray-based embeddings to build adaptable, plug-and-play vision systems. Code available at: https//roboticimaging www.ggiitth/htte io/RoRE
INTRODUCTION
Recent advances in vision transformers have shown their ability to unify geometry and appearance across multiple views, enabling rapid and accurate scene understanding 3D perception tasks Jin et al. (2025); Wang al. (2025). A critical design choice in these architectures is how to inject camera information into the model so that it can accurately align visual tokens with the underlying 3D scene structure. While absolute relative positional encodings have been proposed, it remains an open question how to best represent camera geometry in a way that is both expressive and gener- alisable.
This challenge is amplified in heterogeneous settings, where inputs may differ in resolution, field of view, and alignment. Multi-modal scenarios further exacerbate the problem: for instance, RGB and thermal cameras exhibit radically different photometric characteristics while still needing to be integrated into coherent representation. Existing approaches often make strong assumptions about camera intrinsics, sensor type, or rely on manually engineered fusion strategies Lu et al. (2025), which limits flexibility.
In this work, we are interested in building robust 3D vision models that work for a range of camera families. that end, we introduce RoRE, a ray-based extension RoPE Su al. (2024). Instead of embedding patches by index, RoPE parameterises each patch as ray, directly encoding where in the scene looking. To realise this, we extend RoPE two key ways: (i) the rotation frequen- cies learned rather than fixed, and (ii) asymmetric rotations forward- -ackward symmetry and encourage more uniform attention across the scene. This formu- lation allows transformers designed for conventional imagery to naturally extend to new camera geometries and sensing modalities.
applied to break the inherent
Unlike recent fose-free architectures such DuST3R Wang et al. (2024b), VGGT Wang et al. (2025), and MapAnything Keetha et al. (2025), our approach is designed for settings where camera poses available be easily obtained, for instance, in fixed multi-camera rigs commonly

---

Published as a conference paper at ICLR 2026
found on vehicles (Geiger et al., 2013; Ettinger et al., 2021). Pose-free methods are valuable in scenarios where calibration is difficult or impossible, but they typically require substantially larger networks and greater computational resources. In contrast, RoRE leverages available pose informa- tion to deliver a more efficient and generalisable solution.
We further demonstrate our method's ability fuse multi-modal information, specifically RGB- thermal inputs. Thermal cues complement visible imagery and enable perception in challenging conditions such as fog, smoke, darkness, occlusion, supporting applications including asset in- spection and search-and-rescue, A unified embedding of RGB and thermal rays also benefits down- stream tasks such as classification, anomaly detection, segmentation, and general multi-modal scene understanding, By operating directly on ray-based inputs from heterogeneous sensors, RoRE pro- vides a shared geometric representation without requiring separate architectures for each modality.
We validate RoRE across five datasets, including perspective and fisheye imagery as well as syn- thetic and captured multi-modal RGB-thermal datasets. Our experiments demonstrate that ray-based rotary embeddings provide improved generalisation across camera geometries and modalities, while enabling a single network to flexibly integrate arbitrary numbers of heterogeneous inputs.
In summary, this paper makes the following contributions: (1) A novel rotary embedding RoRE that unifies absolute and relative encodings ray-based formulation. (2) multi-modal training scheme using modality-specifie tokenisers with shared ray-based embeddings, trained via masked cross-modality prediction for robust fusion. (3) Empirical validation across conventional, fisheye, and
datasets, showing improved generalisation over state-of-the-an baselines. (4) A synthetic indoor multi-modal dataset of 4,000 scenes with ground-truth poses, released with code for data generation and experiments.
This work paves the way for more general network architectures that can be deployed across diverse camera setups, facilitating broader adoption and practical use.
2 RELATED WORK
Positional Embeddings in Transformers. Transformers rely on positional encodings to inject structure into sequences of tokens, due to being permutation invariant. Early transformers adopted additive absolute positional encoding (APE) (Vaswani et al., 2017; Devlin et al., 2019), which adds specific bias to vector based on position value. Relative positional encoding (RPE) extends this idea by representing pairwise relationships between tokens. One key feature of this is transla- tional invariance, which leads improved generalisation. This is typically done with either additive bias terms depending on relative position between tokens (Shaw et al., 2018) or rotary positional embedding where the angle of rotation is based on position (Su et al., 2024).
Vision transformers (Dosovitskiy et al., 2021) have the same challenge in 2D where the position of the patches being encoded need to be injected. This has been done in both APE (Oquab al., 2023) and RPE. However RoPE has become a popular choice of RPE for embedding the patch location (Heo et al., 2024).
Geometric Vision. Geometric vision is a line of work with a goal to integrate information from multiple cameras into unified scene understanding, with range of tasks such as novel view syn- thesis (Suhail et al., 2022), pose (Shavit et al., 2021) or depth (Yang et al., 2024a;b). Transformer methods such as DuST3R (Wang al., 2024b) and VGGT (Wang et al., 2025) demonstrate that large transformer models can reconstruct geometry and synthesise views without explicit 3D su- pervision. The Large View Synthesis Model (LVSM) (Jin et al., 2025) uses pose information for implicit rendering and achieving strong performance with minimal architectural assumptions. No- tably, the decoder-only variant of LVSM reduces the architecture to a single stack of self-attention layers. In this work adopt the LVSM architecture the basis for our experiments given its simplicity and strong performance.
Multi- Modal Vision. Beyond multiple viewpoints, recent research addresses fusing heterogeneous modalities into a shared representation. They target a range of different tasks such as cross-modal super resolution (Amold et al., 2024) glass segmentation with RGB-thermal fusion (Huo et al., 2023). Meng & Fukao (2025) fused RGB and thermal information to produce robust depth esti- mation. Bachmann et al. (2022) proposed MultiMAE which learss a general- transformer
2

---

Published as a conference paper at ICLR 2026
backbone that can process RGB, depth, and other modalities in unified token space, however it requires confocal images, which is something we address in this work.
Hassan et al. (2024) adapt NeRFs (Mildenhall al., 2021) to work with RGB and thermal images while Chen et al. (2024) adapts Gaussian splatting (Kerbl et al., 2023) techniques to work for ther- mal images and Lu et al. (2025) extends it further fuse thermal and RGB imagery into one scene understanding. These works demonstrate the benefits of complementary information across sensors and motivate extending view-synthesis transformers to multi-modal inputs, where alignment and fu- sion strategies remain open challenges. To our knowledge, no work exists that perform feedforward modal novel view synthesis, this is key development of our work.
Injecting Pose and Modality Information. A central question in transformer-base implicit ren- dering is how inject camera pose and modality information. To faithfully capture camera pose and viewing geometry Sajjadi et al. (2022) embeds this information through additive means to the tokens. Similarly, Jin et (2025) employs Plücker ray embeddings to represent viewing geome- try using APE. CaPE (Kong et al., 2024) and GTA (Miyato et al., 2024) investigate conditioning attention on relative camera transformations, based on rotational biases. Li et al. (2025) propose PRoPE which encodes entire camera frustums as relative positional embeddings. These works use the fact that relative embedding should generalise better, however these works move away from the ray-based representations, and this hinders generalisation.
Other approaches have opted to rely only on APE, as in MapAnytiing (Keetha et al., 2025), where they state the RoPE method tends to lead to unnecessary biases. This is something aim to mitigate in this work.
For embedding additional modalities VRoPE (Liu et al., 2025), Qwen2- VL (Wang et al., 2024a) and other related approaches extend rotary embeddings to handle a temporal dimension. Multi- MAE (Bachmann al., 2022) emheds multiple modalities into single latent space. This is done using modality specific patch encoding on inputs and specific modality heads for outputs, we adopt similar approach in this work.
3 ROTARY RAY EMBEDDING
A central goal of this work is to combine the benefits of relative positional encodings with the generality of ray-based representations. Ray embeddings provide a unified way represent im- age patches across diverse camera types, while relative encodings capture geometric relationships between views. To achieve both, we build on the RoPE formulation and introduce key modifica- tions: embedding rays directly, learning rotation frequencies, and using asymmetrie position values to reduce attention biases. Using this embedding design, we develop multi-modal geometric trans- former that leverages proposed RoRE, adapting the architecture handle heterogeneous sensor modalities and integrate diverse camera inputs into a coherent geometric representation.
3.1 PRELIMINARIES
Su et al. (2024) introduced RoPE which is designed to perform rotations of d dimensional vector as way embed relative positional information into a new vector Trotated- Mathematically this is:
Protated fRoPE( m):
[ (2, (m) = R,,,,
(2)
where Ry, is dimensional rotation matrix constructed by multiple 2D rotation matrices:
R = diag [m0)(, (m+2), R2d R2d(m axd
(3)
here (²2(()) is the SO(2) rotation matrix with angle 0. The 0 is predefined based on the following:
0, = -100002(-11))) 2(/-1)/d
(4)
for i E [1, d/2). This provides decay of the rotation frequencies (Vaswani et al., 2017). For further details and formal explanation see Su et al. (2024).
3

---

Published as a conference paper at ICLR 2026
While this was originally applied to dimensional positional information for language models, has also been applied to two dimensional positional embedding of pixel indices (Weinzaepfel ct al.,
2023; Wang et al., 2024b; Leroy et al., 2024). To do this the vector is split into two [2 (1) (2)
where one half is rotated according some pixel index and the other half is rotated based on some pixel index v. Giving:
Protated
|f(0 (2) u). f( (2) v)].
(5)
In our case we do not want to pixel indices instead we want to encode rays.
3.2 EMBEDDING RAYS
In this work, we embed single ray for each image patch. Specifically, we use the ray corresponding to the patch centre, computed as the average of the rays of all pixels within that patch. This ray is then used to embed the position of given patch. Building from base RoPE, we now have higher dimensional positions, in our case the 6 that are required to represent the Plücker ray. Which has position (or moment in the case of Plücker coordinates) dimensions t and direction dimensions d. Meaning new strategy needs to developed to embed this information. straightforward extension is to break up the embedding further into 6 parts = (1//0 [6/6 ..... x////
leading to:
Protated
i (a 1 tt)
(2) ( (1) (10/2/1))))
a: 12 12), (0/8 de) (5/ (5) dy). (2/0 ( (G)
d.
We note, the more dimensions being embedded the more fragmented the vector becomes, essentially putting additional constraints on the latent space. The position and direction components differ fundamentally in magnitude and semantic meaning. The magnitude of the frequencies required for translations values is likely to be different to that of the direction vectors.
We propose another approach by replacing the standard handcrafted base frequencies Eqn, 4 with learned frequencies for each dimension, superimposing their contributions without fragmenting the embedding space. This allows the network to learn how position dimensions interact with different parts of the latent space. Our frequencies 0 pxd/2 have size p 2. where 6 represents the ray dimensions and d is the query/key token dimension.
The final rotation around given plane in the d dimensional space is the superposition of all the learned values scaled by their respective ray-position values:
R2d (P,0,) = =R2d CPp 0ip)
where P. is the position value vector (y dey dyy dz) containing the position values for patch, 0, is the learned frequencies across all position dimensions for a given rotation plane i [1, .... d/2]. Our RoRE formulation then becomes:
Protated = .RORE (,
(8)
FRORE (x, P) == R,,,,,
(9)
where
R. diagiR RARE 0,))), R2d RARE (P, P2) RRd R (P, (a/2)..
(10)
The learned @ parameter randomly initialised using uniform initialisation between 0 0.5. is left to future research to look into alternative initialisation strategies and how this effects performance.
Fig. shows the learned rotation frequencies. It is encouraging that the model discovers de- cay structure similar to the handcrafted schedule, despite being trained without any explicit con- straints. This behaviour aligns with the established research: representing ray geometry requires a spectrum of frequencies, with higher-frequency components capturing fine-grained variations and lower- freerffr components capturing broader spatial trends. The resulting learned decay therefore mirrors the intended multi-scale behaviour of classical positional embedding, providing evidence that the learnt embedding parameters can autonomously recover a meaningful and interpretable fre- quency structure.
4

---

Published as a conference paper at ICLR 2026
Learat Rotation Frequencies
Learut vs Hamderafted Rotation Frequencciss
0.15
Learnt. Poeition Freq
Normalised Preqencc
Learu: Poeition Freq
Learut Directinn Freq
0.8
Learu; Direction Freg
Frequeney
Handerafted Prog
3.10
0.6
0.4
0.05
0.2
0.00
0.0
10
15
200
10
15
100
Frequency Index
Frequency Index
Figure 1: Comparison of learned vs handcrafted frequency. Left compares the learned frequency for the position and direction dimension, it shows the magnitude of rotations that has been learned is larger for position than it is for direction. Right is comparing the normalised position and direction frequencies to the standard handcrafted frequency from Eqn. 4. While similar the learned frequen- cies differ from the handcrafted ones.
There are also clear differences between the position (moment) and direction dimensions, both in frequency decay and scale. This expected, as the two quantities encode different geometric in- formation. Positional components span broader normalised range [0, 1]. whereas direction vectors vary more subtly due to camera motion constraints and overlapping fields of view. Consequently, the model allocates higher effective frequencies to direction channels and lower ones to position channels. The slightly sharper decay for direction likely reflects the finer rotational relationships be- tween neighbouring rays. Overall, these patterns indicate that RoRE learns meaningful multi-scale structure without requiring handcrafted schedules.
A key benefit of this method is it removes the manual hyper-parameter selection process that is required for the handcrafted method. We note that the ablation study (Tab. 4) shows very similar performance between the method outlined in Eqn. 6 and the learnt method in Eqn. 8. Due to the benefits of the learned-basee based approach outlined above we use this method for our proposed approach.
3.3 ASYMMETRIC ROTATIONS
Standard RoPE, originally developed for NLP, is designed to emphasise local interactions by causing attention magnitudes to decay the positional distance between tokens increases Su et al. (2024), which is a natural property of RoPE's formulation. While beneficial for sequence modelling, this behaviour is undesirable in 3D vision, where rays that are far apart in image space may still hold important geometric relationships. To remove this distance-dependdnt attenuation, we extend an approach taken in VRoPE (Liu al., 2025) where a shifted negative counterpart of each positional component, ensuring that encoded magnitudes remain consistent across the ray domain. This mod- ification preserves RoPE's rotational properties while preventing the unintended decay in attention, making the embedding better suited to ray-based scene representation. This can be expressed as:
|t+ ",,t,d d _ = [t. d, d] - [0, baheft, O, bahsfe]. P
(11)
where P is the position vector for a given patch, with t and d being the translation and direction components of the ray respectively. The bahift is equal to in our case, as the position and direction d values normalised have maximum value of 1. In practice in the 0 pxxds the p dimension is actually twice the size of the original position vector.
Fig. 2 shows visual comparison of embedding rays with and without the asymmetric positioning. For this comparison we take unit vectors for query and key tokens and compute the attention score between them after the rotary ray embedding has been applied. This is calculated for 3 images with different poses, with the attention scores being normalised. Without the asymmetric positioning the attention score is not uniform across the patches meaning is biased toward rays near the query ray. While the asymmetric positions provide much closer to uniform attention across the frames.
3.4 MULTI-MODAL GEOMETRIC MODELS
A central goal of this work is to design models that are adaptable to diverse sensing modalities while retaining geometric consistency. We achieve this by leveraging transformers' ability to understand
5

---

Published as a conference paper at ICLR 2026
Image b-1
Image
Inuge 6++1
Pach
Stardard RoPE
Positions
Artention Scote
Tw
T
Pard
Image t-1
Image
Imago 1-1-1 Asymunetric
Futeh
Positions
Pardh
Figure 2: Attention Comparison. Attention between frames different positions using the Plücker parametrisation. The attention score between query patch, identified in red, and all other patches shown. Unit query and key vectors are used for this demonstration. The standard RoPE position values bias attention rays the query ray. This is problematic because geometric correspon- dences need not be spatially local. The asymmetric approach removes this bias providing a more uniform attention across possible position values.
multimodal as shown in MultiMAE (Bachmann et al., 2022), with the generic ray based positional embedding. This enables single forward pass jointly reason over RGB, thermal, and depth information. Unlike confocal formulations, our method operates directly on posed images, using photometric self-supervision learn cross-modal correspondences.
Each modality is equipped with its own input tokeniser and output head within an encoder-decoder structure, following the general design of LVSM (Jin et al., 2025). Further architectural details and a diagram are provided in Appendix 1.1. To represent modality information, we use both absolute embeddings and modality-aware RoRE embeddings. This adds a modality class to the position vec- tor P, resulting moddllity
[P ('modality), where C'modality is numerical class modality of the patch. While relative encoding is not directly applicable to discrete modality classes, embedding them within the same framework ensures consistency with the geometric positional encoding.
While our multimodal formulation does not require depth supervision, we include head to show that RoRE supports explicit geometric estimation, In this work, we use ground-trume depth from dataset to produce metrically meaningful depth alongside RGB and thermal outputs. natural extension is to adopt self-supervised depth learning Zhou et al. (2017), enabling operation in settings without ground-truth depth.
preph-ppreditiio
Masked input strategies have proven effective recent vision research, particularly in encouraging models to develop generalisable and semantically rich internal representations (He al., 2022). Inspired by this, we extend masked input strategies to multi-modal settings, where we mask input image patches based a fixed masking ratio. This allows the model to learn interpolate across both missing spatial regions and absent modalities.
Training samples consist of randomly selected context and target views drawn from varying modality combinations to encourage robustness under asymmetric input conditions. The network is trained with a combination of photometric losses (MSE and perceptual) and depth loss that enforces geometric consistency. While the depth loss is not necessary for CGBBtterme fusion and rendering, is an additional constraint on geometry and allows for explicit depth estimation from the network. Full details of the training setup and loss formulations are provided in Appendix 1.1.2.
4 EXPERIMENTS
We perform two sets of experiments: RGB experiments evaluating robustness across camera geome- tries, and (ii) Rhe-maermal experiments evaluating the model's ability to generalisation multi-modal data. Unless otherwise stated, all experiments ferent but known poses. These images correspond to those shown in the qualitative examples.
with two input images from scene with dif-
Implementation Details. For the RGB experiments, we adopt the LVSM architecture Jin et al. (2025), which provides a simple yet highly effective baseline for view synthesis. This choice en- ables controlled and standardised comparisons with alternative methods, though proposed em- bedding scheme could be integrated into a wide range architectures. To reduce computational requirements, scale the model size down following Li et al. (2025).

---

Published as a conference paper at ICLR 2026
Table 1: Novel view synthesis results. Results from RealEstate 10K (training domain) and DL3DV (unseen but similar domain), All methods perform comparably, with LVSM performing marginally worse and PRoPE performing marginally better. Method marked with f represents concurrent work.
RealEstate 10k
DL3DV
Iteration Time
Method
SSIM(t) PS
LPIPS(
PSNR(1)
SSIM(1)
LPIPS(L)
Seconds
LVSM
26. 18
0.834
0.076
19.48
0.604
0.281
1.287
GTA
26.74
0.846
0.069
19.55
0.614
0.281
1.647
PRoPE
26.81
0.848
0.068
19.68
0.620
0.278
1.454
RoRE (burs)
26.65
0.845
0,070
19.77
0.619
0.279
1.326
Table 2: Quantitative evaluation under varying focal lengths. Models are tested without retrain- ing by cropping target and query images to simulate changes in camera intrinsics. Methods with stronger representation constraints (GTA and PRoPE) fail to adapt, while LVSM and RoRE remain robust. RoRE consistently outperforms LVSM, demonstrating the advantage of a relative ray- embeddings for generalisation across intrinsics variations.
RealEstate 10k
DL3DV
Method
PSNR(1)
SSIM(1)
LPIPS(L)
PSNR(t)
SSIM(t)
LPIPS(()
LVSM
21.95
0.744
0.219
19.86
0.653
0.349
GTA
14.81
0.523
0.459
14.47
0.469
0.564
PRoPE
14.71
0.516
0.486
14.28
0.454
0.617
RoRE (ours)
22.66
0.770
0.211
20.31
0.678
0.335
For the multi-modal experiments, we employ a modified architecture in which target and query im- ages interact through tross-attention. This design is motivated by prior work such as DuST3R Wang et al. (2024b) and related models, where cross-attention has proven to be a reliable mechanism for integrating information across views. Additionally we are learning depth maps we replace the Plücker coordinates with the simpler raymaps, that encode rays as position and direction, instead of moment and direction. Further implementation details, including model configurations, training parameters, hardware setup, and training time, are provided in Appendix A.I.
Datasets. We evaluate our approach across diverse set of datasets spanning conventional per- spective imagery, fisheye imagery, and multi-modal RealEstate10K 10K (Zhou et al., 2018), a large-scale dataset of posed perspective videos that serves as the primary basis for our models. To assess generalisation to unseen conventional imagery, we further evaluate on DL3DV (Ling et al., 2024), which contains data of a similar type but is not used during training. To test robustness to novel camera geometries, we employ FIORD (Gunes et al., 2025), a fisheye dataset that provides a challenging departure from the perspective imagery seen dur- ing training. For multi-modal experiments, we introduce MultiModalBlender, simulated dataset of RGB, thermal and depth images. Large-scale multi-modal datasets are scarce, particularly at the scale required for training transformers, and this dataset enables training with modsl fusion. To validate performance qualitatively on real multi-modal imagery, we make use of the Thermal- Gaussian dataset (Lu et al., 2025), which provides data captured in real-world conditions. Together, these datasets allow us to evaluate generalisation across camera geometry, and sensing modality.
data. For training, we use
Baselines. For validating our relative ray-based embedding method we compare to methods of novel view synthesis that perform positional embedding in different ways with different information: LVSM (Jin al., 2025) which is an absolute positional embedding only method as described in their paper; GTA (Miyato et al., 2024) which uses both ray based absolute embedding and their relative encoding of camera extrinsics; and finally concurrent work PRoPE (Li et al., 2025), which embeds their own camera base absolute embedding and modified GTA relative embedding that also embed camera intrinsic using a projection matrix. All methods including ours uses the exact same model architecture keeping all parameters the same except for varying the embedding method.
We are unaware of any existing alternative feedforward multi-modal models, as such we do not perform direct comparisons to alternative multimodal feedforward methods for our multi-modal approach instead we show different operating modes to characterise its performance. We evaluate reconstruction quality using PSNR, SSIM, and LPIPS for both RGB and thermal outputs, noting that perceptual metrics such as LPIPS were not originally designed for thermal imagery and therefore cannot be directly compared to their RGB counterparts.
7

---

Published as a conference paper at ICLR 2026
Table Quantitative evaluation on distorted and fisheye inputs. Barrel-distorted RealEstate 10K images and native FIORD fisheye images are used as inputs without retraining. RoRE generalises robustly to both cases, while GTA and PRoPE fail due to the absence of explicit ray-direction en- coding.
Distorted REIOK
Fisheye FIORD
Method
PSNR(()
SSIM(T)
LPIPS(()
PSNR(1)
SSIM(t)
LPIPS(L)
LVSM GTA
21.99
0.725
0.142
22.52
0.732
0.310
18.58
0.605
0.188
11.64
0.456
0.596
PRoPE
18.57
0.605
0.188
11.90
0.408
0.673
RoRE (ours)
23.96
0.802
0.124
23.55
0.746
0.284
Input Images
LVSM
GTA
PRoPE
RoRE
Ground Truth
Distorted
Fisheye
Figure 3:
Qualitative results on distorted and fisheye inputs. RoRE preserves scene structure under both bartel-distorted perspective images (top) and native fisheye images (bottom), whereas competing methods produce severe artefacts or fail to reconstruct meaningful views.
4.1 RORE EMBEDDING
Tab. reports novel view synthesis results on RealEstate and DL3DV. All models are trained on RealEstate 10K, with evaluation on the same dataset reflecting in-domain performance, and DL3DV providing an unseen but closely related test set. Across both datasets, the methods achieve broadly comparable results: PRoPE performs slightly better the training domain, while LVSM marginally lower. While these results do not highlight a clear advantage for our method, they establish that RoRE remains competitive standard benchmarks, with its benefits becoming more evident in settings that require greater generalisation, as demonstrated in subsequent experiments.
Varying Intrinsics. We evaluate robustness to changes in camera intrinsics by varying the focal length of target and query images through cropping, with randomly chosen magnification of up to This experiment was conducted without retraining the models. As shown in Tab. 2 and Fig. in Appendix 1. methods with stronger representation constraints, such as GTA and PRoPE, fail to adapt. In contrast, LVSM and our RoRE handle these variations effectively, with RoRE consistently outperforming LVSM. While PRoPE can address this case with additional training. our results highlight the inherent advantage of ray-based embeddings, which are naturally invariant to changes in intrinsics.
Distorted and Fisheye Inputs.
We next test robustness to non con-perspective inputs, using (i) perspective images from RealEstate 10K with added barrel distortion and (ii) native fisheye images from FIORD shown in Tab. and Fig. 3. Distorted inputs are paired with perspective queries, and all evaluations are conducted without re- training. Our method demonstrates consistently stronger generalisation than competing approaches, outperforming LVSM by over 1 dB in PSNR. while GTA and PRoPE fail due to the absence of explicit ray-direction encoding. The fisheye case particularly relevant, since rectification would reduce field of view; RoRE can handle these inputs directly, capturing the global distortion, though some local inaccuracies remain, see Sec. A.2.2 for further details.
Discussion. These results reflect the differing representational biases of the methods. PRoPE uses a constrained, camera-specific formulation based on projection-matee matrix relative encodings, which aligns well with conventional perspective data such as RE10K, with GTA imposing even more
8

---

Published as a conference paper at ICLR 2026
Table 4: Ablation study on RE10K. The study shows adding the relative embedding in any form improves the performance. The asymmetric positioning provides another modest improvement to performance. Applying the learnt frequency vs handcrafted produces identical results but as stated provides a more general approach. RoRE performs similarly whether the additional absolute em- bedding applied or not.
Absolute Emmh
Relative Emh.
Learnt Frequencies
Asymmetric
PSNR(1)
SSIM(1)
LPIPS(()
X
26.18
0.834
0.076
X
26.56
0.843
0.071
X
V
26.65
0.845
0.070
X
26.57
0,842
0.072
V
/
26.65
0.843
0.070
26.65
0.845
0.070
Input Images
Predicted Images
Ground Truth
P1
P
Figure 4: Qualitative results on the MultiModalBBender dataset. Pi and P2 denote different camera poses. This figure shows the model's ability to infer missing regions of one modality using cues from another in a geometrically consistent way: (red region) green chair partially visible in the RGB input is completed using thermal cues, while red chair absent in RGB inputs has its shape inferred but colour misestimated, (blue region) the thermal domain, a partially visible chair is reconstructed with accurate geometry and appearance by leveraging RGB information. Depth maps remain consistent across all modality settings.
restricted variant with just the extrinsics. RoRE, by contrast, encodes full rays and learns multi- dimensional frequency interactions, resulting in more expressive and geometry-agnostic repre- sentation. This flexibility leads to substantially stronger generalisation under intrinsics changes, distortion, and fisheye inputs (Table 2, 3). However, its less constrained embedding space can yield slightly lower performance tightly scoped perspective datasets such REIOK and DL3DV (Ta- ble 1).
Ablation Study. We ablate different components of our method, shown in Tab. 4. Firstly, the learnt frequencies refers to the method outlined Sec. 3.2. The method without it refers to the process outlined in Eqn. 6. Asymmetric refers the method outlined in Sec. 3.3. Including asymmetric po- sitioning provides a modest increase to performance. Using learned frequencies yields performance comparable the handcrafted schedule. However, we note that this formulation is more general solution that removes the need for additional hand tune parameters and handcrafted elements, for this reason our proposed method utilises these learnt frequencies, as they do not impact performance or inference time.
Since the RPE is separate and complementarily the APE we can use any combination of the embedding methods. The study demonstrates that the performance boost comes from using the RPE compared the APE, when including both the performance roughly the same, this is an indication that the network relies on the most appropriate embedding information. Our relative embedding method could work just fine without the absolute embedding, however we do include in the other experiments as it does show slightly higher SSIM score.
4.2
MULTI-MODAL SCENE UNDERSTANDING
Simulated Results. We train single model capable of handling different modality combinations and numbers of input images. Quantitative results the MultiModalBlender dataset (Tab. 5) show robust performance across all modality configurations, with slightly reduced depth accuracy for thermal-only inputs due lower texture information. The lower absolute metrics seen compared to

---

Published as a conference paper at ICLR 2026
Table 5: Quantitative results on the MultiModalBlender dataset. One model under different in- put modality configurations (RGB-RGB, RGB-thermal, and thermal-therman The results demon- strate that single model can generalise across all cases.
RGB
Thermal
Depth
Input Images
PSNR(1)
SSIM(1)
LPIPS(L)
PSNR(1)
SSIM(1) LPIPS(L)
AbsRel(())
RMSE(L)
6,(1)
GGB-RGB
22.995
0.674
0.218 0.254
0.060
0.024
0.965
RGBB-thrrml
21.494
0.634
20.481 21.662
0.807 0.825
0.172 0.153
0.060
0.025
0.964
themma-thermal
0.065
0.027
0.959
Imput Images
Predicted Images
Ground Truth
Inpor Images
Predicted Images Ground Truth
Building
Truck
Figure 5: Qualitative results on the ThermalGaussian dataset (Lu et al., 2025). The model generates consistent RGB-thermal renderings without additional training.
RealEstate 10K due more complex motion in the dataset. Qualitative examples (Fig. illus- trate the model's ability to fuse modalities; partially visible objects in one modality are completed using cues from the other. For example, (red region) a green chair partially visible in the RGB input is completed using structural cues from the thermal image, while red chair absent from the RGB inputs has its shape inferred correctly but its colour misestimated. Depth predictions remain co- herent, reflecting a unified cross-modal understanding of scene geometry. Additional experiments, including progressive input masking and multi-camers evaluations are provided in Appendix A.2.4.
Real-world Results. Fig. 5 shows qualitative renderings on the ThermaGGaussian dataset (Lu et al., 2025). Due to the dataset's limited size, we evaluate via inference only. The results demonstrate that the model can process real-world RGB-thermal inputs and produce accurate renderings, even environments different from the training domain, highlighting its potential for real-world deploy- ment. Some edge effects present when rendering beyond the spatial extent of inputs; for instance, the truck cabin is extrapolated using nearby visual information. While performance promising, iimulation-to-rest gap remains, hypothesise, due to differences in scene type, motion, and simulated thermal fidelity, representing an important avenue for future work.
5 CONCLUSIONS
We introduced RoRE, a ray-based rotary embedding that generalises effectively across diverse cam- eras and imaging geometries. Our results show that ray-space embeddings yield stronger perfor- mance, rather than extrinsic or projection parameters, under varying camera geometries, such as in the fisheye case. We also extended transformer architectures to support multi-modal inputs and constructed a synthetic RGB-thermal dataset enable this research. The resulting model achieves unified, geometrically consistent scene understanding from non-confocal images, showing encour- aging results on both simulated and real-world data.
One limitation of this formulation is that each patch is represented by single ray, meaning patch size and orientation are not encoded explicitly. Although these factors are likely inferred implicitly by the network, this does constrain the expressiveness of the representation. Incorporating these parameters directly is an interesting direction for future work. Additional future work includes extending additional input modalities (e.g.. depth polarisation) and validating multi-modas modal fusion at larger scales in woal-world conditions. This work takes a step toward flexible, plug-and- and- play vision systems that move beyond conventional camera configurations.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
This research was supported part through the NVIDIA Academic Grant Program. This research was supported in part by funding from Ford Motor Company.
ETHICS STATEMENT
The development and training of the large-scale models used in this work required substantial com- putational resources, which in turn consume significant amounts of energy. To promote transparency and awareness around sustainability, we roughly monitored the total energy consumption associated with our development and experiments. See Appendix A.3 for details.
REPRODUCIBILITY STATEMENT
Implementation details have been supplied in Appendix A. with configurations, as well as the dataset and code being publicly available.
REFERENCES
Cyprien Arnold, Philippe Jouvet, and Lama Seoud. SwinFuSR: an image fusion-inspired model for RGB- RGB-guided thermal image super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3027-3036, 2024.
Roman Bachmann, David Mizrahi, Andrei Atanov, and Amir Zamir. Multimae: Multi-modal multi- task masked autoencoders. European Conference on Computer Vision, pp. 348-367. Springer, 2022.
Qian Chen, Shihao Shu, and Xiangzhi Bai. Thermal3D-GS: Physics-induced 3d gaussians for ther- mal infrared novel-view synthesis. In European Conference on Computer Vision, pp. 253-269. Springer, 2024.
Benoit Courty, Victor Schmidt, Sasha Luccioni, Goyal-Kamal, MarionCoutarel, Boris Feld, Jérémy Lecourt, LiamConnell, Amine Saboni, Inimaz, supatomic, Mathilde Léval, Luis Blanche, Alexis Cruveiller, ouminasara, Franklin Zhao, Aditya Joshi, Alexis Bogroff, Hugues de Lavoreille, Niko Laskaris, Edoardo Abati, Douglas Blank, Ziyao Wang, Armin Catovic, Marc Alen- con, Michal Stechty, Christian Bauer, Lucas Otávio N. de Araújo, JPW, and MinervaBooks. mico2/codecarbon: v2.4.1, May 2024. URL 11171501.
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume (long and short papers), pp. 4171-4186, 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. International Conference on Learning Representations, 2021.
Scott Ettinger, Shuyang Cheng, Benjamin Caine, Chenxi Liu, Hang Zhao, Sabeek Pradhan, Yuning Chai, Ben Sapp, Charles R Qi, Yin Zhou, et al. Large scale interactive motion forecasting for autonomous driving: The waymo open motion dataset. In Proceedings the IEEE/CVF Interna- tional Conference on Computer Vision, pp. 9710-97101 2021.
Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The kitti dataset. International Journal of Robotics Research (IJRR), 2013.
Ulas Gunes, Matias Turkulainen, Xuqian Ren, Amo Solin, Juho Kannala, and Esa Rahtu. FIORD: A fisheye ondoor-outdoor dataset with LIDAR ground truth for 3d scene reconstruction and bench- marking. Scientific Computing and Imaging Applications (SCIA), 2025.
11

---

Published as a conference paper at ICLR 2026
Mariam Hassan, Florent Forest, Olga Fink, and Malcolm Mielle. ThermoNeRF: Joint RGB and thermal novel view synthesis for building facades using multimodal neural radiance fields. arXiv preprint arXhv:2403.12154 2024.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked au- toencoders are scalable vision learners. In Vision and Pultern Recognition, pp. 16000-16009, 2022.
P Proceedings of the IEEE/CVE Conference on Computer
Byeongho Heo, Song Park, Dongyoon Han, and Sangdoo Yun. Rotary position embedding for vision transformer. In European Conference on Computer Vision, pp. 289-305. Springer, 2024.
Dong Huo, Jian Wang, Yiming Qian, and Yee-Hong Yang. Glass segmentation with image pairs. IEEE Transactions on Image Processing, 32:1911-1926, 2023.
Haian Jin, Hanwen Jiang. Hao Tan, Kai Zhang, Sai Bi, Tianyuan Zhang, Fujun Luan, Noah Snavely, and Zexiang Xu. LVSM: A large view synthesis model with minimal 3d inductive bias. In The Thirteenth International Conference on Learning Representations, 2025.
Nikhil Keetha, Norman Müller, Johannes Schönberger, Lorenzo Porzi, Yuchen Zhang, Tobias Fis- cher, Arno Knapitsch, Duncan Zauss, Ethan Weber, Nelson Antunes, Jonathon Luiten, Manuel Samuel Rota Bulò, Christian Richardt, Deva Ramanan, Sebastian Scherer, and Peter Kontschieder. MapAnything: Universal feed-forward metric 3D reconstruction. In arXiv:2509. 134 2025.
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3D gaussian splat- ting for real-time radiance field rendering. ACM Trans. Graph., 12(4):139-1 2023.
Xin Kong, Shikun Liu, Xiaoyang Lyu, Marwan Taher, Xiaojuan Qi, and Andrew J Davison. Escher- net: A generative model for scalable view synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9503-9513, 2024.
Vincent Leroy, Yohann Cabon, and Jerome Revaud. Grounding image matching in 3D with MASt3R, 2024.
Ruilong Li, Brent Yi, Junchen Liu, Hang Gao, Yi Ma, and Angjoo Kanazawa. Cameras relative positional encoding. arXiv preprint arXiv:2507. 10496, 2025.
Lu Ling, Yichen Sheng, Zhi Tu, Wentian Zhao, Cheng Xin, Kun Wan, Lantao Yu, Qianyu Guo, Zixun Yu, Yawen Lu, et al. D13dv-10k: A large-scale scene dataset for deep learning- based 3d vision. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22160-2222 206002219,22222 2024.
Zikang Liu, Longteng Guo, Yepeng Tang, Tongtian Yue, Junxian Cai, Kai Ma, Qingbin Liu, Xi Chen, and Jing Liu. Vrope: Rotary position embedding for video large language models. arXiv preprint arXiv:2502. /664, 2025.
Rongfeng Lu, Hangyu Chen, Zunjie Zhu, Yuhang Qin, Ming Lu, Le zhang, Chenggang Yan, and anke xue. TaurmaGGaussian: Thermal 3D gaussian splatting. In The Thirleenth International Conference on Learning Representations, 2025.
Zelin Meng and Takanori Fukao. RGB-thermal infrared fusion for robust depth estimation in com- plex environments. arXiv preprint arXiv:2503.0443 04821, 2025.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1)999-106, 2021.
Takeru Miyato, Bernhard Jaeger, Max Welling, and Andreas Geiger. GTA: A geometry-aware atten- tion mechanism for multi-view transformers. In The Twelfth International Conference on Learn ing Representations, 2024.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniee, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arX1v:2304.077/93, 2023.
12

---

Published as a conference paper at ICLR 2026
Mchdi SM Sajjadi, Henning Meyer, Etienne Pot, Urs Bergmann, Klaus Greff, Noha Radwan, Suhani Vora, Mario Lučić, Daniel Duckworth, Alexey Dosovitskiy, et al. Scene representation trans- former: Geometry-free novel view synthesis through set-latent scene representations. In Proceed- ings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 6229-6238, 2022.
Yoli Shavit, Ron Ferens, and Yosi Keller. Learning multi scene absolute pose regression with trans- formers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp- 7733, 2021.
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representa- tions. arXiv preprint arXiv:1803 02555 2018.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: En- hanced transformer with rotary position embedding. Neurocomputing, 568: 127063, 2024.
Mohammed Suhail, Carlos Esteves, Leonid Sigal, and Ameesh Makadia. Light field neural render- ing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8269-82799 8279,,7799888 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- fion processing systems, 30, 2017.
Jianyuan Wang, Minghao Chen, Nikita Karaev, Andrea Vedaldi, Christian Rupprecht, and David Novotny. Vggt: Visual geometry grounded transformer. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 5294-5306, 2025.
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing lision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409.1211 2024a.
Shuzhe Wang. Vincent Leroy, Yohann Cabon, Boris Chidlovskii, and Jerome Revaud. Dust3r: Ge- ometric 3d vision made easy. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 20697-20709, 2024b.
Philippe Weinzaepfel, Thomas Lucas, Vincent Leroy, Yohann Cabon, Vaibhav Arora, Romain Brégier, Gabriela Csurka, Leonid Antsfeld, Boris Chidlovskii, and Jérôme Revaud. Croco v2: Improved cross-view completion pre-training for stereo matching and optical flow. In Proceed- ings of the IEEE/CVF International Conference on Computer Vision, pp. 17989- 2023.
Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng. and Hengshuang Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10371-10381, 2024a.
Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything v2. Advances Neural Information Processing Systems, 37:21875-21911 2024b.
Tinghui Zhou, Matthew Brown, Noah Snavely, and David G Lowe. Unsupervised learning of depth and ego-motion from video. In Proceedings of the IEEE conference on computer vision and pultern recognition, pp. 1851-1858, 2017.
Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnification: Learning view synthesis using multiplane images. ACM Transactions Graph. Proceedings Special Interest Group on Computer Graphics and Interactive Techniques, 37, 2018.
13

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 IMPLEMENTATION DETAILS
A.1.1 RGB EXPERIMENTAL DETAILS
Model Architecture. The model parameters are outlined in Tab. The total number of parameters for the model is 24.49M.
Table 6: Multi-modal configuration values.
Section
Parameter
Value/Setting
Model
Layers
Emdedding Dimension Heads
768
16
Positional Embedding
BORELYSMICTA/PROPP
Ray Parameterisation
Plicker
Head Type
Patch Embeding
Embedding Type
Linear
Patch Size
8x8
Dataset
Target Views
2
Optimizer
Leurning Rate
4.00le4
Warm-up Stops
2500
Training Steps
Data Loader
Batch Size (per GPU)
Image Size
256x256
Loss
MSE Weight
1.0
Perceptual Loss Weight
0.5
Training. Training was performed on 2 RTX6000 Ada GPUs, Training time for a single run was approximately 30 hours.
A.1.2 MULTI-MODAL EXPERIMENTAL DETAILS
Model Architecture. Model parameters are provided in Tab. 7 and a graphical illustration of net- work architectures is depicted in
Table 7: Multi-modal configuration values.
Section
Parameter
Valne/Setting
Model
Large
Decoder Backbone
VIT Rase
Positional Embodding
RoRE
Ray Parameterisatin
raymap
Head Type
DPT
Patch Emboding
Embedding Type Patch
ceav
Dotaset
Context Views
Target Views
Optimizer
Learning
Warm-up Steps
500
Data Loadle
Batch Size (train) Image Size
256x256
Loss
MSE Weight (Aune)
1.0
Weight (Apipa))
Dopth Loss Weight (Adepth)
0.75
Masking
Masking Ratio
50%
Training. We pre-train network RealEstate10K, precondition the network, before train- ing on multi-modal data. This done because RealEstate IOK a larger dataset and should help generalise. Training was performed 2 RTX6000 Ada GPUs, with the initial RealEstate 10K
14

---

Published as a conference paper at ICLR 2026
Encoder
Encoder Layer
Decuder Layer
Imput Images
Embedding
Trandormer
Some Encoding
Topea
Nem
Nore
E
Salf Minelles
Liver Lawer
bear
Naatt
Altestiee
Lsee Lover
Direction
Imput Exrriastes Input Intrimics
Nerta
Ray Info
tiken
Decoder
Render Renlering
Rendered
Eacoding
Heads
Images
RGB
Thermal
Usee
Ray
Depth
Dicections
Oucput
Extrinsios
Ray
Intrinsies
Tikmn
Figure 6: Multi-modal, multi-camera architecture. The architecture has two stages, encoder where images are patchified and encoded into multi-modal scene encoding. This encoding can then used by the decoder render novel views. This architecture makes use of our ray based RoPE to enable the embedding of camera intrinsics and extrinsics. This embedding is used in both the Encoder and Decoder layers. Otherwise these encoder and decoder layers are fairly standard self-attention and cross-attention layers respectively.
pretraining taking around 4 days to complete and the final multi-modal model being trained for days.
Loss Functions. To train the network, employ a combination of photometric and geometric supervision. Specifically, the total loss function consists of two appearance- losses and one depth-based loss:
L Amse + Alpips + Adepth Edepth-
(12)
where Lisee is the MSE loss between predicted and ground-trus -tunh RGB or thermal reconstructions, and Lipips is the LPIPS, which captures higher-level structural and semantic differences between the reconstructed and reference images. To enforce geometric consistency, we also include a depth loss, Edepth, defined as:
N d - d.l V4 Vd.,,
Laepth
(13)
where d, and d, represent the ground-trus and predicted depths at pixel i, and denotes the spatial gradient operator. The loss combines both absolute depth error and gradient-based smoothness, formulation commonly used in monocular depth estimation encourage accurate relative depth. This loss works well for bounded scene c.g. indoors. If the method was being applied in scenes with large depth values e.g. outside, we would employ losses resilient to the reduced accuracy associated with larger depths.
The scalar weights Amses Alpips and Adepth balance the contributions of the respective terms, values for these given in Tab. 7. This multi-term loss encourages the model produce reconstructions that are both photometrically accurate and geometrically consistent, which critical for effective multi-view, multi-modal synthesis.
15

---

Published as a conference paper at ICLR 2026
Input Images
LVSM
GTA
PRoPE
RoRE
Ground Truth
DL3DD RE10K
Figure 7: Varying intrinsics in scene. When the camera intrinsics varies within scene without any additional training, we both GTA and PRoPE fail to interpret the new cameras. The authors of PRoPE show that with training PRoPE is capable, however RoRE natively understands this.
Input Images
RoRE
Ground Truth
Figure 8: Fisheye failure case. Example failure case of RoRE of fisheye images, the local patches that RoRE produces can become misorientated. (red) shows an inset RoRE's output, (blue) shows same inset from ground truth.
A.2 ADDITIONAL RESULTS
A.2.1 RORE RESULTS
Fig. show qualitative results on the RealEstatelOKK and DL3DV datasets with varying intrinsics. Intrinsics were synthetically adjusted using cropping to increase the focal length. It can be observed that GTA and PRoPE fail to interpret the varying intrinsics values. LVSM performs better but looking at the REIOK example the rendering is partially warped. RoRE produces the best results.
A.2.2 FISHEYF RESULTS
Fig. 8 shows failure case of RoRF on fisheye imagery. This shows that while RoRE is able to render the global geometry quite well the local patch geometry is poorly reconstructed. One possible reason for this error is RoREs lack of ability to explicitly model patches size and orientation. As these properties are implicitly estimated during training, when taken outside of that training domain, these estimations are likely less accurate.
A.2.3 NETWORK SCALING
To confirm that our RoRE embedding is able to scale with larger networks, doubled the network size from to 12 layers and observed a substantial boost in performance, see Tab. 8 for numerical results. This level of increase in performance is typical for transformers (Jin et al., 2025), suggesting that our RoRE formulation doesn't negatively effect scaling performance.
A.2.4 MULTI-MODAL RESULTS
Masked Input. The use masked input tokens during training allows the model to handle partial occlusions and missing data inference. By learning to reconstruct scenes from incomplete inputs, the network becomes robust to real- weal-world scenarios where sensors may be partially obscured by dirt, water, glare.
16

---

Published as a conference paper at ICLR 2026
Table 8: Scaling network size. The RoRE embedding is able to scale larger networks. We doubled the size of the network and observed a characteristic performance gain from transformer scaling.
# Layers
# Parameters
PSNR(1)
SSIM(t)
LPIPS(
6
24M 48M
26.65
0.845
0.070
12
28.79
0.88
0.048
Table 9: Progressive masking of input images.
RGB
Thermal
Depth
Masked
PSNR(1)
SSIM(1)
LPIPS(4)
PSNR(1)
SSIM(t)
LPIPS(L)
AbsRel(U)
RMSE(L)
6,(1)
21.494
0.634
0.254
20.481
0.807
0.172
0.060
0.025
0.964
10%
20.875
0.618
0.269
20.021
0.799
0.181
0.063
0.025
0.962
30%
19.612
0.576
0.303
18.933
0.779
0,205
0.067
0.027
0.957
50%
18.287
0,525
0,350
17.755
0.752
0,239
0.076
0.031
0.945
70%
16.913
0.477
0.409
16.367
0.722
0.286
0.105
0.042
0.884
90%
14.902
0.448
0.496
14.183
0.693
0.366
0.207
0.082
0.537
We evaluate this robustness by progressively increasing the fraction of masked input tokens infer- ence, with quantitative results in Tab. 9 and qualitative examples in Fig. 9. Performance gradually declines as masking increases, but remains surprisingly stable up to moderate levels.
At 10-50% masking, reconstructions and depth predictions remain high-quality; even with 50% of input patches masked, key scene elements, such as the black box on the bottom shelf, are accurately recovered. A more noticeable drop occurs 70% masking, where finer details lost, yet the network still predicts plausible global structure and broadly consistent depth maps.
Multi-Camert Reconstruction. To evaluate the scalability and generalisability of our multi-modal transformer, we assess its performance with varying numbers and combinations of input images. Unlike previous models that assume fixed number and modality, framework can accept any number of views, mixing RGB and thermal inputs, constrained only by memory and compute at inference.
We perform two experiments, summarised Tab. 10. In the first, we vary the number of inputs, using equal RGB and thermal splits (1 RGB thermal up to 4 RGB thermal). In the second, we fix six inputs and vary the RGB-to-thermal ratio to explore modality dominance effects.
Results show that increasing input number improves photometric and depth predictions, yielding more complete reconstructions with fewer artefacts. When varying modality composition, more thermal inputs enhance thermal fidelity but slightly reduce RGB quality, and vice versa. Depth estimates remain largely stable. performing slightly better when RGB images dominate, likely due their higher spatial resolution and texture cues.
Table 10: Varying number of input images and modalities. Our proposed approach is able to handle range of input images across varying ratio of modalities.
Themnol
Depth
Fneade
Inpur Images
PSNR(1)
SSIM(T)
LPIPS())
PSNR(1)
SSIM(t)
[PIPS(])
AbsRol[.)
RMSB(1)
5:(1)
TBK
ms
(I-RGB (1-themmal)
19.738
0.588
0.301
0.084
0.033
0.929
35.413
35.513
18.610
0.783
02155
0.100
0.066
01509
34.312
34,9112
(1-RCIR, I-hermal)
30-470
0.605
0.780
20.068
01800
0.181
0.062
0.06
0.958
37.493
40.662
(2 RGB, thermal)
21.751
0,638
0.246
21.008
0.816
0.161
0.053
0.022
0.966
47.489
44.182
S (4-RGB, 4-thermal)
22.767
0.56
0.225
22 22.104
01.33
0.140
0.044
0.19
11799
95.566
$4,289
6(I-RGB,
Sahermal)
30J22
0.615
0.269
22 02.082
01.33
0.143
0.051
0,022
1191
71.211
40.486
6/2-RGB, 4-termmal)
22.013
0.642
0.241
21 83.
0.829
0.145
0.050
0.021
0.922
71.451
49.569
RGB. 3-
0.649
21.580
49.441
(4-RGI
24hermal)
0.656
0.156
40.401
RGB, thermal)
17

---

Published as a conference paper at ICLR 2026
Masked %
Input Images
Rendered Images
0%
10%
30%
50%
70%
000
Ground Truth
Figure 9: Progressive masking of inputs. Progressive masking of multi-modas modal inputs, the results show that the method is robust to the reduction information.
A.3 ENERGY USAGE
The development and training of large-scale machine learning models requires substantial computa- tional resources, which in turn consume significant amounts of energy. In the interest of transparency and promoting sustainability in Al research, tracked the total energy consumption associated with the experiments and model development presented in this work. Numerical values are provided in Tab. 11
Table 11: Energy Consumption. Energy consumption during model development and estimated equivalent emissions
Metric
Estimate
Total Energy Used
2554 kWh
COz-equivalent emissions
1600 kg
Equivalent vehicle distance driven
13,350 km
Equivalent household energy usage
230 days
Energy usage was monitored using the open-source CodeCarbon Python package Courty et al. (2024), which estimates energy consumption based hardware usage and local electricity grid
18

---

Published as a conference paper at ICLR 2026
intensity. Over the course of this work, the total energy consumed was approximately 2,900 kilowatt- hours (kWh), which is roughly equivalent to 230 days of typical household electricity usage.
By reporting these figures, aim to provide context around the computational and energy costs of large-scale model development. We encourage the broader research community to adopt similar practices for monitoring and reporting energy consumption, supporting efforts toward more envi- ronmentally responsible and efficient Al research.
A.4 GENERATIVE AI USAGE
ChatGPT was used as proofreading tool to aid grammer and sentence structuring. Where text was modified by generative AI, the content was reviewed for possible errors, inaccuracies, and bias. All ideas and content presented in this paper were conceptualised without the aid of AI.
19


Reviews:
[{"review_id": "okqdBNeWqx", "reviewer": "Reviewer_jq2U", "summary": "This paper introduces a Rotary Ray Embedding approach that encodes image patches as rays to enhance multi-modal scene understanding. The proposed method demonstrates state-of-the-art performance and exhibits superior robustness under varying focal lengths and distorted input images", "questions": "1. Although the proposed method achieves superior performance under varying focal lengths and distorted input settings, its performance on general novel view synthesis tasks is inferior to PRoPE. It is recommended to analyze the reasons behind this discrepancy.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well written and clearly organized.\n- The proposed method achieves state-of-the-art performance and demonstrates superior robustness across challenging conditions.", "weaknesses": "1. Although the proposed method achieves state-of-the-art performance overall, it remains inferior to the concurrent work PRoPE in novel view synthesis results.\n2. According to Table 4, all variants exhibit similar performance. In particular, the second variant and the full method achieve identical results across all three metrics, suggesting that the proposed learned frequencies may not contribute significantly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KRGTOzslfa", "reviewer": "Reviewer_VCbo", "summary": "This paper proposes a Rotary Ray Embedding (RoRE) scheme that encodes camera parameters and images directly as ray-based scene tokens. RoRE feeds rays (camera origin + direction derived from intrinsics/extrinsics) into a transformer and applies a ray-conditioned variant of Rotary Position Embedding (RoPE) so that the model can handle perspective, fisheye, and RGB–thermal inputs with the same architecture.", "questions": "* In L141–L142, is $\\mathbf{R}_m^d$ a typo? Should it be $\\mathbf{R}_m^n$?\n* Any intuition on why initializing $\\theta$ with a uniform distribution (L214) leads to an exponentially decaying final frequency in Figure 1?\n* In Figure 6, should the RoPE module be replaced with the RoRE module? Can the experimental setup in Table 6—comparing different positional-embedding schemes—be understood as a replacement of this RoPE (or RoRE) module?\n* A depth loss is added in L732–L750. How is ground-truth depth at pixel $i$ obtained? Is it required during training? What happens if $\\lambda_{\\text{depth}} = 0$?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The motivation and the solution—encoding everything into the ray—are clear to me. Most concurrent or prior works either stay with token-level ray maps but only for pinhole cameras, use relative camera encodings like PRoPE but are not multimodal, or handle thermal/RGB but not arbitrary cameras. This paper attempts to do all three at once to justify generalizability via ray-level embedding. Tables 2 and 3 support this by providing tasks under different camera intrinsics and different types of inputs (distorted or fisheye cameras). Table 9 indicates that multi-view rendering across modalities is possible.", "weaknesses": "“Improved generalization and cross-modal consistency” is not fully reflected across all experiments. I would consider raising the score if the authors can make some experimental designs more convincing.\n\n* In Table 4, the PSNR result basically says: “if one keeps APE and adds our RPE, the numbers don’t really move.” That slightly weakens the claim that the ray component alone is doing the work. Are there more contrastive results on challenging tasks (e.g., a fisheye dataset)?\n* Table 5 and Figure 5 are not convincing to me regarding why multi-modal training succeeds. The authors admit there are no known prior works (L364) for this particular problem. However, it might be possible to compare with a fusion-based transformer to assess whether transfer from a single modality to a different modality is feasible. Currently, the result has a PSNR around 22, which is not particularly promising.\n* Again in Table 9, I wonder if “multiple images” implies rendering multiple views in one pass. If so, what is the difference between rendering 8 images once and rendering 1 image eight times, apart from time? Would rendering multiple RGB views confer any advantage if the backbone is drawn from RoRE/LVSM/GTA/PRoPE?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4MojvmcLXO", "reviewer": "Reviewer_JXd5", "summary": "The paper proposes a novel positional encoding method for generalizable multiview vision transformers. The proposed Rotary Ray Embedding (RoRE) combines the benefits of relative positional encodings with ray-based position encoding methods, both of which have shown promising results in vision tasks and multi-view 3D scene perception. RoRE demonstrates intriguing robustness properties against changes in camera intrinsics and across heterogeneous sensor modalities. The experimental results validate the robustness and effectiveness of the proposed positional encoding method across various tasks.", "questions": "- How is P = [t, d] computed for a patch? Is it the ray through the patch center? Average of all pixel rays?\n- Why does standard RoPE produce biased attention that decays as position values diverge? How exactly does P = [t, -t+bias, d, -d+bias] \"equalize the distance between two poses\"?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The method demonstrates robust performance on multiple tasks including novel view synthesis under varying focal lengths, barrel & fish-eye distortion, and multi-modal RGB-thermal fusion. \n- The evaluation spans five datasets covering perspective imagery , fisheye cameras, and multi-modal setups, demonstrating breadth. The zero-shot generalization experiments (varying intrinsics, distorted inputs) are particularly valuable.", "weaknesses": "- **ablation study with marginal improvements**: Table 4 shows very small differences between ablation conditions (e.g., PSNR varies only 0.08-0.09 dB, SSIM differs by ~0.002-0.003). These marginal improvements raise questions about statistical significance of the reported differences and whether the proposed components genuinely contribute to performance.\n- **Lack of interpretable analysis for learned frequencies**: Figure 1 shows position dimensions have larger rotation magnitudes than direction dimensions, but what does this mean conceptually? How does this relate to the semantic differences between position and direction?\n- **Unclear practical motivation for RGB-thermal fusion**: While technically interesting, the paper doesn't establish compelling real-world applications for the joint RGB-thermal novel view synthesis. It would be beneficial to include real-world scenarios where joint RGB-thermal fusion would be valuable to help readers understand the practical significance of the method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YhAo3icUWj", "reviewer": "Reviewer_Q8Ad", "summary": "This paper presents RoRE, a novel positional encoding technique for vision transformers. Built upon RoPE (rotatory encoding) that uses 2D patch index encoding, RoRE enhances each image patch using a Plucker ray with learned rotary positional encoding. The authors claim that doing so adds prior knowledge about the camera type into the model (regular cameras vs fisheye). Two innovations in RoRE are (1) learning the rotation frequencies of the positional embedding rather than fixed frequencies, and (2) applying asymmetric rotation to break symmetric biases in attention. The authors evaluated their model on five datasets, and RoRE shows to handle different camera/modalities with strong generalizability.", "questions": "-\tLine 161, recommend revising to: “, which has 3 position (or moment in the case of Plucker coordinates) dimensions t, and 3 direction dimensions d.” This improves readability.\n-\tIn the qualitative results, what does each of the two input images represent? Recommend to explain somewhere.\n-\tLine 303, 381 has a grammatical mistake. “is can be see…”\n-\tIs the masking ratio a fixed value or does it follow a schedule during training?\n-\tIn Figure 4, what are P1 and P2? Different modalities? Furthermore, what is being evaluated in multi-modal understanding? The predicted images seem to be from the same pose as the input, so are you testing image reconstruction?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "-\tThe method itself appears simple and thus should be easy to reproduce.\n-\tLearned 6-dimension positional encoding empirically aligns with the intuition that the frequencies should decay and shows more uniform patch-wise attention.\n-\tExperiments show performance gain compared to previous works (excepting concurrent work PRoRE).\n-\tThe ablation studies cover most new components in the model.\n-\tThe paper is well written and easy to follow. The differences from related works are explained clearly.", "weaknesses": "-\tThe idea of using Plucker rays as positional encoding has been introduced at least in prior work LVSM, which assigns pixel-wise Plucker ray embedding. (However, LVSM does not use positional encoding but simply passes the rays through a linear layer.)\n-\tA position value P_p is defined for each image patch. However, the pixels within an image patch could have different position values (rays). If just one value is used for an entire patch, then does the size of each patch matter?\n-\tIn Table 4, the changes in metrics are fairly small. Recommend that authors further analyze why. The caption saying it is a boost is largely overstating.\n-\tIn Table 5, the PSNR from thermal needs to be separated from the metrics for RGB.\n-\tThe multi-modal scene understanding is a little bit distant from the main topic. It is hard to see what the authors want to show. Does the author want to demonstrate RoRE is a good sensor fusion tool? If so, this part needs much more detail and analysis. There isn’t any comparison to other works either.\n-\tThe failure cases in Line 420 should still be shown in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)