RayI2P: Learning Rays for Image-to-Point Cloud Registration
Authors: Xinjun Li
Wenfei Yang
Zhixin Cheng
Jiacheng Deng
Fei Wang
Chen Qian
Tianzhu Zhang
Venue: ICLR 2026 Poster (2026)
Keywords: Image-to-Point Cloud Registration
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Image-to-point cloud registration aims to estimate the 6-DoF camera pose of a query image relative to a 3D point cloud map. Existing methods fall into two categories: matching-free methods regress pose directly using geometric priors, but lack fine-grained supervision and struggle with precise alignment; matching-based methods construct dense 2D-3D correspondences for PnP-based pose estimation, but are fundamentally limited by projection ambiguity (where multiple geometrically distinct 3D points project to the same image patch, leading to ambiguous feature representations) and scale inconsistency (where fixed-size image patches correspond to 3D regions of varying physical size, causing misaligned receptive fields across modalities). To address these issues, we propose a novel ray-based registration framework that   first predicts patch-wise 3D ray bundles connecting image patches to the 3D scene and then estimates camera pose via a differentiable ray-guided regression module, bypassing the need for explicit 2D-3D correspondences. This formulation naturally resolves projection ambiguity, provides scale-consistent geometry encoding, and enables fine-grained supervision for accurate pose estimation. Experiments on KITTI and nuScenes show that our approach achieves state-of-the-art registration accuracy, outperforming existing methods.

Published as a conference paper at ICLR 2026
RAYI2P: LEARNING RAYS FOR IMAGE-TO-POINE CLOUD REGISTRATION
Xinjun Li', Wenfei Yang' Zhixin Cheng',Jaachere Deng', Fei Wang', Chen Qian', Tianzhu Zhang
University of Science and Technology of China
National Key Laboratory of Deep Space Exploration, Deep Space Exploration Laboratory Time Research
{1xj3017, chengzhixin, dengie, wangfei91|0mail wst.lil..stteeee.. eduttceedddcoomm {yangwf, eustae,
Blanchen@sensetiine com
ABSTRACT
Image-to-point cloud registration aims to estimate the 6-DoF camera pose of a query image relative to a 3D point cloud map. Existing methods fall into two cat- egories: matching-free free methods regress pose directly using geometric priors, but lack fine-graines supervision and struggle with precise aligmment; matching based methods construct dense 2D-3D correspondences for PnP-based pose estimation, but are fundamentally limited by projection ambiguity (where multiple geomet- rically distinct 3D points project the same image patch, leading to ambiguous feature representations) and scale inconsistency (where fixed-size image patches correspond to 3D regions of varying physical size, causing misaligned receptive fields across modalities). To address these issues, we propose novel ray-based registration framework that first predicts patch-wise 3D ray bundles connecting image patches the 3D scene and then estimates camera pose via a differentiable ray-guided regression module, bypassing the need for explicit 2D-3D correspon- dences. This formulation naturally resolves projection ambiguity. provides scale- consistent geometry encoding, and enables fine-grained supervision for accurate pose estimation. Experiments on KITTI and nuScenes show that our approach achieves tite-of-the-rrr registration accuracy, outperforming existing methods.
INTRODUCTION
Image-to-point cloud registration refers to the process of estimating the 6-degree-of-freedee (6- DoF) camera pose of given 2D image relative to pre-constructed 3D point cloud map. This task fundamental wide range of computer vision applications, including 3D reconstruction (Dong et al. (020), AR/VR (Billinghurst et al.) 2015). SLAM (Durrant- & Bailey) (006). and visual localization (Sarlin et al. 2023). The central challenge lies in the heterogeneous nature of the input modalities: 2D images encode appearance information in dense, regular grids, whereas 3D point clouds represent spatial geometry as sparse, unordered points. This modality gap makes it inherently difficult to design shared feature representations and establish reliable 2D-3D correspondences.
To address this challenge, existing methods can be broadly categorized into matching-fre free (Li & Lee 2021 Jeon & Seo 2022) and matching- based (Ren al. 2022 Zhou et al. 2023 Wang al., 2024a Kang et 20044 Bie et al. 2025) Li et al.) 2023) approaches. Matching free methods lever- age geometric priors to directly predict camera pose, circumventing the need for explicit correspon- dence construction. cloud registration as a combination of frustum classification and pose optimization. first predicts whether each 3D point lies inside the camera's 's frustum, and then iteratively adjusts the camera pose to ensure that all predicted in-frustum points are projected within the image plane. However, as illustrated in Figure [ [), this frustum- based optimization only provides coarse supervision, and the resulting poses are often inaccurate due to the lack of fine- -neigre alignment. contrast, matching- based methods have achieved superior performance by establishing dense 2D-3D correspondences. followed by pose estimation using geometric solvers such as PAN-RANSAC (Lepetit et al. 2009)
representative method, Deep12P (Li & Lee (021). formulates image-to-point
"Coreepooniigg author

---

Published as a conference paper at ICLR 2026
Iterative Pose Solver
Overlap Detector
Pose
Predictor
Solver
Figure 1: (a) Illustraiion of the iterative pose optimization process in matching- methods such as DeepI2P. Green points indicate 3D points predicted to lie within the camera frustum. The frus- tum color transitions from black to red, showing the progression of estimated camera pose during iterative optimization. The green frustum represents ground-trute camera pose. Due to the coarse supervision from frustum classification, the final estimated pose often deviates significantly from the ground truth. (b) Two key challenges of existing matching-based approaches: (1) projection- induced correspondence ambiguity: multiple geometrically distinct 3D points project to the same image region; (2) depth-induced scale inconsistency: fixed-size image patches correspond to 3D re- gions of varying physical size. Each line denotes ground-trutt 2D-3D correspondence between an image patch and 3D region.
(c) Our proposed ray-based registration pipeline predicts a set of 3D rays (colored lines), each representing the projection of an image patch into 3D space. The predicted rays are then used to estimate the camera pose (red frustum), which closely aligns with the ground truth (green frustum). This formulation naturally mitigates the limitations of previous methods.
Fischler & Bolles 1981). These methods typically extract modality features and align them through attention mechanisms /Vaswani et al., 2017 and contrastive learning (Chopra et al. 2005 Schroff et al. 2015 Sun al. 2020). Some recent works (Zhou al. 2023 Wang al. 2024a)
further introduce virtual points or voxels as intermediate representations to bridge the modality gap.
Despite these efforts. vatching-based approaches still face two critical challenges: (1) Projection- induced correspondence ambiguity: as shown at the top of Figure [b), due to perspective projec- tion, single image pixel or patch may correspond to multiple spatially disjoint 3D points distributed along viewing ray. These 3D points can differ significantly in geometry (e.g., curvature, surface normals, or semanties), yet they are all mapped the same image region. As result, models are forced to align geometrically dissimilar 3D features to single image feature, leading to ambiguous or collapsed feature representations. This undermines the learning of discriminative similarity met- rics and results in unreliable 2D-3D correspondences, ultimately harming pose estimation accuracy. (2) Depth-induced scale inconsistency: as shown at the bottom of Figure [Tb), fixed-size image patches correspond to 3D regions of vastly different physical scales depending on their depth. For instance, a small object nearby and a large object in the distance may occupy similar image areas but represent vastly different 3D scales. This causes a mismatch in receptive field alignment: image features are extracted from fixed local neighborhoods, while corresponding 3D features vary significantly in spatial extent depending on depth. Such scale inconsistency makes it difficult to learn consiseent similarity metrics and establish semantically meaningful matches. Together, these issues limit the reliability and generalization capability of learned correspondences, especially in complex outdoor environments.
In this work, we take fundamentally different perspective: instead of modeling 2D-3D corre- spondences directly, we model rays that implicitly connect image patches to the 3D scene. We observe that under the pinhole camera model, each image patch naturally corresponds to a potential ray in 3D space, originating from the camera center and extending into the 3D scene. By learning to predict these rays, we can effectively bridge the modality gap and enable accurate pose estimation without relying explicit 2D-3D matching. To realize this idea, we propose novel ray-based framework for image-to to- co-point cloud registration as shown in Figure Tc). Specifically, we design a two-stage ray-based cross-modal registration pipeline: ray prediction module that integrates image
2

---

Published as a conference paper at ICLR 2026
and point cloud features via attention mechanisms (Vaswani al. 2017) to infer consistent 3D rays for each image patch, and a differentiable ray-guided pose regression module that jointly estimates camera's rotation and translation from predicted ray bundles. Our ray-based representation provides several key advantages. First, it resolves projection-induced correspondence ambiguity by replacing discrete pixel-to-point matching with continuous ray formulation, where each image patch is in- terpreted as a 3D ray that encodes both its viewing direction and origin in the point cloud. Second, it alleviates depth-induced scale inconsistency by modeling scene geometry in terms of directional cues rather than fixed spatial extents. Since ray directions are invariant to the depth of the underly- ing structure, our approach maintains consistent representations regardless of whether a patch corre- sponds to a nearby object or a distant surface. Third, associating each image patch with ray bundle provides fine-grained geometric supervision. Rather than merely predicting whether a 3D point is visible, our model learns how each image patch projects into space, yielding stronger and more spa- tially aware signals during training. This finer-graines modeling enhances pose accuracy, especially in scenes with complex geometry partial observations. Despite the overall simplicity, extensive experiments on KITTI and nuScenes show that our framework achieves state-of-the-an registration performance, consistently outperforming existing matching-free frree and matching- based methods.
The main contributions are summarized as follows: (1) We propose a novel ray-based paradigm for image-to-point cloud registration, which effectively addresses the core limitations of prior ap- proaches by modeling image patches as continuous 3D ray bundles, thereby resolving projection- induced correspondence ambiguity and depth-induced scale inconsistency, and enabling fine- grained, direction-aware pose supervision beyond coarse geometric constraints. (2) Extensive exper- iments on KITTI and nuScenes demonstrate that our method achieves state-of-the-arr performance in cross modal registration accuracy, validating the effectiveness of our ray-based representation.
2 RELATED WORKS
In this section, we provide a brief overview of related works on image-to-image registration, point cloud-to-point cloud registration, and image-to-point cloud registration.
Image-to-Image Registration. This task aims to align images of the same scene taken under vary- ing conditions (e.g., time, viewpoint) by estimating a spatial transformation. Existing methods fall into two main categories: detection- baaed and detection -free, Detection-
methods extract and match sparse keypoints, either hand-crafted (Lowe 20044] Rublee et al. [201]) or learned (Gao al.] 2023 De'Tone et al. 2018 Sarlin et al., 2020 Dusmanu et al. 2019 Revaud et al. 2019). but often struggle in low-texture regions. Detection- methods (Shen et al. 2023 Zhou et al. 2021 Sun et al. 2021 Wang et al.] 2024b) bypass keypoint detection by estimating coarse-to-fine patch- and pixel- correspondences, enabling robust alignment.
Point Cloud-to-Point Cloud Registration. The goal here is to estimate rigid transformation between two 3D point clouds. Traditional approaches rely on matching sparse keypoints (Drost et al. 2010: Rusu et al.) 2009; Bai et al., 2020 Deng et al. 2018), but are sensitive to density vari- ations, occlusions, and noise. Detector- free methods (Yu et al. 2021: Lu et al.) 2023] Qin et al.] 2023] instead learn dense semi-dense correspondences directly from global geometric context. CoFiNet YYu et al.) 2021), for example, uses a coarse-to-fine pipeline without explicit keypoint de- tection. Recent works further integrate learned pose estimators (Qin al. 2023 replace classical
RANSAC (Fischler & Bolles 1981), achieving better robustness.
Image-to-Point Cloud Registration. Compared to intra-modal registration, this task is more chal- lenging due to the modality gap. A common pipeline involves establishing 2D-3D correspondences
followed by pose estimation using PaP-RANSAC (Lepetit et al., 2009 Fischler & Bolles 1981). Early methods such as 2D3D-MatchNe (Feng et al. 2019) first use modality-speciffc keypoint de- tectors (Lowe) 2004 Zhong 2009) to get keypoints and then extract keypoint features via CNNs (Si- monyan & Zisserman 2015) or PointNet (Qi et al.) (017)) These independent hand-crafted keypoint detectors for different modalities leads to a poor registration accuracy. Recent learning-based ap- proaches propose to predict dense 2D-3D correspondences. Corr12P (Ren et al.) (022) estimates overlapping regions and matches 2D/3D features, while VP2P- (Zhou et al. 2023) adopts voxel-based representations and differentiable PnP (Chen et al. 2022), though both are still sen- sitive to modality discrepancies. CoFil2P (Kang et al.] (2024) follows the design of LoFTR (Sun) et al. 2021) but overlooks modality gaps, limiting its generalization. ICI. (Li et al. 2025) intro-
3

---

Published as a conference paper at ICLR 2026
Ray Prediction Module
Ray-guided Pase Regression Module
Extodkiing Mr,
For Veeer
Poler K.
NLP
Feer Chedt
Copporaricud Fistano W,t,
Chorlapping Aegion Dejecter
Poat wood My
Macka
Recie
yi N.W,
Path Bays
SILF
Notoeell PG
1 Concosmation Uprnation
i Elemeeshwwiee Add
Overview of our proposed image-to- point cloud registration framework. Given a pair of image I and point cloud P, we first extract downsampled patch features F, and point features F p. respectively. An overlapping region detector then predicts binary mask Mr indicating whether each 3D point lies within the camera frustum. Next, cross-modas attention fuses patch and point fea- tures into enriched patch features F, Based on Fy, a lightweight MLP predicts 3D rays for each image patch, representing potential projection in 3D space. Finally, the ray-guided pose regression module estimates camera pose (R. t) by jointly leveraging predicted patch rays r, reference rays r' (computed from camera intrinsic), and fused patch features F
Figure 2:
duces an implicit correspondence learning framework for direct pose regression, yet its reliance on weak geometric priors hampers performance under large viewpoint changes or sparse inputs. GraphI2P (Bie et al., 2025) incorporates external depth prediction (Bhat et al. 2023) to reduce do- main gaps, but incurs high computational cost. Beyond matching- -bssed methods, some approaches directly predict camera pose without explicit correspondence construction by leveraging geometric priors. Deepl2P (Li & Lee (02T), for instance, reformulates the task frustum classification followed by inverse projection. However, its coarse 2D-3D associations often fail to achieve high- precision alignment.
METHOD
3.1 OVERVIEW
Given an image I # and point cloud P E Nx3 from the same scene, goal is to determine the camera pose Tyt in coordinate system of P. which consists of a rotation matrix Rgt E (3(3) and translation vector tgt R3. In this paper, we propose a ray-based image- to-point cloud registration method composed of two main stages: ray prediction module to infer consistent 3D rays for each image patch, and a differentiable ray-guided pose regression module that jointly estimates camera's 's rotation and translation from predicted ray bundles, shown in Figure 2
3.2 BACKGROUND: RAY-BASED CAMERA REPRESENTATION
Traditional cameras are usually parameterized by extrinsic rotation R E SO(3), translation t e R and intrinsic calibration K Rax3 While compact, this low-dimensional representation dif- ficult to regress directly from complex visual and geometric features because of strong geometric constraints and nonlinearities. To address this difficulty, we instead adopt more expressive for- mulation by representing the camera bundle of rays associated with image patches, inspired by generalized camera models (Grossberg & Nayar 2001 Schoos et al., 2020) and recent ray-based representations (Zhang et al. 2024), Each image patch, centered at pixel coordinate u, corresponds to 3D ray r, E encoded in Plücker coordinates (Plücker 1828):
r; = (d., mz), where m; = P: x d:
Here, d, 3 denotes the ray's direction, P; E R any 3D point on this ray, and m; E R the associated moment vector, invariant to the choice of p;. Notably, when d, is normalized, the norm of the moment m, represents the distance of the ray from the origin.
Given camera parameters (R, t, K), we can generate rays by unprojecting image patches into 3D:
d; R. K ü m;
[ (t) di
4

---

Published as a conference paper at ICLR 2026
where ü, E R& is the homogeneous coordinate of u, The direction d, points from the camera center into the 3D scene, and the moment m, captures the ray's 's's offset from the origin.
Conversely, camera can be approximately recovered from predicted rays r; [d,, m.]. Camera center с is obtained by minimizing distances from point to all rays:
= arg min minppeee
PERS
Σ
|pp x d, - m;l1?
(3)
The rotation R is then estimated by aligning ray directions with pixel vectors:
Σ HHd x رزية
P = arg Hill
(4)
where the homography matrix P can be decomposed into K and R via RQ-decomposition. Finally, the translation vector is recovered as = R c. By bridging compact camera parameters and over- parameterized ray bundles, this two two-way conversion yields representation that combines geometric interpretability with modeling flexibility, serving as the cornerstone our framework.
3.3 RAY PREDICTION MODULE
Feature Extraction. For the input image I E R #fxxxx33 we use ResNet (He et al. 2016) to extract downsampled 2D features and flatten them to obtain patch features F, / " where H.
H, W. W. For the input point cloud P E NR3 we use KPConv (Thomas et al. 2019) to
extract downsampled point features Fp NCN.C. We also record the corresponding downsampled patch coordinates E, E R H. and point coordinates Ep E N.33
Overlapping Region Detector. To avoid ambiguous supervision from non-overlapping 3D points, we adopt the strategy from ICL eLi al. 2025), with a minor modification adding 3D coordinates input. The resulting binary mask Mp indicates visible 3D points within the camera view.
Ray Prediction via Cross-Modal Feature Fusion. In this part, our goal is to predict associated 3D rays for each image patch by fusing patch features with relevant 3D information from the point cloud. Given extracted patch features F, and point features F, Fp, we first introduce learnable positional embeddings to encode spatial cues. Specifically, we map patch coordinates E, into patch positional embedding PE, using a MLP. Likewise, point coordinates Ep are mapped into point positional embedding PEp. The resulting embeddings are added the corresponding patch and point features to produce position-awar representations:
F F, PE,, Fp= Fr PEp.
(5)
To suppress interference from irrelevant geometry, apply predicted point mask Mp to retain only visible points from the point cloud. We then apply a transformer-basee fusion module (Vaswani let al. 2017) consisting of multiple self and cross attention layers, executed in alternate fashion for iterations. In each iteration, a self attention layer allows image patches to exchange contextual information within the image, followed by a cross attention layer where each patch attends to the visible 3D points. Through L rounds of alternate interaction, the patch features are progressively refined with both global image context and geometry-aware cues from the point cloud, enabling the network to reason about each patch's 's potential ray in 3D space. Finally, the fused patch features F, are passed through a MLP head to predict patch rays E I Ht, W.x6 for every patch, representing its origin and direction in 3D space.
Focus loss. To encourage each image patch to attend more to geometrically relevant 3D points, we propose a focus loss that guides the attention distribution in cross attention layers. Specifically, we encourage each image patch to assign higher attention scores to 3D points whose projections fall within local neighborhood (a circle with radius of pixels) centered at the patch. Let e RH W.H.N.N, denotes the averaged attention map across all cross-attention layers, E,,; IR² be the center coordinate of image patch i, and P.J IR2 be the 2D projection of point j in the image plane. We define an indicator function:
1g = {1) 0, otherwise
1. if E - <
(6)
5

---

Published as a conference paper at ICLR 2026
Then the focus loss is formulated as:
H. H.W. No
foc =
H. 1
H.W.
This loss guides attention toward spatially local, geometrically meaningful regions, enhancing ray prediction accuracy and accelerating convergence.
3.4 RAY-GUIDED POSE REGRESSION MODULE
In practice, predicted rays may suffer from noise, ambiguity, or misalignment. As result, directly applying geometric solver, as described in Equation @ and may lead to unstable or inaccurate pose estimations. We provide visual evidence of this instability in the Appendix [ To address this, we propose learnable ray-guided pose regression module that estimates the camera pose from fused patch features f, predicted patch rays r, and reference rays r' fully differentiable manner. The predicted patch rays are defined in LiDAR coordinate system, whereas the reference rays r' E IR.,W..6 represent aatch-associated rays in camera coordinate system. The reference rays are computed from known camera intrinsic K and patch coordinates E,, applying Equation | with the rotation matrix serve geometric anchors to guide pose regression. Our ablation study shows that this learnable formulation outperforms geometric solvers in both accuracy and robustness.
ray-guided and
and translation vector t = These rays
Specifically, we first apply MLPs to transform fused patch features f. predicted patch rays and reference rays r', and concatenate them to obtain the ray-guided features F,. This fusion provides the network with unified representation that combines cues from fused patch features and direc- tional constraints from both predicted and reference rays, enabling reliable and spatially grounded pose estimation. To incorporate global spatial context, perform average pooling over F, and concatenate the pooled result back to F,., forming context-enriched features F.- These are further refined by MLP and pooled again to yield a compact pose representation vector Vpose. Finally, two lightweight MLP heads take Ypose as input and predict the rotation R and translation t, respectively. Here, we adopt the 6D representation of rotation (Zhou al., 2019) to parameterize rotation due to its continuity and suitability for learning in (O(3).
3.5 LOSS FUNCTION
Given ground-truth camera pose (Rgts, tgt) and intrinsic matrix K, we design a composite loss to jointly supervise the model. The overall loss consists of three terms: ray regression loss Lray. camera pose loss Leam. and a focus loss Lfo introduced Equation 7 Each term is tailored to supervise specific sub-task, collectively guiding the model towards accurate and robust registration.
Ray regression loss. To supervise ray prediction, we design a ray-level regression loss tailored to setting. Given ground-truth camera parameters, ground truth ray bundles Tgt be computed via Equation 2 We apply an L2 loss over all patches to enforce accurate ray alignment:
H.W.
Gray
Σ l|rges -1sl22-
(8)
H.W.
Camera pose loss. We define camera pose loss Leam to directly supervise predicted camera pose:
Leam
R..t
R.I/2 + ||t.g
t||2-
To jointly optimize all components, we define the total loss as the sum of the three sub-losses:
Ciotal
Lrey Leom + Looe-
(10)
EXPERIMENTS
4.1 IMPLEMENTATION DETAILS
In this work, we implement the proposed model in Pytorch (Paszke et al. .2019) and adopt a single NVIDIA RTX 3090 GPU for training. We adopt 4-stage ResNet (He et al. 2016) as the image

---

Published as a conference paper at ICLR 2026
Table 1: Registration accuracy on the KITTI and nuScenes datasets, Here represents method that
adopts external powerful depth estimation model (Bhat et al. 2023) for image depth estimation.
Category
Method
KITTI
nuScenes RRECH 14
KTHiml
RRE("N
Aoo
RTE(m)
Acc(
Cor12P (Ren al..
2022
3.78166.16
5,89120.34
72.42
6.041600766
3.73-19.03 2.15.1..33
49.00
VP2P-maie maich Zhou
2023
0.75
3.29.17.99
83.04
0.89.1.44
88.33
Matching-based
FreeReg? Wing el ai
A124a
0,95=1005
2.06133221
91.68
CoFi12P Kang et
0.31=0.20
1.24+0844
1.21
2.54#8893
e 2025
0.20=0:21
1.24++233
97.49
0.63=0.44
2.13-3.75
GraphiZPP Bie Bm 2025
0.32:0088
1.65+
99.61
0,49:: 1.22
1.73-111.63
99.48
Grid Cls. + PnP & Lec
2021
3.64=3.46
19.19+228.0
11.22
3,02=2.40
12.66+211
2.45
Matching free
De)p12P(3D) [1
Deep12P(2D)
20:4
3.54
24.73++31 69
18.66
25.95
3,77
2.78
2,88+2. 12
20.65++ 12:24
6.801
2.26
Ours
0.09=0.08
0.63=10.1
99.75
0.39
1,48+5772
96.61
Input
VP2P-matee
ICL
Ours
GT
10.63m/ 83.36'
2.47m/ 22.84
1.24m 23.67'
0.09m / 0.35
RTE(m) RRE(")
5.16m / 51.40"
3.31m/ 19.58
1.17m 26.42
0.1 0.31
RTE(m) RRE(")
9.11m / 64.44
1.25m / 28.99
1.51m/ 39.09'
0.1 / 0.67
RTE(m) / /RE(")
Figure 3: Qualitative comparison of Image-to-Poing Point Cloud registration results KITTI dataset.
backbone network, where the output channel dimension is 512. For the 3D backbone, we use 4-stage KPConv (Thomas et al. 2019) where the output channel dimension is also 512. We set the channel dimension Cj to 256 and set Cpose to 512. All the attention layers have 512 features channels with 4 attention heads and ReI.U activation. The default iteration time and focus radius are set to 2 and 32, respectively. All parameters in the proposed model are randomly initialized and trained from scratch with the Adam optimizer. We train the whole network with the total loss Ciotal for 20 epochs. The learning rate is set to 10-4, and the weight decay is set to be 10-6
4.2 DATASETS
We conduct experiments on two mostly used benchmarks: KITTI and nuScenes.
KITTI Odometry (Geiger et al. 2013). KITTI Odometry dataset comprises 22 driving sequences collected in urban scenarios, with of them providing ground-truth calibration files. Following standard protocol (Li & Lee 202T), we use sequences 0-8 for training and sequences 9-10 for testing. To simulate misalignment, we apply random 2D translations within 10 on the ground plane and arbitrary yaw-axis rotations. All input images are resized to resolution of 160 x 512, and point clouds are uniformly downsampled to 40,960 points for both training and evaluation.
nuScenes (Caesar et al., 2020). We generate image-point cloud pairs using the official nuScenes SDK, where point clouds are aggregated from nearby LiDAR frames, while images are taken from the current framee We follow the official train/test split, using 850 scenes for training and 150 scenes for testing. The same mis-registration strategy as in KITTI is applied. For consistency, we downsample the image resolution to 160 320 and retain 40,960 points per point cloud.
4.3 EVALUATION METRICS
To assess registration performance, we follow the protocol from VP2P-match (Zhou et al. 2023). reporting three key metrics: average Relative Translation Error (RTE), average Relative Rotation Error (RRE), and registration accuracy (Acc). Unlike Corr12P (Ren et al. 2022), which filters out high-error samples before computing averages, retain all test pairs during evaluation to better reflect real-world robustness. Following VP2P-match (Zhou et al.) 2023), we define Acc as the proportion of samples where the estimated transformation achieves RTE 2m and RRE 5%.
7

---

Published as a conference paper at ICLR 2026
Predicted Camera
Attention map
Figure 4: Visualization of predicted and ground truth camera poses, rays, and attention maps. Left: Predicted camera poses (red) and GT poses (green), with red lines representing predicted rays for all image patches. Right: Predicted rays (red) and GT rays (green) overlaid 3D point cloud, along with attention maps highlighting which 3D regions this image patch attends to. Our model produces accurate ray predictions and pose estimations, while attending geometrically meaningful regions, showcasing the effectiveness and interpretability of our ray-guided registration framework.
4.4 COMPARISON WITH STATE-OF-THE-ART METHODS
Baselines. Table Tlists several representative matching-fie free and matching matching-based baselines for com- parison with method.
Quantitative Comparison. Table summarizes the quantitative performance of our method against dataset, existing approaches. On the KITTI dataset, method consistently outperforms all baselines across all evaluation metrics, including GraphI2P, which benefits from an auxiliary high-quality depth es- timator. Specifically, our method achieves the highest registration accuracy (Acc), while reducing RTE by 0.1lm and RRE by 0.61 compared to ICL. On the nuScenes dataset, our approach also shows strong generalization, surpassing ICL by 0.24m RTE and 0.65 in RRE. These results high- light the effectiveness of our ray-based representation and demonstrate that a simple architecture, combined with our formulation, sufficient achieve accurate image-to-poine cloud registration.
Efficiency comparison of different methods. Table [ we provide comparison of model size and infer- ence time across different methods, with results obtained on the chine using a single RTX 3090 GPU. Methods such Dee (Li & Lee 2021) and Corr12P (Ren et al.) 2022), which include time-consuming post-
Table 2: Efficiency comparisons.
Method
Model size (MB)
Inference Time (s)
Deep12P(21) & Lee
2021
100.12
23.47
Deep12P(3D)
100.12
35.61
ma-
Cor12P Ren m 202
141.07
8.96
PV2P-miich [ hoou cl ml
2023
30.73
0.19
CoFi12P (Kang el al. 2024
0.18
ICI. [1. (ljelaa 2025
45.21
0.24
Ours
49.24
0.11
processing steps for pose estimation, show significantly slower inference speeds. In contrast, our method achieves inference speed approximately 80x (or more) faster, showcasing a clear effi- ciency advantage. VP2P-match (Zhou et al. 2023), CoFil2P (Kang et al., 2024), and ICL (Li et al., 2025) all perform computations at the original resolution in a coarse-to- creases their computational cost and inference time. In contrast, our method operates at downsam- pled resolution, significantly reducing the computational demand while maintaining similar model parameters. As result, our method achieves much faster inference time, making more efficient without compromising performance.
manner, which
Qualitative Comparison. Figure 3 presents qualitative comparisons between our method and rep- resentative baselines. To facilitate visual inspection, we project the point cloud onto image plane using estimated transformation and known camera intrinsic, with point color encoding actual depth value. Across diverse road scenes, our method achieves superior registration accuracy compared VP2P-match (Zoou et al. 2023) and ICI. (Li al. 2025), demonstrating the effectiveness of
8

---

Published as a conference paper at ICLR 2026
Table 3: Ablation studies on Ray-guided Pose Regression Module. FPF: Fused Patch Features PR: Patch Rays r; RR: Renference Rays r'; CPS: classical pose solver (Equation and 4). included, excluded.
KITTI RRE())
NuSeencs RRE(")
FPF
PR
RR
CPS
RTB(m)
Acc(
RTB(m))
Acc(
/
X
0.33.10.18
2.07-L1.67
94.48
0.48.10.32
3.23.1
89.79
X
/
X
X
0.
1.14:11.1
98.66
0.50:10.32
2.27118.88
93.75
V
/
X
X
0.10:1:009
0.73+10991
99.41
0.41
2.10:18.44
94.25
X
V
0.10:
0.82:4.0.78
99.62
0.39
2.51+:20.93
94.49
X
X
0.10=
0.74+L0.75
99.64
0.43--0.33
1.89-1-6.32
95.38
V
V X
0.09++0.88
0.63+H0.71
99.75
0.39--0.29
1.48-15.72
96.61
our ray-based formulation in enabling more reliable and structure-awarc registration. We further visualize predicted camera poses and corresponding predicted rays alongside ground-truth values. As shown in Figure 4 our method produces rays that are closely aligned with the ground truth, and estimated camera poses exhibit minimal deviation from the ground truth. To better understand the cross-modal interaction, we also visualize attention maps for individual rays. These visualiza- tions confirm that the model learns to attend to geometrically consistent regions in the point cloud, validating the interpretability and effectiveness of our inrentioo-guidern ray prediction mechanism.
4.5 ABLATION STUDY
Ablation on Ray-guided Pose Regression Module. To better understand the contribution of each component in our Ray-guided Pose Regression Module, we conduct ablation studies by selectively removing or replacing fused patch features (FPF), patch rays (PR), reference rays (RR), and clas- sical pose solver (CPS). As shown in Table 3 using only fused patch features (Row 1) performs poorly, while introducing patch rays (Row 2) brings a large improvement, highlighting the benefit of our ray-based camera representation. Combining fused patch features with patch rays (Row 3) yields further gains, showing their complementarity. While classical pose solver (Row 4) achieves reasonable results, it is less stable than learning-basedd -bsed formulation, with notably larger mean and variance rotation errors on the nuScenes dataset. Additional visual comparisons are provided in the Appendix 5 further illustrating the instability of directly applying classical pose solver. In con- trast, incorporating both patch rays and reference rays into the learnable regression framework (Row 5), the model shows improved robustness. The full model (Row 6) further achieves the best overall performance, highlighting both the necessity learnable regression module and the contribution of each component in our Ray-guided Pose Regression Module.
Ablation on Focus Loss foc- We investigate the effect of focus radius a, which governs the spatial constraints in cross-attention between patch and point features. As shown in Table 4 increasing from 8 to 32 enhances performance, indicating that a larger local neighborhood provides richer 3D context for registration. However, further increasing beyond 32, along with the removal of the focus loss, leads to performance degradation. This is likely due to the inclusion of irrelevant or noisy points, which diminish the effectiveness of focused interactions. Overall, best balance between geometric context and locality, and is therefore used in all experiments. More detailed ablation results are provided in Appendix 6
32 strikes the
Table 4: Ablation studies on focus radius o.. Here X indicates method that doesn't 't use Lfoe-
KITTI
nuScenes RRE(")
RTE(m)
RRB())
Acc(%)
RTE(m)
Acc(%)
0.10
1.02:11.00
99.18
0.42:10.31
1.94:16.22
94.44
0.119+0.09
0.75++0.95
99.30
0.41+L0.30
1.87++6.39
94.70
0.09-10.08
0.63110.7
99.75
0.39-10.29
1.48155.22
96.61
128
0.09110.09
0.9111086
99.43
0.41
1.7716.557
95.43
5
CONCLUSION
In this paper, we present a novel ray-based framework for image-to-pone toint cloud registration that overcomes key limitations of both matching-based and matching-free approaches. By modeling

---

Published as a conference paper at ICLR 2026
each image patch 3D ray, method learns a ray-guided representation that captures contin- uous and spatially consistent geometric relationships across modalities. The framework employs a two-stage pipeline: cross-modal attention to predict dense ray bundles, followed by differentiable ray-guided pose regression. This design effectively mitigates projection ambiguity and scale incon- sisteney, while offering stronger geometric cues for accurate pose estimation. Extensive experiments on KITTI and nuScenes show that our simple yet effective architecture achieves state-of-the- performance in both accuracy and robustness. These results highlight the potential of ray-based reasoning for bridging the gap between images and point clouds in cross moss- registration.
ACKNOWLEDGEMENTS
This work was partially supported by the National Natural Science Foundation of China (NO. 6230070181, (42555551, and Youth Innovation Promotion Association of CAS.
ETHICS STATEMENT
This work adheres to the ICLR Code of Ethics. Our research does not involve human subjects, sensitive personal data, or practices that may raise immediate ethical concerns. All datasets used publicly available and widely adopted in the community, and we ensure that the use of these datasets complies with their respective licenses. The proposed methodology is intended for advanc- ing research in vision and robotics, and we do not foresee direct misuse harmful applications beyond the standard risks associated with general-purpose machine learning techniques. In addi- tion, our study avoids introducing biases related to gender, race, or socioeconomic factors, and maintain transparency by documenting the experimental setup and implementation details. We are committed open research practices and will release all code, datasets, and model weights to the community to support ethical and responsible use of our contributions.
REPRODUCIBILITY STATEMENT
We have made extensive efforts to ensure the reproducibility of our work. Detailed descriptions of the model architecture, training strategies, and hyperparameters are provided in the main text and appendix. All derivations and proofs are included for theoretical results where appropriate. The datasets used in our experiments are publicly accessible, and we include evaluation protocols in the appendix. To further facilitate reproducibility, we will release the complete implementation, trained model weights, and instructions for data preparation upon acceptance. Together, these resources will allow the community to reproduce our results and extend our work with minimal effort.
REFERENCES
Xuyang Bai, Zixin Luo, Lei Zhou, Hongbo Fu, Long Quan, and Chiew-Lan Tai. D3feat: Joint learning of dense detection and description of 3d local features. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 6359-6367, 2020.
Shariq Farooq Bhat, Reiner Birkl, Diana Wofk, Peter Wonka, and Matthias Müller. Zoedepth: Zero- shot transfer by combining relative and metric depth. arXiv preprint arXiv:2302.122880 2023.
Lin Bie, Shouan Pan, Siqi Li, Yining Zhao, and Yue Gao. Graphi2p: Image-to-pore cloud registra- tion with exploring pattern of correspondence via graph learning. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 22161-22171, 2025.
Mark Billinghurst, Adrian Clark, Gun Lee, et al. A survey of augmented reality. Foundations and Trends@ in Human Human-Computte Interaction, 8(2-3):73-27) 272, 2015.
Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for au- tonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11621-11631. 2020.
10

---

Published as a conference paper at ICLR 2026
Hansheng Chen, Pichao Wang, Fan Wang, Wei Tian, Lu Xiong, and Hao Li. Epro-pnp: Generalized end-to-end probabilistic perspective- nopiitts for monocular object pose estimation. Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2781-2790, 2022.
Sumit Chopra, Raia Hadsell, and Yann LeCun. Learning similarity metric discriminatively, with application to face verification. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, volume 1, pp. 539-546, 2005.
Haowen Deng, Tolga Birdal, and Slobodan Ilie. Ppfnet: Global context aware local features for robust 3d point matching. In Proceedings of the IEEF/CVF Conference on Computer Vision and Patlern Recognition, pp. 195-205, 2018.
Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. Superpoint: Self-supervised interest point detection and description. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition Workshops, pp. 224-236, 2018
Zhen Dong, Fuxun Liang, Bisheng Yang, Yusheng Xu, Yufu Zang, Jianping Li, Yuan Wang, Wenxia Dai, Hongchao Fan, Juha Hyyppä, et al. Registration of large-scale terrestrial laser scanner point clouds: A review and benchmark. ISPRS Journal of Photogrammetry and Remote Sensing, 163: 327-342, 2020.
Bertram Drost, Markus Ulrich, Nassir Navab, and Slobodan Ilic. Model globally, match locally: Efficient and robust 3d object recognition. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 998-1005, Icee, 2010.
Hugh Durrant-Whig Whyte and Tim Bailey. Simultaneous localization and mapping: part IEEE robotics & automation magazine, 13(2):99-110, 2006.
Mihai Dusmanu, Ignacio Rocco, Tomas Pajdla, Marc Pollefeys, Josef Sivic, Akihiko Torii, and Torsten Sattler. D2-net: A trainable cnn for joint description and detection of local features. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 2019.
Mengdan Feng, Sixing Hu, Marcelo H Ang, and Gim Hee Lee. 203d-matchnet: Learning match keypoints across 2d image and 3d point cloud. In 2019 International Conference on Robotics and Automation, pp. 4790-47977 4796. IEEE, 2019.
Martin A Fischler and Robert Bolles. Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM, 24 (6):381-395 1981.
Yuan Gao, Jianfeng He, Tianzhu Zhang, Zhe Zhang, and Yongdong Zhang. Dynamic keypoint detection network for image matching. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(12):14404 144114 2023.
Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets roboties: The kitti dataset. The International Journal of Robotics Research, 32(11):1231-1237, 2013.
Michael D Grossberg and Shree K Nayar. A general imaging model and a method for finding its parameters. In Proceedings Fighth IEEE International Conference on Computer Vision. ICCV 2001, volume 2, pp. 105-115. IEEE, 2001.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016.
Yurim Jeon and Seung- Woo Seo. Efghnet: A versatile smage-to-point cloud registration network for extreme outdoor environment, IEEE Robotics and Automation Letters, 7(3):7511-7577 2022.
Shuhao Kang, Yougi Liao, Jianping Li, Fuxun Liang, Yuhao Li, Xianghong Zou. Fangning Li, Xicyuanli Chen, Zhen Dong. and Bisheng Yang. Cofii2: Coarse-to-fine correspondences-bun image point cloud registration. IEFE Robotics and Automation Letters, 2024.

---

Published as a conference paper at ICLR 2026
Vincent Lepetit, Francesc Moreno-Nogre Noguer, and Pascal Fua. Epri: An accurate O(n) solution to the pup problem. International journall of computer vision, 81:155-166, 2009.
Jiaxin Li and Gim Hee Lee. Deepi2p: Image-to-point cloud registration via deep classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 15960-15969, 2021.
Minhao Li, Zheng Qin, Zhirui Gao, Renjiao Yi, Chenyang Zhu, Yulan Guo, and Kai Xu. 2d3d-matr: 2d-3d matching transformer for setection-frree registration between images and point clouds. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 14128-14138 2023.
Xinjun Li, Wenfei Yang, Jiacheng Deng, Zhixin Cheng, Xu Zhou, and Tianzhu Zhang. Implicit correspondence learning for smage-to-poin cloud registration. In Proceedings the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp- 16922-16931, 2025.
David G Lowe. Distinctive image features from ccale-invariant keypoints. International journal of computer vision, 60:91-110, 10, 2004.
Fan Lu, Guang Chen, Yinlong Liu, Yibing Zhan, Zhijun Li, Dacheng Tao, and Changjun Jiang. Sparse-to-dense matching network for large-scale lidar point cloud registration. IEEE Transac- fions Pattern Analysis and Machine Intelligence, (5():11270-11282 2023. doi: 10.1109/ TPAMI.2023.326555
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury. Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems, 32, 2019.
Julius Plücker. geomerrrsh Entwicklungen, volume 2. GD Baedeker, 1828.
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 652-660, 2017.
Zheng Qin, Hao Yu, Changjian Wang, Yulan Guo, Yuxing Peng. Slobodan Ilie, Dewen Hu, and Kai Xu. Geotransformer: Fast and robust point cloud registration with geometric transformer. IEEE Transactions Pattern Analysis and Machine Intelligence, 45(8):9806 180-99980- 2023.
Siyu Ren, Yiming Zeng, Junhui Hou, and Xiaodong Chen. Corri2p: Deep image-to-por toint cloud registration via dense correspondence. IEEE Transactions Circuits and Systems for Video Technology, 33(3): 1208, 2022.
Jerome Revaud, Cesar De Souza, Martin Humenberger, and Philippe Weinzaepfel. R2d2: Reliable and repeatable detector and descriptor. Advances in Neural Information Processing Systems, 32, 2019.
Ethan Rublee, Vincent Rabaud, Kurt Konolige, and Gary Bradski. Orb: An efficient alternative to sift or surf. In International Conference Computer Vision, pp.
Iece, 2011. 1.
Radu Bogdan Rusu, Nico Blodow, and Michael Beetz. Fast point feature histograms (fpfh) for 3d registration. In IEEE International Conference on Robotics and Automation, pp. 3212-3217
IEEE, 2009.
Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. Superglue: Learning feature matching with graph neural networks. In Proceedings of the IEEE/CVF Confer- Computer Vision and Pattern Recognition, pp.
2020.
Paul-Edouard Sarlin, Daniel DeTone, Tsun- Yi Yang, Armen Avetisyan, Julian Straub, Tomasz Mal- isiewiez, Samuel Rota Bulo, Richard Newcombe, Peter Kontschieder, and Vasileios Balntas. Ori- enternet: Visual localization in 2d public maps with neural matching. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, pp. 21632-21644 2023.
12

---

Published as a conference paper at ICLR 2026
Thomas Schops, Viktor Larsson, Mare Pollefeys, and Torsten Sattler. Why having 10,000 parame- ters in your camera model is better than twelve. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2535- 2020.
Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 815 815-823, 2015.
Zehong Shen, Jiaming Sun, Yuang Wang, Xingyi He, Hujun Bao, and Xiaowei Zhou. Semi-dense feature matching with transformers and its applications in multiple- view geometry. IEEE Trans- actions Pattern Analysis and Machine Intelligence, 45(6):7726-7738, 2023.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. International Conference on Learning Representations, 2015.
Jiaming Sun, Zehong Shen, Yuang Wang, Hujun Bao, and Xiaowei Zhou. Loftr: Detector-free local feature matching with transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8922-8931, 2021.
Yifan Sun, Changmao Cheng, Yuhan Zhang, Chi Zhang, Liang Zheng, Zhongdao Wang, and Yichen Wei. Circle loss: A unified perspective of pair similarity optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2020.
Hugues Thomas, Charles R Qi, Jean-Emmanuel Deschaud, Beatriz Marcotegui, François Goulette, and Leonidas J Guibas. Kpconv: Flexible and deformable convolution for point clouds. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6411-6420, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances Neural Infor- mation Processing Systems, volume 30, 2017.
Haiping Wang. Yuan Liu, Bing WANG, YUJING SUN, Zhen Dong, Wenping Wang. and Bisheng Yang- Freereg: Image-to-point cloud registration leveraging pretrained diffusion models and monocular depth estimators. In The Twelfth International Conference on Learning Representa- fions, 2024a. URI.https: /ttps::///peee///////
net. forum? wtwp:mmm..ddbPPPh.TP2
Yifan Wang, Xingyi He, Sida Peng, Dongli Tan, and Xiaowei Zhou. Efficient loftr: Semi-dense local feature matching with sparse-like speed. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 21666-21675, 2024b.
Hao Yu, Fu Li, Mahdi Saleh, Benjamin Busam, and Slobodan Ilc.. Cofinet: Reliable coarse-to-fine correspondences for robust pointcloud registration. Advances in Neural Information Processing Systems, 34:23372-2388888 2021.
Jason Y. Zhang. Amy Lin, Moneish Kumar, Tzuu-suaan Yang, Deva Ramanan, and Shubham Tul- siani. Cameras as rays: Pose estimation via ray diffusion. In The Twelfth International Confer- Learning Representations, 2024. URL https: pettps://ppene htt::///ppeneeeiieee net forum?id EanCFCWA
Yu Zhong. Intrinsic shape signatures: shape descriptor for 3d object recognition. In IEEE inter- national Conference Computer Vision Workshops, pp. 689-696. IEEE, 2009.
Junsheng Zhou, Baorui Ma, Wenyuan Zhang, Yi Fang, Yu-Shen Liu, and Zhizhong Han. Dif- ferentiable registration of images and lidar point clouds with voxelpoint-to-pixll matching. In Advances in Neural Information Processing Systems, volume 36, pp. 51166-51177, 2023.
Qunjie Zhou, Torsten Sattler, and Laura Leal-Taixe. Patch2pix: Epipolar-guided pixel-level cor- respondences. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4669 4678, 2021.
Yi Zhou, Connelly Barnes, Jingwan Lu, Jimei Yang, and Hao Li. On the continuity of rotation representations neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5745-5753, 2019.
13

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.I EVALUATION METRICS
To quantitatively evaluate the performance of image-to-point cloud registration, we adopt two widely used metrics that measure the accuracy of estimated 6-DoF (Degrees of Freedom) transformation: Relative Translation Error (RTE) and Relative Rotation Error (RRE). RTE assesses the devia- tion in translation, while RRE quantifies the discrepancy in rotation between predicted and ground- truth camera poses.
A.1.1 RELATIVE TRANSLATION ERROR (RTE)
RTE measures the Fuclidean distance between predicted translation vector and ground-trute truth trans- lation vector tyt, thereby reflecting the accuracy of estimated camera position in the global coordi- nate frame. It is defined as:
RTE ||
t 122-
(11)
This metric is expressed in meters and provides an interpretable measure of positional deviation. A lower RTE indicates a more accurate translation estimate, with an ideal value zero corresponding to perfect alignment.
A.1.2 RELATIVE ROTATION ERROR (RRE)
RRE evaluates the rotational misalignment between estimated rotation matrix R and ground-trui truth rotation matrix Rgt- Specifically, it is computed based on the residual rotation that aligns the pre- dicted orientation to the ground truth:
Ryd R R.
(12)
We convert the residual rotation matrix Ryet Buler angles y [n, 72, ra]" (typically using the ZYX convention), and define RRE as the sum of the absolute angular deviations along the three principal axes:
RRE Σ
Eh(1-
(13)
RRE is reported in degrees, and a smaller value corresponds to higher rotational accuracy.
A.1.3 SUMMARY
Together, RTE and RRE provide comprehensive assessment of pose estimation accuracy. These metrics are particularly suitable for evaluating cross-modal registration methods, where both posi- tional and orientational consistency between modalities critical.
A.2 CLASSICAL POSE SOLVER VS. OUR RAY-GUIDED POSE REGRESSION MODULE
To further analyze the stability of different pose estimation strategies, we provide visual comparisons between classical pose solver (Equation 3 and 4 and our ray-guided pose regression module. As shown in Figure 5 classical pose solver often produces unstable results when predicted rays noisy or partially misaligned, leading to larger rotation errors and variance. This is because classical solvers rely heavily fixed mathematical models and assumptions that do not adapt well to noisy or partially misaligned data. In contrast, our proposed module is learnable and thus can adapt to the specific characteristics of the input data. By learning from diverse examples, the module able handle noise and misalignment more effectively, producing more stable pose estimates even under challenging conditions. This adaptivity and robustness make our method more reliable, especially in real-world scenarios where noise, occlusion, and partial misalignment are common.
A.3 ABLATION STUDIES ON PATCH SIZE
We conduct an ablation study to examine how the patch size associated with each ray influences registration performance, shown in Table Each ray corresponds to a p p local image patch,
14

---

Published as a conference paper at ICLR 2026
Classical Pose Solver
Ray-guided Regression Module
(Ours)
Figure 5: Visual comparison between classical pose solver and our proposed ray-guided pose re- gression module. Classical pose solver suffers from unstable predictions under noisy rays, whereas our method yields more robust and accurate pose estimations. Predicted rays and camera pose are visualized in red, while ground-truie -truth rays and pose are shown in green.
controlling the spatial granularity of feature extraction. We find that moderate patch size (p 8) achieves the best performance, balancing spatial resolution and contextual coverage. Smaller patches (p 4) lack context, while larger ones (p 16 32) reduce spatial precision, both leading to degraded performance.
Table 5: Ablation studies on patch size associated with each ray. Each ray corresponds to X
local image patch.
KITTI
nuScenes RRE(")
RTE(m))
RRE(") )
Acc(%)
RTE(m)
Acc(%))
0.12::0.09
0.85+0.96
99.34
0.42:10.31
1.8416.06
94.71
8
0.09-10.08
0.63
99.75
0.39:10.29
1.48
96.61
16
0.21
0.95#0.93
99.43
0.43+0030
1.82+6.02
95.25
32
0.18+0.14
0.83+0.89
99.34
0.44+00.1
2.44+6.04
90.31
A.4 ABLATION ON FOCUS LOSS ffoc
We investigate the impact of focus radius a, which controls spatial constraints in cross-attention between patch and point features. As shown in Table 6 increasing from 8 to 32 improves per- formance, suggesting that a broader local neighborhood provides richer 3D context for registration. However, further increasing beyond 32 degrades performance, likely due the inclusion of ir- relevant or noisy points that weaken focused interactions. Overall, = 32 offers the best trade-off' between geometric context and locality, and is adopted in all experiments.
A.5 LIMITATIONS AND FUTURE WORK
While our method achieves competitive performance on challenging outdoor datasets, it still exhibits certain limitation primarily associated with the reliance on overlap prediction, which common challenge shared by almost all outdoor image-to-poine cloud registration methods.
A.5.1 ROBUST POSE ESTIMATION UNDER INACCURATE OVERLAP PREDICTION
Our approach incorporates a learned overlapping region detector module to guide cross attention between 2D image patches and 3D point clouds. In practice, find that even when the predicted
15

---

Published as a conference paper at ICLR 2026
Table 6: Ablation studies focus radius Here X indicates method that doesn't use Looe-
KITTI
nuScenes RRE(O
RTH(m))
RRE())
Acc(%))
RTH(m)
Acc((6)t
x
0.10:10.09
1.02+1.000
99.18
0.42:10.31
1.94
94.44
8
0.11=00.09
0.75
99.30
0.41-00.0
1.87:6639
94.70
16
0.10:10.13
0.76-0.85
99.52
0.41+00.0
1.85:1.66
95.12
32
0.09:10.08
0.6310.71
99.75
0.39-10.29
1.48
96.61
64
0.07=1000
0.67=0.81
99.48
0.40:10.29
1.79:16.00
95.81
128
0.09=
0.91
99.43
0.41=00229
1.77=6.57
95,43
256
0.17:1011
0.94
99.39
0.42:10.30
1.85:16996
95.39
overlapping region is only partially correct, the model can still achieve accurate pose estimation as shown in Figure [ This is primarily because cross attention mechanism is capable of selectively attending informative rays within correctly predicted overlapping subset. Moreover, rays outside the predicted overlapping region can also benefit from their geometric and contextual relationships with rays inside the correctly predicted area, enabling accurate alignment despite partial supervision. These observations suggest that our method exhibits strong robustness to imperfect overlap predic- tion and does not rely heavily on highly accurate overlap masks to perform successful registration.
A.5.2 FAILURE CASES UNDER COMPLETELY INCORRECT OVERLAP PREDICTION
In contrast to partially correct overlap predictions, when the predicted overlapping region is entirely incorrect-i.e it contains no part of the true overlapping area-the model is fundamentally un- able to establish any meaningful cross-modal interactions. Under this condition, the cross attention mechanism is misled and lacks access to any informative cues, resulting in failed ray-level reasoning across the modalities. Consequently, none of the predicted rays be reliably estimated, and the final pose prediction becomes highly inaccurate, as shown in Fig. 7 This failure mode, although observed only in rare extreme cases, reveals fundamental limitation of the current framework: when the predicted overlap region is entirely incorrect, the model lacks valid guidance for effective cross-modal interaction. Despite its infrequency, this issue highlights the dependency on overlap supervision and motivates future research toward wwerlap-indepenee pose estimation strategies.
A.5.3 TOWARD OVERLAP-INDEPENDOE REGISTRATION
Although our framework exhibits a certain degree of robustness to overlap prediction errors, its performance still fundamentally depends on the overlap estimation being least approximately cor- rect. In future work, we plan to eliminate this reliance by designing overlap-free pose estimation methods. One promising future direction is to remove the explicit overlap prediction module alto- gether, allowing the network to implicitly learn to attend to the correct overlapping regions on its own By leveraging latent cross-modal alignment cues, the model can infer relevant associations without relying explicit supervision of overlap areas. Such approach is expected to enhance both the robustness and generalization capability of the system, especially in challenging scenarios with ambiguous noisy geometry.
A.6 MORE IMPLEMENTATION DETAILS
Here we offer more implementation details about our proposed method. For feature extraction, we adopt backbones commonly used in existing approaches (Li et al. (023). Specially, we adopt 4-stage ResNet (He et al. 2016) as the image backbone, where the output channel dimension is 512. The output feature map is downsampled by factor of relative to the input image, yielding a resolution of 20 x 64 for KITTI and 20 40 for nuScenes. For the 3D backbone, we use 4-stage KPCony (Thoms et al. 2019) where the output channel dimension is 512. The point clouds are voxelized with an initial voxel size of 15em for both the KITTI dataset and nuScenes dataset. The batch size is set as 2. All experiments are conducted on a single RTX 3090 GPU. We implement our code using PyTorch 1.13.1 and CUDA 11.7.
16

---

Published as a conference paper at ICLR 2026
Overlap Prediction
Ray & Pose Prediction
Figure 6: Robust Pose Estimation Under Inaccurate Overlap Prediction. Visualization of pre- dicted overlapping regions and final registration results. Despite imperfect overlap prediction, our method achieves accurate pose estimation, showing strong robustness. Correctly identified overlap- ping points are shown in blue, missed overlaps in green, and false positives in red. Predicted rays and camera pose visualized in red, while ground-truth rays and pose shown in green.
A.7 THE USE OF LARGE LANGUAGE MODELS (LLMs)
In this paper, we utilized ChatGPT, a large language model (LLM), to assist in the refinement and polishing of our writing. Specifically, ChatGPT was employed improve the clarity, coherence, and overall presentation of the text. It helped with tasks such as rephrasing sentences, correcting grammatical errors, and suggesting improvements in academic tone. However, the contributions of the LLM were limited to writing assistance only and did not extend to the conceptualization, analysis, or development of the research. All ideas, methodologies, and experimental work presented in this paper were independently conceived and executed by the authors.
17

---

Published as a conference paper at ICLR 2026
Overlap Prediction
Ray & Pose Prediction
Figure 7: Failure Cases Under Completely Incorrect Overlap Prediction. Visualization of a rare but critical failure mode where the predicted overlapping region contains no part of the true over- lapping area. In this extreme case, the cross attention module is misled and lacks any informative guidance due to completely incorrect overlap prediction, leading to failed ray reasoning and inaccu- rate pose estimation.
18


Reviews:
[{"review_id": "TgEsjZrwvP", "reviewer": "Reviewer_6CqR", "summary": "This paper proposed a method for image-to-point cloud registration. Matching-based approaches is challenged by projection ambiguity and depth-induced scale inconsistency. To adress these problems, they introduce a differentiable ray-guided regression module which regress camera pose from predicted Plu ̈cker rays, thereby naturally resolving projection ambiguity and depth-induced scale ambiguity. The authors conduct experiments on KITTI and nuScenes. Method are evaluated by Relative Translation Error (RTE), average Relative Rotation Error (RRE), and registration accuracy (Acc). Compared to existing state-of-the-art approaches, the proposed method achieves strong accuracy, while remaining computationally efficient.", "questions": "- Can the ray errors could be mitigated using diffusion-based methods?\n- If the error threshold for Acc is set smaller, can the regression-based method achieve precision comparable to that of matching-based methods?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper is clearly motivated and well written. The paper intoduces a novel ray-guided pose regression module which addresses projection ambiguity and depth-induced scale inconsistency. Its technical rationale is supported by findings in the generalized camera models. The experimental setup is comprehensive, covering multiple datasets, metrics and baselines.", "weaknesses": "- The author did not analyze the causes of errors in the predicted rays. The neural regression module seems more like a compensatory measure; however, its use may raise concerns about generalization.\n- Some of the illustrative figures are not very clear. In Figure 3, does the color of the points represent the depth error or the actual depth value?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "j8evKs18hG", "reviewer": "Reviewer_RNei", "summary": "This paper proposes an image-to-point cloud registration method that estimates the 6-DoF camera pose of a query image relative to a 3D point cloud map. The proposed ray-based registration framework first predicts patch-wise 3D ray bundles connecting image patches to the 3D scene, then estimates the camera pose via a differentiable ray-guided regression module.", "questions": "What is the key challenge in image-to-point cloud registration compared to ray-based representation for pose estimation?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed ray-based registration framework for image-to-point cloud registration is interesting.\n- The experimental results verified the effectiveness of the proposed method.", "weaknesses": "- Additional discussion comparing the proposed method with other ray-based representation methods [1] should be added.\n- The approach appears to be a direct application of ray-based representation for pose estimation to the image-to-point cloud registration task. The specific challenge this addresses should be further clarified.\n\n[1] Jason Y. Zhang, Amy Lin, Moneish Kumar, Tzu-Hsuan Yang, Deva Ramanan, and Shubham Tulsiani. Cameras as rays: Pose estimation via ray diffusion. In *The Twelfth International Conference on Learning Representations*, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BVCpkSTyiK", "reviewer": "Reviewer_JaMP", "summary": "This paper addresses the problem of registration between a 3d point cloud and an image of the same scene. Compared to the problems of 3d registration between two\npoint clouds or image matching between two images, this problem is significantly more difficult. This difficulty comes from the ambiguities introduced by projecting 3d points \nto images and finding the correct matches in the images. Learning based approaches have been dominating the field. The main novelty in this paper is the use of the rays to \nparameterize the pose of the point cloud to the camera coordinate system. The rays are represented using Plücker coordinates, and this has been shown to be beneficial for \ncamera pose estimation and for 6d object pose estimation. This parameterization seems to be easier to learn and brings other benefits, like more precise alignment and \nfewer projection ambiguities, and scale inconsistencies. The paper is well-written and not difficult to follow. The novelty seems to be limited, since all elements of the \nproposed method have been known before and just combined using the ray representation. The overall pipeline has two stages composed of the ray prediction module, which basically does \n3d point cloud and image feature fusion and is fairly standard and doesn't depend on the ray parameterization of the image patches. The second stage is a ray-guided regression module\nthat uses ray parameterization of the image patches, fused multi-modal features, and regresses the pose in terms of rotation and translation. The pipeline is supervised and applied to autonomous driving scenarios, where alignment between LIDAR scans and RGB images has been performed.", "questions": "Can you give more precise intuition behind using the reference rays? How do the guides and stabilize the regression process?\n\nWhat will happen if you do not predict rays, but matches between 3d points and patches?\n\n\nThe results from Table 1 show that GraphI2P is actually performing very well and it is on par or better than the proposed method. What's the reason? What is the advantage of the proposed approach compared to GraphI2P?\n\nIn the ablation study, the results in Table 3 are a bit confusing. It is unclear what the fused features bring, as the rotation estimates in KITTI are more stable (smaller std) than in NuScenes. How do you explain this? What will happen if you have FPF and CPS only? RR seems to improve rotation estimation. Why?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The paper is well-motivated and well-written. It was easy to follow it. Using ray parameterization for estimating the pose of the point cloud in the camera coordinate system\nThis makes sense, as it has been demonstrated for related problems, such as image pose estimation [1] and object pose estimation [2]. However, both approaches use this ray parameterization \nfor underlying diffusion models. Here, this is not the case. What I find interesting is the use of reference rays in the ray-guided pose regression. However, this is not well motivated.", "weaknesses": "The proposed method has limited novelty. The first part of the feature fusion, known as the ray prediction module, is not new and doesn't benefit from the camera ray parameterization; however, it utilizes fused features and predicts image patch rays. The motivation behind the use of the reference rays and predicted rays is not given. The paper \ncontains numerous ad-hoc design choices that are not clearly motivated or explained. See questions section for more details.\nOn the experimental side, I find the evaluation limited. It has been done only on two autonomous driving datasets. Why is this choice made for only this dataset and not some indoor datasets. Better motivation is needed for readers who are not interested in autonomous driving. I suppose the reason is that LIDAR and RGB images are not registered in the autonomous vehicles.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)