Alligat0R: Pre-Training through Covisibility Segmentation for Relative Camera Pose Regression
Authors: Thibaut Loiseau
Guillaume Bourmaud
Vincent Lepetit
Venue: NeurIPS 2025 spotlight ()
Keywords: pre-training
co-visibility
3D
pose estimation
pose regression
Primary area: applications
TL;DR: A new pre-training method for two-view pose estimation, which is classifying each pixel as co-visible, occluded, or outside field-of-view in the other view.

Abstract: Pre-training techniques have greatly advanced computer vision, with CroCo’s cross-view completion approach yielding impressive results in tasks like 3D reconstruction and pose regression. However, cross-view completion is ill-posed in non-covisible regions, limiting its effectiveness. We introduce Alligat0R, a novel pre-training approach that replaces cross-view learning with a covisibility segmentation task. Our method predicts whether each pixel in one image is covisible in the second image, occluded, or outside the field of view, making the pre-training effective in both covisible and non-covisible regions, and provides interpretable predictions. To support this, we present Cub3, a large-scale dataset with 5M image pairs and dense covisibility annotations derived from the nuScenes and ScanNet datasets. Cub3 includes diverse scenarios with varying degrees of overlap. The experiments show that our novel pre-training method Alligat0R significantly outperforms CroCo in relative pose regression. Alligat0R and Cub3 will be made publicly available.

AlligatOR: Pre-Training through Covisibility Segmentation for Relative Camera Pose Regression
Thibaut Loiseau'
Guillaume Bourmaud Vincent Lepetit
LIGM, Ecole des Ponts, Univ. Gustave Eiffel, CNRS, France
2 Univ. Bordeaux, CNRS, Bordeaux INP, IMS, UMR 5218, France
(thibaut. loiseau, vincent. hepetteteeettttepc..... guillaume. bourmaud@uu-orrdeuux.
CroCo
AlligatOR (Ours)
view
Reference view?
Reconstruction
Coviibilityy segmentatin
Target
Mask1
Mask
Outsida FOV
Requires pairs =50% ovarlap Limited geometric diversity
Wirkes with any deggrse cuerlap interpretable undarstandding Significart geomatric diversity
Figure 1: We introduce AlligatOR, a novel pretraining method for binocular vision. AlligatOR ex- plicitly segments pixels covisible, occluded, or outside field-of-view, overcoming the fundamental which attempts to reconstruct potentially non-covisible regions.
limitation of CroCo 447 48
Abstract
Pre-training techniques have greatly advanced computer vision, with CroCo's cross-view completion approach yielding impressive results in tasks like 3D re- construction and pose regression. However, cross-view completion ill-posed non-covisible regions, limiting its effectiveness. We introduce AlligatOR, a novel pre-training approach that replaces cross-view learning with a covisibility segmen- tation task. Our method predicts whether each pixel in one image is covisible in the second image, occluded, or outside the field of view, making the pre-training effective in both covisible and non-covisible regions, and provides interpretable predictions. support this, we present Cub3, a large-scale dataset with 5M image pairs and dense covisibility annotations derived from the nuScenes and ScanNet datasets. Cub3 includes diverse scenarios with varying degrees of overlap. The experiments show that our novel pre-training method AlligatOR significantly out- performs CroCo in relative pose regression. AlligatOR and Cub3 will be made publicly available.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
Pre-training techniques have revolutionized computer vision by enabling large models to learn rich representations [5] [2 T9 18 55 In 3D computer vision, CroCo 47 48 pioneered cross- view completion as a pretext task, where one view is partially masked and reconstructed using visible portions along with second reference view of the same scene (see Figure I left). This approach allows to learn powerful 3D features and has made possible impressive results on downstream tasks such as 3D reconstruction 45
14 49 42 3D pose regression
[ camera calibration 266]. 41)
reconstruction [52 23 44)
and Gaussian splatting 35 51].
Despite its success, the cross-view completion pretext task has fundamental limitation: pretraining only effective in covisible regions, as the task is ill-posed in non-covisible regions (see Figure I
As consequence, Croco 47 48
relies on pairs with at least 50% of covisible regions.
In this paper, we present AlligatOR, a novel pre-training approach that offers an alternative to the cross-view completion objective through covisibility segmentation task. Instead of reconstructing masked regions, our method explicitly predicts whether each pixel one image is: (1) covisible in the second image, (2) occluded, or (3) outside the field of view (FOV). This formulation offers several advantages: it makes the pre-traie trainining effective in both covisible and non non-covisible regions, provides interpretable predictions that reveal the model's geometric understanding, and it aligns more directly with the correspondence reasoning required downstream binocular vision tasks.
To enable our approach, we introduce Cub3, large-scale dataset comprising two sub datasets, each of 2.5 million image pairs with dense covisibility annotations derived from nuScenes 4 and ScanNet [8] datasets. Cub3 includes challenging scenarios with varying degrees of overlap between views, providing more diverse training signal than previous approaches.
One might notice that AlligatOR is not strictly self-supervised, unlike some earlier pre-training methods, it relies on covisibility annotations. However, the same can be said for CroCo, which requires filtering image pairs to retain only those with at least 50% overlap. The creation of our covisibility annotations for nuScenes more complex than for ScanNet, as the ground truth poses and depth maps are not available, however the process is fully automated. This process is very similar the one used in RUBIK 28 and relies on registered video sequences and depth predictions.
Our contributions can be summarized as follows:
1. We introduce covisibility segmentation as novel pre-training objective for binocular vision tasks, replacing the tross-completion-Noe based approach of prior work while maintaining the same network architecture.
We create and release Cub3, a large-scale dataset with dense covisibility annotations derived from nuScenes and ScanNet datasets.
We show that AlligatOR provides an effective alternative to CroCo pre-training when fine-tuned on the relative pose regression task, particularly for challenging scenarios with limited overlap between views.
4. We demonstrate the effectiveness of our pre-training and provide insights into the model's cross- view reasoning capabilities through interpretable segmentation outputs.
Our experiments demonstrate that explicitly learning to understand covisibility relationships between image pairs leads to more robust and transferable features for relative pose regression compared to reconstruction-base. approaches.
Related Work
Pretraining on Image Pairs. To the best of our knowledge, CroCo 47 pioneered the extension
of masked image modeling (20 to image pairs, introducing cross-view completion. CroCo v2 48 applied this cross-vies view completion framework large amounts of data. P-Match [56] introduced variation where both images are partially masked to pre-train an image matching model. Another variant, masked appearance transfer, was proposed in (53 for object tracking, with a similar approach
in [36]. Let us highlight that the foundational models DUSt3R [45] and MASt3R (26) are built on
CroCo. While all these works rely on -rews completion, we depart from this framework and propose novel pretraining objective based on covisibility segmentation.

---

39 32) to [i] Pre-training
Sugrnu Sigrontatiin Pronr
VIT enoonder
deconder
Formar
Fine turling
MLP,
Sharce
Puzch
VIT emooder
Trar sfarner cleoddrr
MI.P
Segrennttiion Feod
Figure Overview of AlligatOR. (a) During pre-training, we use the same architecture as CroCo but replace the reconstruction objective with a covisibility segmentation task, without masking, where each pixel in one view classified as covisible, occluded, or outside FOV with respect to the other view. (b) For fine-tuning on the relative pose regression task, we pool features from both views, process them through shared MLP, and use separate heads for predicting translation and rotation.
Considering covisibility across image pairs.
Ground-truth covisibility masks are widely used 6
in image matching methods [31 12, 27 54
37
46
13
43
40
I
17
25
16
22
16 also leverage these masks to learn matchability scores, which are used at test time to select correspondences. Additionally, [2I] jointly learns covisibility and relative planar pose between 360 panoramas. To the best of our knowledge, our work is the first to learn visual representations from covisibility segmentation.
compute overlaps and select training pairs. Several approaches
32
Fine-tuning for Relative Pose Regression. Reloc3R ( recently fine-tuned the foundational model DUSt3R [45]. which is based CroCo, large-scale data for relative pose regression. In our experiments, we adopt a similar strategy to evaluate our novel pretraining method against CroCo. However, fine-tune AlligatOR for metric relative translation, whereas Reloc3R predicts only the direction of the relative translation.
3 Method
In this section, describe novel pre-training approach, AlligatOR, which replaces the cross-view completion objective used in CroCo [7 48 by segmentation task. We first outline the overall architecture, which remains largely similar to CroCo, and then detail our covisibility segmentation pre- pre-training objective. Finally, we explain how our model can be fine-tuned for downstream tasks such relative pose regression.
3.1 Architecture Overview
Our architecture closely follows the design of CroCo, which consists of an encoder and decoder. The encoder is ViT [10] that processes the input images by dividing them into non-overlapping patches and encoding them into feature representations. The decoder is another transformer that combines information from both views, using cross-attention layers, make predictions.
We adopt symmetric architecture for our forward pass, where both images are processed in the same way without any masking. This differs from CroCo, which uses asymmetric approach that heavily masks one image while leaving the other unmasked. Our symmetric design not only more efficient but also better aligned with downstream binocular vision tasks, which typically process unmasked images.
As shown in Figure 2 (a), given two images I and I, of the same seene taken from different viewpoints, we first divide both into non-overlapping patches, denoted as tokens P1 (P)....,PP) and P2 {P]..... PN

---

The encoder Eg processes the tokens from both images independently:
F1 (P(), F2 (P2)
(1)
and the decoder D then takes these features and processes them to enable cross-view view reasoning:
D (F1. O2 = (F2.
(2)
where 02 and O2 represent the output features from the decoder. The decoder contains cross-attention mechanisms that allow information exchange between features from both views.
3.2 Covisibility Segmentation Pre-Training
key difference between our approach and CroCo lies in the pre-training objective. Instead of reconstructing masked pixels, we formulate the pre-training task as covisibility segmentation problem. Our goal is to predict for each pixel in each image whether is:
1. Covisible: the pixel corresponds to 3D point that is also visible in the other image.
Occluded: the pixel corresponds to a point that is occluded in the other image.
3. Outside FOV: the pixel corresponds to 3D point that is outside the field of view in the other image.
Formally, for each patch token, the decoder produces a feature vector that is processed by a fully- connected layer output probabilities for each of the three classes, for each pixel:
Yuk: softmax Ojk + b)j
where Yuy E IR represents the predicted probabilities for the three covisibility classes for pixel j in patch k of image i, and W and b are learnable parameters.
During pre-training, we optimize the network using a cross- cross-entropy loss in each image:
N.
Leee
In In(gg.a)))
(4)
N
where Vij.ery is the predicted probability for the ground-truth class Gyj in pixel of image and N; the number of pixels in each image. The total cross-entropy loss is the sum over both images:
Lee = Lee, Leez-
This formulation offers several advantages over the www.vvewwwooomttionnnn based approach. First, makes the pre-training effective in both covisible and non-covisible regions, we explicitly model cases where pixels have no covisible region in the other view. Second, provides interpretable outputs that directly reveal the model's understanding of scene geometry. Third, it better aligns with downstream tasks such as pose regression, where both images are fully visible.
3.3 Fine-Tuning for Relative Pose Regression
After pre-training, we fine-tune our model for the task of relative pose regression. We add a pose regression head on top of the pre-trained encoder-decoder architecture, while keeping the original covisibility segmentation head. This design allows the model to leverage the geometric understanding acquired during pre-training.
The pose regression head consists of several components as shown Figure 2 (b). First, we apply global average pooling to the decoder outputs for both images:
fl GlobolAvgPool(O)), f2 GlobalAvgPool
(5)
The pooled features concatenated and processed through shared MLP:
fstured MLP((f, fz))
(6)
We then use separate heads for predicting the metric relative translation vector i E R and the relative rotation represented quaternion 9 E R
f. MI.P (fatune)) q = MLP (L.Pq(fshurea))
(7)
We use unit quaternions with 1.2 normalization, treating them as 4D vectors in the homoscedastic loss function.

---

Our fine-tuning stage consists of two phases:
1. First, we freeze the pre-trained encoder, decoder and covisibility segmentation head, and only train the pose regression head. During this phase, we homoscedastic loss [2 combining MSE losses for translation and quaternion predictions:
1 20
pose
8||
11 -qll log + log oq
(8)
20
where t and are the ground-trute metric translation and quaternion respectively, and oe and dq are learnable parameters that automatically balance the two loss terms.
In the second phase, we unfreeze the backbone and the covisibility segmentation head, and train the full network with joint loss that combines the pose regression loss and the covisibility segmentation loss:
1 Lee log Sseg
Sjoint
Ppsse
20
where Ssg is learnable parameter.
This training strategy ensures that the model maintains its interpretable covisibility segmentation capabilities while being optimized for the pose regression task.
Cub3: A Large-Scale Covisibility Dataset
To enable our covisibility segmentation approach, we introduce Cub3, a large-scale dataset comprising two sub datasets each of 2.5 million image pairs with dense covisibility annotations derived from both the autonomous driving nuScenes dataset 4 and the indoor ScanNet [ dataset. Cub3 will be made publicly available.
4.1 Dataset Construction
nuScenes- - We leverage the covisibility estimation pipeline introduced in RUBIK 28 to generate pixel-level covisibility annotations for image pairs from nuScenes. In brief, this pipeline uses monocular metric depth predictions from UniDepth 30) and surface normals from Depth Anything reconstructions,
automatically classify each pixel as either covisible, occluded, or outside FOV with respect to another image. We apply this pipeline to all possible image pairs within each scene, resulting in approximately 34 million annotated pairs.
v2 [50). combined with camera poses from COLMAP 33
ScanNet
The pipeline is similar but simpler for ScanNet, as we rely on the ground truth depths and camera poses to get all covisibility annotations.
For our experiments, we created two dataset variants, each containing 5M image pairs (2.5M from nuScenes and 2.5M from ScanNet):
Cub3-50: Image pairs with at least 50% overlap, similar to the criterion used in CroCo [48] This dataset provides direct comparison point with existing cross-view completion approaches.
Cub3-all: Image pairs with at least 5% overlap. This dataset, contains challenging image pairs test the robustness of methods to handle low-overlap scenarios that common in real-world applications.
Figure 3 shows examples or covisibility annotations image pairs from Cub3, illustrating how our process classifies pixels into the three categories across varying levels difficulty.
It is also worth noting that our annotation process, especially on nuScenes, inherits some limitations from the underlying monocular depth estimation models. The depth predictions may occasionally struggle with challenging scenarios such as reflective surfaces, transparent objects, or regions with complex geometry or far-away objects. Consequently, some annotations our dataset, particularly the distinction between covisible and occluded pixels, may contain noise. Despite these imperfections, our experiments show that the scale and diversity of Cub3 enable effective training of robust covisibility segmentation models.
5

---

Figure 3: Covisibility annotation examples from Cub3 for nuScenes (top) and ScanNet (bottom). For each image pair, show the corresponding covisibility maps with color-coding for covisible, occluded, and outside FOV regions. Note how our annotation process handles varying degrees of overlap and challenging viewpoint changes. Let highlight that some annotations, particularly the distinction between covisible and occluded pixels, may contain noise, especially for nuScenes, and demonstrate in the experiments that AlligatOR is highly robust to this noise.
100003
252003
Duraset
Docaset
100003
200003
Cub3
200003
100003
200003
Overipp 1961
Scale rat.
Visapo angle
Ovariap (S)
Scale 1at.0
Viserpo angle
Figure 4: Distributions of overlap, scale ratio, and viewpoint angle in Cub3-all and Cub3-50 for nuScenes (left) and ScanNet (right).
4.2 Dataset Statistics
illustrates the distribution of datasets across three geometric criteria introduced in RUBIK 28): overlap percentage, scale ratio, and viewpoint angle. The Cub3-all dataset's distribution heavily skewed towards very challenging pairs, which are particularly difficult for cross-completion methods like CroCo. In contrast, our covisibility segmentation approach is designed to benefit from such challenging cases effectively.
Figure 4
Given our computational resources, Cub3 is currently limited to either autonomous driving scenarios from nuScenes, indoor scenes from ScanNet. But such scenarios have very important applications, and given its size, variety, and challenges, Cub3 is already sufficient to validate AlligatOR- -very -verry much like the initial version of CroCo 47 was demonstrated "proof-of-concept"
With 5M annotated pairs, Cub3 provides sufficient data train and validate approach on both outdoor and indoor scenarios.
Scale:
Geometric variety: It covers a wide range of overlaps, scales and viewpoint angles within the driving and indoor contexts.
Challenging scenarios: includes pairs with minimal overlap that test the limits of CroCo, as was done in the original paper.
Our experiments on the RUBIK benchmark, which was specifically designed evaluate challenging autonomous driving scenarios, as well as results on ScanNet1500 test set demonstrate the effectiveness of our approach. The promising results we obtain motivate future work to extend Cub3 to more diverse environments beyond urban driving scenes and indoor scenes. However, even within this specific domain, our method shows significant improvements over CroCo, particularly in geometrically challenging configurations.

---

Experiments
5.1 Implementation Details
We implement AlligatOR using PyTorch and conduct all experiments NVIDIA A GPUs. The model architecture follows the design described in Section 3.1, The input images are classically resized to have maximum dimension of 512 pixels while maintaining their aspect ratio. More information is provided in the supplementary material.
5.2 Main Results
Table I: Results on RUBIK [28] and ScanNet 500 S for metric relative pose regression. For experiments, fine-tuning is performed using Cub3-all.
RUBIK
Scan
Pre-training
Pre-Training Set Backbone
5' 0.55m
5" 2m
10" 55
10"
/0.25m
10" /0.5m
10" Im
CroCo
Cub3-50
*
4.4
21.8
48.1
48.3
64.1
69.3
CroCo
Cub3-all
*
2.4
8.9
25.2
8.7
18.3
24.9
AlligatOR
Cuh3-50
*
9.3
32.4
58.4
47.5
59.9
63.4
AlligatOR
Cub3-all
*
24.0
55.3
82.3
78.2
92.5
CroCo
Cuh3-50
12.4
38.3
66.7
75.7
87.4
91.5
CroCo
Cuh3-all
6
12.5
38.6
66.2
64.1
79.1
85.0
AlligatOR
Cub3-50
21.2
53.3
78.1
82.8
91.7
93.9
AlligutOR
Cub3-all
24.6
60.3
81.9
83.5
92.5
95.1
From scratch
Cub3-all
10.8
29.1
52.3
37.0
51.8
57.2
Metric Relative Pose Regression Performance. We evaluate our proposed AlligatOR pre-training approach the metric relative pose regression task and compare it with CroCo pre-training using the same architecture and fine-tuning protocol. [ the results on the RUBIK (28) benchmark, which specifically designed to evaluate performance across different geometric challenges in autonomous driving scenarios, and on the indoor Sca
[8] benchmark.
As shown in Table AlligatOR consistently outperforms CroCo across different training and eval- uation configurations. Notably, when pre-trained on Cub3-all and fine-tuned with the backbone frozen, AlligatOR achieves significantly higher performance and reaches tfate-of-thee -fttt-rrtttttttt RUBIK benchmark (55.3% success rate at 52/2m and 82.3% at 10%/5m) compared to the CroCo model pre-trained the same dataset (which only achieves 8.9% and 25.2% respectively in the frozen backbone configuration). This substantial improvement demonstrates the effectiveness of our covisibility segmentation pre-training approach, especially for challenging scenarios with varying degrees of overlap.
the
Impact of Training Data Distribution. A key observation is the contrasting behavior of AlligatOR and CroCo when trained on different data distributions. While CroCo performs similarly or better when trained easier pairs (Cub3-50) compared to harder pairs (Cub3-all), AlligatOR shows different trend, Our method systematically improves when trained on the full distribution (Cub3-all), This demonstrates that AlligatOR effectively learn from challenging examples, while CroCo struggles to benefit from them.
Figure 5 further illustrates this advantage by breaking down performance across different overlap percentages, scale ratios and viewpoint angles. While both methods perform well on high-overlap pairs (80-100%), AlligatOR trained Cub3-all maintains strong performance
as overlap decreases, whereas CroCo's accuracy drops dramatically for pairs with less than 40% overlap. This demonstrates AlligatOR's ability to generalize across wide range of geometric configurations, which crucial for real-world applications where overlap cannot be guaranteed.
Fine-tuning Strategy Comparison. We also compare different fine-tuning strategies, specifically impact of freezing versus unfreezing the backbone during fine-tuning- One can see that AlligatOR trained on Cub3-all is already quite high with frozen backbone and always outperforms CroCo even when backbone is unfrozen. This suggests that our pre-training strategy is better aligned with the downstream pose regression task.

---

100
AlligatOR,
AlligatOR, Cub3-50
Success Rate (%)
CroCo Cub3-all
CroCo Cub3-50
5-23
20-40 42-60 60-80
92-120
1-11
15-23
23
4.8
0-30 20
60-120
Overlap (%)
Scale ratio
Viewpoint angle (*)
Figure 5: Performance of CroCo and AlligatOR across different geometric challenges RUBIK. Results show accuracy at the 5 threshold for models trained on different datasets (Cub3-50 Cub3-all) and for frozen backbones. AlligatOR trained Cub3-all consistently outperforms other configurations, particularly for challenging cases with low overlap. large scale differences, and extreme viewpoint changes.
Comparison with Training from Seratch. To isolate the impact of our pre-training approach, we compare the performance of model trained from scratch for relative pose regression versus those initialized with AlligatOR or CroCo pre-training. The bottom row of scratch achieves notably lower performance, highlighting the value of the representations learned during pre-training.
that training from
Robustness to Label Noise. To address concerns about sensitivity to covisibility segmentation errors, conducted sensitivity analysis by training AlligatOR ScanNet with 20% random label noise injected during pre-training. Surprisingly, this actually improved performance across all thresholds (+2.2%, +2.4%, +0.8% respectively). shown in Table 2 This counterintuitive result suggests that label noise acts as form of regularization similar to label smoothing (38). encouraging learning of more generalizable features. Importantly, this demonstrates that our approach remains robust even when covisibility annotations are imperfect.
Table 2: Sensitivity analysis: Impact of label noise on AlligatOR performance on Sc0nNet1 Results show that 20% random label noise during pre-training actually improves performance, demonstrating robustness annotation errors.
SeanNet1500
Method
Noise
10 00.25m
10" /0.5m
10" /1
AlligatOR
0%
85.5
92.5
95.1
AlligatOR
20%
87.7
94.9
95.9
CroCo
N/A
75.7
87.4
91.5
Table 3: RUBIK Benchmark. The results reported from 28 Success rate (in %) for each method across individual geometric criterion bins. Best and second-hest values for each column are shown in bold and underlined respectively.
Ovarlp (5)
Scale Raiio
Vicwpoint Angle (")
Whole Dataset
Time (ms)
80 100
60
30
40 60
20
40
5:20
10 15
1,5 2.5
2.5
4.0
4.0
6.0
0:30
30 60
60
120
120
180
Detector free Wethodd
LAFTR E7)
$7.2
85.4
.7.2
17.5
5.0
51.6
10.1
233
0.6
43.2
27.9
15.1
0.0
24.9
185
BLaFTR 22
56.4
90.3
50.8
22.1
6.3
51.2
15.6
4.4
0.7
42.2
308
18.2
0.1
26.6
124
Formnrrreer
72.2
72.3
44.5
21.9
7.4
46.0
14.9
59
1.6
42.5
27.2
16.0
0.1
24.8
108
RoMa [3
67.0
98.3
84.5
52.7
71.2
56.2
3.0
DUSISR
30.4
55.3
Alligattit (Ours)
100L0
$1.3

---

Reference
Target
Masked
CroCo recons.
AlligatOR segmentations
Figure 6: Qualitative comparison. CroCo's reconstructions are blurred in masked non non-covisible regions, while AlligatOR often correctly identifies covisible, occluded, outside FOV regions across varying degrees of overlap and viewpoint changes.
5.3 Qualitative Analysis
Covisibility and Reconstruction Visualization. Figure/b]shoss qualitative examples the covisibil- ity segmentation and CroCo reconstructions produced by models pairs from Cub3. AlligatOR successfully identifies covisible regions, occluded areas, and regions outside FOV across varying degrees of overlap and viewpoint changes. These visualizations provide insights into the geometric understanding acquired by our model. Conversely, while Croco succeeds in reconstructing masked covisible regions, the reconstructions are blurred in masked non-covisible regions cross-completion ill-posed in such areas.
5.4 Comparison with state-of-the-art on RUBIK
In Table we evaluate the performance AlligatOR fine-tuned for metric relative pose estimation against state-of-the-art methods on RUBIK 28 Let us highlight that:
AlligatOR was trained nuScenes (on scenes that not part of RUBIK),
whereas none of the state-of-the-ard methods used nuScenes as training set.
This gives advantage AlligatOR over the other methods. However, RUBIK allows the evaluation over large ranges of three geometric criteria- overlap. scale ratio and viewpoint angle- which we found interesting. Despite this advantage, argue that AlligatOR's 's performance remains remarkable, as pretraining with CroCo, in contrast, leads to a significant performance drop (see Table I].
With an overall success rate of 60.3%, AlligatOR ranks first. Notably, it significantly outperforms all other methods on the hardest bin across the three geometric criteria, which confirms the advantage of its pretraining challenging pairs (see Figure 4. one might expect a performance drop on easier bins. However, this does not happen: for instance, AlligatOR still ranks first for small scale ratios and small viewpoint angles. Finally, AlligatOR is among the fastest methods, it directly regresses the relative pose rather than producing intermediate correspondences.
difficult image pairs. While AlligatOR's training set was highly skewed toward

---

5.5 Out-of-Domain Generalization
To demonstrate the generalization capabilities of our method beyond the training domains, we evaluate AlligatOR on zero-shot correspondence estimation using the ETH3D dataset [34]. We use the correlations from decoder features estimate correspondences and measure performance using Average EndPoint Error (AEPE) as described in M.
Zero-shot Correspondence Estimation. As detailed in the supplementary material, AlligatOR consistently outperforms CroCo when trained on the same datasets, and even surpasses CroCo v2 which was vre-traine on 5 datasets. This demonstrates that our covisibility segmentation pre-training learns more generalizable features for dense correspondence tasks.
Comparison with Official CroCo v2. We also compare against the official CroCo v2 model, fine-tuned Cub3-all dataset for pose regression. As shown in Table 4 AlligatOR signifi- cantly outperforms CroCo v2 across all metrics, demonstrating the effectiveness of our covisibility segmentation approach even when compared a model pre-trained on multiple datasets.
Table Comparison with official CroCo v2 fine-tuned on Cub3-all dataset for pose regression on RUBIK benchmark. AlligatOR significantly outperforms CroCo v2 across all metrics.
RUBIK
Method
5 0.55m
59 2m
10" 5m
AlligatOR (Frozen)
24.0
55.3
82.3
CroCo v2 (Frozen)
5.8
15.8
34.1
AlligatOR (Unfrozen)
24.6
60.3
81.9
CroCo v2 (Unfrozen)
14.6
44.5
73.1
6
Limitations
While our approach demonstrates significant improvements over CroCo, it has several limitations:
First, our current evaluation is limited to urban driving scenes from nuScenes and indoor scenes from ScanNet. The performance of AlligatOR on more diverse environments, such as natural scenes extreme weather conditions, remains to be investigated.
Second, our pre-training method requires ground truth depth maps and poses to produce dense covisibility annotations. While we investigated fully automated pseudo ground truth generation for nuScenes, such pre-processing computationally intensive. This dependency on 3D data might limit the scalability of our approach to scenarios where such data is not readily available.
Finally, while we show promising results on metric relative pose regression, the effectiveness of our pre-training approach for other binocular vision tasks, such stereo matching optical flow estimation, remains to be explored.
7 Conclusion and Future Work
We introduced AlligatOR, a novel pre-training approach that replaces cross-view completion with covisibility segmentation. Unlike CroCo's masked reconstruction, our approach explicitly models covisibility relationships, enabling more effective learning from challenging geometric configurations.
We demonstrated significant improvements zero-shot correspondence estimation, particularly on challenging scenes with limited overlap. Our approach's 's robustness geometrically challenging examples and superior generalization capabiliities highlight the effectiveness covisibility segmentation for binocular vision tasks.
CroCo metric relative pose regression and
We created Cub3, a large-scale dataset of 5M image pairs with dense covisibility annotations from nuScenes and ScanNet. Our work demonstrates that explicitly modeling covisibility provides an effective alternative to binocular vision tasks compared reconstruction-bint based methods.
10

---

Acknowledgments
This project has received funding from the Bosch Research Foundation (Bosch Forschungsstffungg) the European Union (ERC Advanced Grant explorer Funding ID #101097259), and was granted access to the HPC resources of IDRIS under the allocation 2024-AD011014905R made by GENCI.
References
[1] Honggyu An, Jin Hyeon Kim, Seonghoon Park, Jaewoo Jung, Jisang Han, Sunghwan Hong, and Seungryong Kim. Cross-view completion models are zero-shot correspondence estimators. In Proceedings of the Computer Vision and Puttern Recognition Conference, pages 1103-1115, 2025.
[2] Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. BEit: BERT pre-training of image transformers. In International Conference on Learning Representations, 2022.
[3] Guillaume Bono, Leonid Antsfeld, Boris Chidlovskii, Philippe Weinzaepfel, and Christian Wolf. End-to-end (instance)-image goal navigation through correspondence as emergent phenomenon. arXiv preprint arXiv:2339.16634 2023.
[4] Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan. Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11621-11631, 2020.
[5] Mathilde Caron, Hugo Touvron. Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski. and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the JEEE/CVF international conference on computer vision, pages 6650-96600 2021.
[6] Hongkai Chen, Zixin Luo, Lei Zhou, Yurun Tian, Mingmin Zhen, Tian Fang, David Mckinnon, Yanghai Tsin, and Long Quan. Aspanformer: Detector-free image matching with adaptive span transformer. In European Conference on Computer Vision, pages 20-36. Springer, 2022.
[7] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. simple framework for contrastive learning of visual representations. In International conference machine learning, pages 1597-1607. PmLR, 2020.
[8] Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. Proceedings of the IEEE conference computer vision and pattern recognition, pages 5828 5828-5839, 2017.
[9] Siyan Dong, Shuzhe Wang, Shaohui Liu, Lulu Cai, Qingnan Fan, Juho Kannala, and Yanchao Yang. Reloc3r: Large-scale training of relative camera pose regression for generalizable, fast, and accurate visual localization. arXiv preprint arXiv:2412.08370 2024.
[10] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 1929, 2020.
[11] Johan Edstedt, Ioannis Athanasiadis, Märten Wadenbäck, and Michael Felsberg. Dkm: Dense kemnliied feature matching for geometry estimation. Proceedings of the IEEE/CVF Confer- ence Computer Vision and Pattern Recognition, pages 17765-17775, 2023.
[12] Johan Edstedt, Georg Bökman, and Zhenjun Zhao. Dedode v2: Analyzing and improving the dedode keypoint detector. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 4245 4245-4253, 2024.
[13] Johan Edstedt, Qiyu Sun, Georg Bökman, Marten Wadenbäck, and Michael Felsberg. Roma: Robust dense feature matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19790-19800, 2024.

---

[14] Sven Elfiein, Qunjie Zhou, Sérgio Agostinho, and Laura Leal-Taixé Light3r-sfm: Towards ferd-forward structure-front cocuurre-frommmee mootoonnion. arXiv preprint TrXiv:2501.19114 2025.
[15] Miao Fan, Mingrui Chen, Chen Hu, and Shuchang Zhou. Occ 2net: Robust image matching based on 3d occupancy estimation for occluded regions. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9652-9662, 2023.
[16] Hugo Germain, Vincent Lepetit, and Guillaume Bourmaud. Visual correspondence hallucination. In International Conference on Learning Representations, 2022.
[17] Pierre Gleize, Weiyao Wang, and Matt Feiszli. Silk: Simple learned keypoints. In Proceedings of the IEEE/CVF international conference on computer vision, pages 22499-22508, 2023.
[18] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self- oof-suuerriied learning, Advances in neural information processing systems,
2020.
[19] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16000-16009, 2022.
[20] Vlad Hondru, Florinel Alin Croitoru, Shervin Minaee, Radu Tudor Ionescu, and Nicu Sebe. Masked image modeling: A survey. arXiv preprint arXiv:2408. 06688, 2024.
[21] Will Hutchcroft, Yuguang Li, Ivaylo Boyadzhiev, Zhiqiang Wan, Haiyan Wang, and Sing Bing Kang. Covispose: Co-visibility pose transformer for wide-baseline relative pose estimation in 360 indoor panoramas. In European Conference Computer Vision, pages 615-633. Springer, 2022.
[22] Wei Jiang, Eduard Trulls, Jan Hosang, Andrea Tagliasacchi, and Kwang Moo Yi. Cotr: Correspondence transformer for matching across images. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6207-6217, 2021.
[23] Linyi Jin, Richard Tucker, Zhengqi Li, David Fouhey. Noah Snavely, and Aleksander Holyn- ski. Stereo4d: Learning how things move in 3d from internet stereo videos. arXiv preprint arXiv:2412.09621, 2024.
[24] Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-tas talki-taskkku learning using uncertainty to weigh losses for scene geometry and semanties. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7482-7491, 2018.
[25] Shinjeong Kim, Mare Pollefeys, and Daniel Barath. Learning to make keypoints sub-pixel accurate. European Conference on Computer Vision, pages 413-431, Springer, 2025.
[26] Vincent Leroy, Yohann Cabon, and Jérôme Revaud. Grounding image matching in 3d with mast3r. arXiv preprint arXiv:2406.0977 2024.
[27] Philipp Lindenberger, Paul-Edouard Sarlin, and Mare Pollefeys. Lightglue: Local feature matching at light speed. In Proceedings the Vision, pages 17627-17633. 2023.
the IEEE/CVF International Conference on Computer
(28) Thibaut Loiseau and Guillaume Bourmaud. Rubik: A structured benchmark for image matching across geometric challenges. arXiv preprint arXiv:2502. 19955, 2025.
[29] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. Transactions on Machine Learning Research Journal, pages 1-31, 2024.
[30] Luigi Piccinelli, Yung-Hsu Yang, Christos Sakaridis, Mattia Segu, Siyuan Li, Lue Van Gool, and Fisher Yu. Unidepth: Universal monocular metric depth estimation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 10106-10116 16, 2024.
12

---

[31] Guilherme Potje, Felipe Cadar, André Araujo. Renato Martins, and Erickson R Nascimento. Xfeat: Accelerated features for lightweight image matching. Proceedings of the IEEE/CVF Conference Computer Vision and Patlern Recognition, pages 2682-26911 2024.
[32] Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. Superglue: Learning feature matching with graph neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1938-4947, 2020.
[33] Johannes L. Schonberger and Jan-Michael Frahm. Structure-from-mooee motion revisited. In Proceed- ings of the IEEE conference on computer vision and pattern recognition, pages 4104 4113, 2016.
[34] Thomas Schops, Johannes I. Schonberger, Silvano Galliani, Torsten Sattler, Konrad Schindler, Marc Pollefeys, and Andreas Geiger. A multi-view stereo benchmark with high-resolution images and multi-camera vidcos. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3260-32600000 3269, 2017.
[35] Brandon Smart, Chuanxia Zheng, Iro Laina, and Victor Adrian Prisacariu. Splatt3r: Zero-shot gaussian splatting from uncalibrated image pairs. arXiv preprint urXiv:2408. 0104408..3122 /3912, 2024.
[36] Zikai Song, Run Luo, Junqing Yu, Yi-Ping Phoebe Chen, and Wei Yang. Compact transformer tracker with correlative masked modeling. In Proceedings of the AAAI conference on artificial intelligence, pages
2023.
[37] Jiaming Sun, Zehong Shen, Yuang Wang, Hujun Bao, and Xiaowei Zhou. Loftr: Detector-free local feature matching with transformers. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 192228931,, 2021.
[38] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Re- thinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818 1822222, 2016.
[39] Dongli Tan, Jiang-Jiang Liu, Xingyu Chen, Chao Chen, Ruixin Zhang, Yunhang Shen, Shouhong Ding, and Rongrong Ji. Eco-tr: Efficient correspondences finding via coarse-to- refinement. In European Conference on Computer Vision, pages 317-334. Springer, 2022.
[40] Shitao Tang, Jiahui Zhang, Siyu Zhu, and Ping Tan. Quadtree attention for vision transformers. In International Conference on Learning Representations, 2022.
[41] Prune Truong, Martin Danelljan, Radu Timofte, and Luc Van Gool. Pde-net+: Enhanced probabilistic dense correspondence network. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(8): 10267-10266, 2023.
[42] Hengyi Wang and Lourdes Agapito. 3d reconstruction with spatial memory. arXiv preprint arXiv:2408. 16061, 2024.
[43] Qing Wang, Jiaming Zhang, Kailun Yang, Kunyu Peng, and Rainer Stiefelhagen. Matchformer: Interleaving attention in transformers for feature matching. In Proceedings of the Asian Conference on Computer Vision, pages 2746-2762, 2022.
[44] Qianqian Wang, Yifei Zhang, Aleksander Holynski, Alexei Efros, and Angjoo Kanazawa. Continuous 3d perception model with persistent state. arXiv preprint 1rXiv:2501.12335 2025.
[45] Shuzhe Wang, Vincent Leroy, Yohann Cabon, Boris Chidlovskii, and Jerome Revaud. Dustăr: Geometric 3d vision made easy. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 20697-20709, 2024.
[46] Yifan Wang, Xingyi He, Sida Peng, Dongli Tan, and Xiaowci Zhou, Efficient loftr: Semi-dense local feature matching with sparse-like speed. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages
2024.
[47] Philippe Weinzaepfel, Vincent Leroy, Thomas Lucas, Romain Brégier, Yohann Cabon, Vaibhav Arora, Leonid Antsfeld, Boris Chidlovskii, Gabriela Csurka, and Jérôme Revaud. Croco: Self-supervised pre-training for 3d vision tasks by cross-view view completion. Advances Neural Information Processing Systems, 35:3502-3516, 2022.
13

---

[48] Philippe Weinzaepfel. Thomas Lucas, Vincent Leroy, Yohann Cabon, Vaibhav Arora, Romain Brégier, Gabriela Csurka, Leonid Antsfeld. Boris Chidlovskii, and Jérôme Revaud. Croco v2: Improved cross-view completion pre-training for stereo matching and optical flow, In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 17969- 17980, 2023.
[49] Jianing Yang, Alexander Sax, Kevin Liang, Mikacl Henaff, Hao Tang, Ang Cao, Joyce Chai, Franziska Meier, and Matt Feiszli. Fast3r: Towards 3d reconstruction of 1000+ images in one forward pass. arXiv preprint arXiv:2501. 13928, 2025.
[50] Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything v2. arXiv preprint arXiv:2406.094414 2024.
[51] Botao Ye, Sifei Liu, Haofei Xu, Xueting Li, Mare Pollefeys, Ming- Haug- Yang, and Songyou Peng. No pose, no problem: Surprisingly simple 3d gaussian splats from sparse unposed images. arXiv preprint arXiv:2410.24207, 2024.
[52] Junyi Zhang, Charles Herrmann, Junhwa Hur, Varun Jampani, Trevor Darrell, Forrester Cole, Deqing Sun, and Ming-Hsuan Yang. Monst3r: A simple approach for estimating geometry in the presence of motion. arXiv preprint arXiv:2410.03825, 2024.
[53] Haojie Zhao, Dong Wang, and Huchuan Lu. Representation learning for visual object tracking by masked appearance transfer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 38696-18705, 2023.
[54] Xiaoming Zhao, Xingming Wu, Weihai Chen, Peter CY Chen, Qingsong Xu, and Zhengguo Li. Aliked: A lighter keypoint and descriptor extraction network via deformable transformation. IEEE Transactions on Instrumenaation and Measurement, 72:1-16, 2023.
[55] Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. Image BERT pre-training with online tokenizer. In International Conference on Learning Representations, 2022.
[56] Shengjie Zhu and Xiaoming Liu, Pmatch: Paired masked image modeling for dense geometric matching. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 21909-21918 2023.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The paper clearly states its main contribution a novel pretraining method for binocular vision that explicitly segments pixels covisible, occluded, or outside field-of- view, overcoming limitations of previous methods like CroCo. The claims are supported by experimental results in Section 5.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The paper acknowledges limitations in Section 6, discussing the scope of the method and potential areas for improvement.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
15

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: The paper is primarily focused on empirical results and does not contain theoretical proofs or theorems.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Section 5 (Experiments) provides detailed information about the experimental setup, including dataset details, training procedures, and evaluation metrics. Further details will also be in the supplementary material, and code and data will be made publicly available.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to model. In general. releasing code and data often one good way to accomplish this, but reproducibilityy can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the partieular way they provide for reproducibiliity In the case of closed-source models, it may be that access to the model is limited in
16

---

reproducible, they some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification:
The paper explicitly mentions code and data will be released in the abstract.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm cc/
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https //nips. wttpcccpppbblc.c hwppplllcciiiiieeeeeeeeppebeeeee guides/ (ooeeeumisssiono ssionPolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Section 5 provides comprehensive details about the experimental setup, includ- ing training parameters, data splits, and evaluation protocols.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Error bars are not reported in the paper because it would require additional compute resources.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
17

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
The paper provides detailed information about the experimental setup, in- cluding training parameters, data splits, and evaluation protocols, as well as the compute resources needed to reproduce the experiments.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: The research focuses on computer vision and pose estimation, with no apparent ethical concerns or violations of the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: The paper focuses on technical contributions to computer vision and does not explicitly discuss socictal impacts.
18

---

Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (c.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., denloymemt of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: The paper does not involve high-risk models or datasets that would require specific safeguards.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
The paper properly cites and credits all referenced works, including datasets like nuScenes and ScanNet, and methods like CroCo. We will add the licences of used assets in the supplementary material.
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
19

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [Yes]
Justification:
The paper explicitly mentions the release of new assets (code, datasets, or models).
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve any crowdsourcing or human subject research.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification; The paper does not involve any human subject research requiring IRB approval.
Guidelines:
20

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
The paper does not use LLMs as a component of its core methodology.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should not be described.
21

---

A Supplementary Material
This supplementary material provides additional details about AlligatOR model, including training curves, implementation details, and visualizations that complement the main paper.
A.1 Pre-training Learning Curves
Figure 7 shows the learning curves for the pre- pre-training phase both CroCo and AlligatOR on the Cub3-50 and Cub3-all datasets. AlligatOR's loss converges smoothly on both datasets, indicating stable training.
AlligatOR: Cub3-50
Dub3-50
2:50
AlligatOR Cub3-55
10-
CoCaa Dib3-50 Crota Dub3-all
AlligatoR: Cub3-all
DroCo/ Qub3-all
1.75
AlligatOR/ Cub3-all
1.50
LoSS
1.2
1.08
1.00
0.7
0.2
25
15
25
10
22
20
Epochs
Epochs
Epochs
Epochs
Figure 7: Learning curves during pre-training for CroCo (reconstruction loss) and AlligatOR (seg- mentation loss) Cub3-50 and Cub3-all datasets for nuScenes (left) and ScanNet (right). AlligatOR shows stable convergence on both datasets.
A.2 Fine-tuning Learning Curves
Figure presents the iinettunig curves for the pose regression task. The plots show the full loss during training (Eq. 9 in the main paper for AlligatOR, Eq. 8 for CroCo). AlligatOR fine-tuned on Cub3-all shows faster convergence and reaches higher accuracy than the same configuration for CroCo, highlighting the transferability of features learned through covisibility segmentation on difficult pairs. The loss increase 5 epochs corresponds to unfreezing the backbone. For AlligatOR, at 5 epochs we also add the segmentation loss to maintain interpretability.
AlligatoR/ /ub3350
CroCo / Cub3 50 CraCo Cub3-al
AlligetRR Cub3--50
Crote / Cub3-50
AlligatoR/ D-b3-all
3.5
AlligetRR/ / Cubikall
CroCo / Cub3-all
00-
1.2
10-
50-
Loss
33
20-
0.5
26-
-5.
2-
5.0
2.5
50
1. Epochs
15.1
120 12. 13.0 Epochs
200
75 25.0 Epochs
125
0.0
75
138
125
330
Epochs
Figure 8: Learning curves during fine-tuning for pose regression for nuScenes (left) and ScanNet (right). AlligatOR pre-trained on Cub3-all converges faster and achieves higher success rates than other methods, demonstrating the effectiveness of our covisibility segmentation pre-training approach.
A.3 Detailed Performance Analysis
Figure provides comprehensive breakdown of performance across different geometric criteria on RUBIK. This visualization emphasizes AlligatOR's ' strong performance across all difficulty ranges, particularly challenging scenarios where traditional methods struggle.
A.4 Implementation Details
We use a ViT-based encoder and transformer decoder backbone similar CroCo, with 24 layers for the encoder and 12 for the decoder. For pre-training, use the Adam W optimizer with a learning
22

---

C 56 1E.EE% < 22m
Frojection cue cowtap
Pnjedten
ection wngl
L25
axF
24
C2
S44
62
387
42
223
6
4.0
CA
64-
43
36
5:30
Col-
01.04
G
26
60-
13.2
474
10
1.0
#1
340
Creco (Cb3-si 6,3% 2mm
Profecion cvD cvalla
Pojedian cest ditanc
Plyeclin We enple
Ldi
242
TXD
AL
22
2L4
33
24
12
XX
333
MUN
Z4
372
TA
35
33
X>
II
22
22
A14
27
SCR
XA
21
R1
92.
Allgub3 /CUB3 SE: 24.0% < 511 2m
Projection Cuee cuwtap
0.05
Prgection over diewne
0.95
Pm Pefln overaaple
22
C2
62-
G2
A,1D
274
12
LND
)2
C4-
6.4
232
122
B1
X0:77
32
20
27
77.12
a
G
16
6a-
100
60.00
103
108
343
Falaiis
All gontkJ CLB2 al 55 28% / 2m
213
223
AL4
KC3
KCB
6:72
662
4
21.77
64-
Nestin
204
614
Gd-
CA.
0/
125
3.0
21 330
1:0
25
Figure 9: Detailed breakdown of performance across different geometric criteria. Success rate either for @ or 1@2m (bottom-left and top-right of each triangle, respectively), when projecting results onto individual geometric criteria of RUBIK. For each method, we show three plots corresponding to the projection over overlap (left), scale ratio (middle), and viewpoint angle (right).
rate of 1.5e-4, weight decay of 0.05, and a batch size of 32 per GPU. We employ a cosine learning rate schedule with 2 epochs of warmup and train for 25 epochs on our Cub3 datasets.
For fine-tuning relative pose regression, we follow the two-phase approach described in Section 3.3 of the main paper:
Phase 1: Freeze the backbone and train only the pose regression head for 5 epochs with a learning rate of lc-4
23

---

Phase 2: Unfreeze the entire network and jointly train with both the pose regression loss and covisibility segmentation loss for an additional 10 epochs with a learning rate of 5e-5
The comprchensive architecture of AlligatOR is illustrated in Figure 2 of the main paper. showing both the pre-training and fine-tuning phases. During pre-training. the model learns to segment pixels in each view covisible, occluded, outside the field of view with respect to the other view. During fine-tuning, we introduce a pose regression head while maintaining the segmentation capability to leverage the geometric understanding acquired during pre-training.
A.5 Data Splits and Sampling Protocol
To ensure proper evaluation and avoid data leakage, we carefully separate training and test data:
Data Splits:
Uses test scenes from nuScenes, while our Cub3 dataset uses only the training split of nuScenes, with completely disjoint scenes.
RUBIK benchmark:
ScanNet1500 benchmark: Derived from the standard ScanNet test split, whereas Cub3 uses scenes from the ScanNet training split exclusively.
Sampling Protocol:
We pre-filter all possible pairs from all training scenes with at least 5% overlap for the "all" version of Cub3, and at least 50% for the "50" version. We then sample from all those pre-filtered pairs to extract the desired number of samples and covisibility masks. The RUBIK benchmark creation follows the protocol described in [28) and is extracted from test scenes, while ScanNet1500 was created from test scenes of ScanNet as first introduced in [32].
A.6
Additional Ablation Studies
A.6.1 Impact of the Number of Classes
We implemented two variants of AlligatOR to investigate the importance of our pre-training with the three classes (covisible, occluded, outside FOV). We tried pre-training with only two classes, either covisible or not (in this case occluded and outside FOV are merged), or inside FOV or not (in this case, covisible and occluded merged) Cub3-all for nuScenes. The results the RUBIK benchmark are presented in Table 5
Table 5: Results on RUBIK for metric relative pose regression when pre-training with only two classes. For all experiments, pre-training and fine-tuning is performed using Cub3-all.
RUBIK
Classes
5 / 00.5m
5 / 2m
10 /5m
Covisible or not Inside FOV or not All 3 classes
22.5
55.7
80.0
24.6
59.6
81.9
24.6
60.3
81.9
While the performance improvement with three classes is modest, we believe that the model's knowledge about occluded regions could be beneficial for other tasks. As noted the main paper, the annotated maps may contain noise between covisible and occluded zones due reliance monocular depth predictions.
A.6.2 Non-metrie Relative Pose Regression
We investigated whether our pre-training is useful for non-metric relative pose regression (regressing only the angle for translation) and compared results with CroCo pre-training the S500 benchmark. For this experiment, we changed our pose regression head using the one from Reloc3r, along with its loss function. The results are shown in Table
AlligatOR significantly outperforms CroCo on this task, demonstrating the versatility of our pre- training approach.

---

Table 6: Results on S50 for non-metric relative pose regression. For AlligatOR, pre-training and fine-tuning performed using Cub3-all, whereas for CroCo, pre-training performed using Cub3-50 and fine-tuning using Cub3-all.
ScanNet1500
Pre-Training
AUC@5
AUC@ 10
AUC@20
CroCo AlligatOR
13.2
34.1
57.1
20.5
43.9
66.2
A.7 Zero-shot Correspondence Estimation ETH3D
To demonstrate the generalization capabilities of our method beyond the training domains, we evaluate AlligatOR on zero-shot correspondence estimation using the ETH3D dataset (34). We use correlations from decoder features to estimate correspondences and measure performance using Average EndPoint Error (AEPE) described in
Table 7: Zero-shot correspondence estimation on ETH3D dataset using AEPE. AlligatOR consistently outperforms CroCo variants, demonstrating better generalization out-of-domain dense matching tasks.
Method
Training Data
AEPE (L)
AlligatOR
nuScenes-all
43.82
CroCo
nuScenes-all
77.61
AlligatOR
nuScenes-50
44.12
CroCo
nuScenes-50
92.98
AlligatOR
ScanNet-all
36.07
CroCo
ScanNet-all
56.70
AlligatOR
ScanNet-50
38.45
CroCo
ScanNet-50
86.60
CroCo v2
5 datasets
51.55
The results in Table show that AlligatOR consistently outperforms CroCo when trained on the same datasets, and even surpasses CroCo v2 which was pre-trained on 5 datasets. This demonstrates that our covisibility segmentation pre-training learns more generalizable features for dense correspondence tasks.
A.8 Additional Qualitative Results
We provide additional visualizations from our nuScenes and ScanNet datasets in Figures 10 and 11 along with more predictions from AlligatOR and CroCo in Figure [2
25

---

TIER
Figure 10: Covisibility annotation examples from Cub3 for nuScenes. For each image pair, we show the corresponding covisibility maps with color-coding for covisible, occluded, and outside FOV regions. Note how our annotation process handles varying degrees of overlap and challenging viewpoint changes. Let highlight that some annotations, particularly the distinction between covisible and occluded pixels, may contain noise, especially for nuScenes, and we demonstrate the experiments that AlligatOR is highly robust to this noise.
26

---

Figure 11: Covisibility annotation examples from Cub3 for ScanNet. For each image pair, show the corresponding covisibility maps with color-coding for covisible, occluded, and outside FOV regions. Note how our annotation process handles varying degrees of overlap and challenging viewpoint changes.
27

---

Reference
Target
Masked
CroCo recons.
AlligatOR segmentations
Figure 12: Qualitative comparison. CroCo's reconstructions are blurred in masked non-covisible regions, while AlligatOR often correctly identifies covisible, occluded, outside FOV regions across varying degrees of overlap and viewpoint changes.
28


Reviews:
[{"review_id": "LgpakB0g7j", "reviewer": "Reviewer_izzw", "summary": "The paper presents a pre-training scheme for 3D reconstruction and pose regression. The proposed scheme is based on covisibility segmentation where each pixel in one image is predicted as being covisible, occluded or outside the field of view in the second image. This is in contrast to an existing pretraining approach termed CroCro that is based on cross-view completion of image regions in a pair of images. The paper also presents a 5M image-pair dataset with dense covisibility annotations for training and validation of the proposed scheme.", "questions": "How sensitive is the proposed scheme to errors in covisibility segmentation?", "limitations": "In addition to the limitations mentioned by the authors, the dependence/sensitivity of the proposed technique on/to errors in segmentation should be discussed.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths: The proposed scheme is conceptually simple and intuitive. Unlike the CroCro scheme it does not require a high degree of overlap between the image pairs which is needed for cross-view completion. The proposed scheme can be trained on both, covisible and non-covisible regions in the image pairs unlike CroCro that requires training on regions that are covisible in both images in the image pair. Experimental results show that he proposed scheme performs better than CroCro on most cases in terms of pose regression accuracy and generalizability. The proposed scheme also shows higher pose regression accuracy compared to techniques that do not employ pre-training. \n\nWeaknesses: The proposed scheme would be expected to be sensitive to covisibility segmentation error. This point needs to be discussed accompanied by a formal sensitivity analysis. The notations in the cross-entropy loss function formulation (page 4, line 112) are confusing and need to be clarified/corrected.", "was_revised": true, "final_justification": "The authors have addressed my concerns regarding the sensitivity analysis in their rebuttal. Hence I retain my original decision to accept the paper (rating 5).", "rebuttal": "We thank the reviewer for their thorough and constructive feedback. We appreciate the recognition of our method's conceptual simplicity and intuitive design. Below, we address the specific concerns raised.\n\n## **Sensitivity to covisibility segmentation errors**\n\nWe agree that a proper sensitivity analysis is important. To address this, we are currently training an additional instance of Alligat0R on ScanNet, a dataset with high-quality covisibility labels, while injecting label noise by randomly altering labels with a 20% probability.\n\nUnfortunately, the training job is still running on our supercomputing cluster — due to heavy usage during the NeurIPS rebuttal period, it is progressing slower than usual (the full run takes 20 hours for pre-training, and about 5 hours for fine-tuning). We aim to complete the experiment and populate the table below early during the discussion phase:\n\n|  |  |  | ScanNet1500 | |\n|:----------|:-------------:|:-------------:|:-------------:|:-------------:|\n| **Method** | **Noise** |  10° / 0.25m | 10° / 0.5m | 10° / 1m |\n| Alligat0R | 0% | **85.5** | **92.5** | **95.1** | \n| Alligat0R | 20% |  |  |  |\n| CroCo | NA | 75.7 | 87.4 | 91.5 |\n\nWe expect this experiment to demonstrate Alligat0R's robustness to segmentation noise, for the following reasons:\n\n- As discussed in lines 165–171 (Section 4.1) and illustrated in Figure 3 of the main paper and Figure 4 in the Supplementary Material, the model trained on nuScenes already operates with noisy covisibility masks, yet achieves state-of-the-art performance on RUBIK, significantly outperforming CroCo. \n- Additionally, Table 1 in the Supplementary Material shows that collapsing our three covisibility classes into two (i.e., merging categories) has minimal performance impact, indicating robustness to label ambiguity and class boundary errors.\n\n\n## **Cross-entropy loss notation clarification**\n\nWe agree that the current notation is unclear. We will revise and clarify it in the camera-ready version."}, {"review_id": "KPx7ZupiJP", "reviewer": "Reviewer_MqH9", "summary": "This paper introduces Alligat0R, a pre-training method for binocular vision that departs from the established cross-view completion paradigm. The authors identify a limitation in prior work like CroCo: the task of reconstructing masked pixels is ill-posed for regions that are not mutually visible (i.e., occluded or outside the field of view). To address this, Alligat0R reframes the pre-training objective as a covisibility segmentation task. The model is trained to predict, for each pixel in one image, whether its corresponding 3D point is covisible, occluded, or outside the field of view in the second image. To support this new pre-training task, the authors have created Cub3, a large-scale dataset of 5 million image pairs with dense covisibility annotations, derived from the nuScenes and ScanNet datasets. The paper demonstrates through extensive experiments that pre-training with Alligat0R leads to significantly better performance on the downstream task of relative camera pose regression compared to a CroCo baseline, especially in challenging scenarios with low image overlap.", "questions": "1. Is it possible to use Dust3R/VGGT to provide pseudo depth and depth labels, and hence to generate Covisibility masks? Given that the Covisibility masks for nuScenes were generated by monocular depth estimation methods such as DepthAnything, the quality of Dust3R/VGGT should be sufficient for this case. (Dynamic pixels may be a potential problem, though)\n\n2. Do the authors have any other plan or idea to simplify the generation of Covisibility masks?\n\n\nThe main concerns is about Point 1 in Weaknesses, i.e., pre-training data requirements. The reviewer would be happy to change the rating if the authors could address these issues.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths: \n\n1. The core idea is simple yet seems powerful. The paper correctly identifies a weakness in the cross-view completion pre-training task (e.g., CroCo), its inability to handle non-covisible regions gracefully. The proposed solution of replacing an ill-posed reconstruction task with a well-defined, multi-class segmentation task directly addresses the identified problem. \n\n2. The results are compelling. The head-to-head comparison with CroCo under identical conditions (Table 1) shows an impressive performance gain for Alligat0R. \n\n\nWeaknesses:\n\n1. The primary weakness of the method lies in its pre-training data requirements. Unlike self-supervised methods that can learn from raw image pairs, Alligat0R requires dense covisibility labels. Generating these labels necessitates access to ground-truth or high-quality estimated 3D information (camera poses and depth maps). Although the authors are transparent about their complex data generation pipeline for nuScenes, this dependency on 3D supervision limits the scalability of the pre-training to datasets where such geometric information is available, and it moves the method away from a purely self-supervised paradigm.\n\nA potential counter-point here is, Croco also requires covisibility in order to filter out image pairs with small overlap. However, it is worth noting that even without filtering, Croco's paradigm still works.\n\n\n2. The paper convincingly demonstrates the benefits of Alligat0R for relative pose regression. However, as mentioned in the limitation of this paper, the broader claim of pre-training for \"binocular vision\" would be strengthened by evaluating on other fundamental tasks like stereo matching or optical flow. This is important as Croco has shown the results on these tasks. It remains a question whether the features learned for covisibility segmentation are as effective for these dense correspondence tasks as they are for global pose estimation.", "was_revised": true, "final_justification": "Most of my concerns have been addressed in the rebuttal and hence I raised my rating. I recommend that the authors include these explanations in the final version of the paper", "rebuttal": "We thank the reviewer for their detailed assessment and recognition of our results. Below, we address the primary concern regarding pre-training data requirements, along with other comments.\n## **Main concern: Pre-training data requirements and scalability**\nWe agree with the reviewer that our method requires covisibility labels, which can pose scalability challenges. However, we would like to emphasize that such labels can be obtained in three practical ways:\n1. Synthetic datasets\n2. Datasets captured using 3D sensors (e.g., ScanNet)\n3. Datasets requiring SfM (e.g., nuScenes)\n\nRecent large-scale models such as MASt3R and VGGT leverage combinations of all three types. For example, VGGT training data consists of 59% synthetic, 24% 3D-captured, and only 18% SfM-derived data (see VGGT slides, CVPR 2025).\n\nWe acknowledge that SfM pipelines like COLMAP can be computationally intensive and require significant engineering, which we agree departs from the principles of pure self-supervision. However, this limitation can be mitigated through:\n\n- Synthetic and 3D-captured datasets, where covisibility labels are trivial to extract. Indeed, we also trained Alligat0R on ScanNet, which uses 3D sensors, and obtained strong results.\n- Reducing SfM cost via modern alternatives, such as VGGT (as suggested by the reviewer). COLMAP takes ~20 minutes for a 200-image scene, while VGGT can generate camera poses and dense depth maps in ~9 seconds for the same data, a 130× speedup. Moreover, VGGT offers potential quality gains by leveraging context from all frames, as opposed to our current pipeline where depth prediction is done independently per image and then aligned. We believe this trend will continue to improve with advances like VGGT-Long, making the creation of large covisibility-annotated datasets increasingly feasible.\n\n## **Other plans to simplify covisibility mask generation?**\n\nOne potential direction is to use dense matching models that would output two possible labels: covisible or not. From Table 1 in the Supplementary Material, we show that having only two classes works well for the task of pose regression as the model has to be aware of the covisible zones in order to match them and make a prediction. However, for other dense tasks such as optical flow or pointmap regression for example, we believe that the \"occluded\" class gives a stronger 3D inductive bias that would help solving those particular tasks.\n\n## **Broader task evaluation acknowledgment**\n\nWe agree that framing this work as pre-training for \"binocular vision\" is important.  To showcase the generalization of our method, we decided to use both our CroCo and Alligat0R models pre-trained on either nuScenes or ScanNet, and evaluate them for the task of zero-shot matching using the correlations from the decoder features on the ETH3D dataset. The metric we use is the Average EndPoint Error (AEPE), computed as the averaged Euclidian distance between estimated and ground-truth flow fields over all valid target pixels (please see *Cross-View Completion Models are Zero-shot Correspondence Estimators*, CVPR 2025). The results are shown in the table below:\n\n|  |  | ETH3D |\n|:----------|:-------------:|:-------------:|\n| **Method** | **Training Data** | **AEPE $\\downarrow$** |\n| Alligat0R | nuScenes-all | **43.82** |\n| CroCo | nuScenes-all | 77.61 |\n| Alligat0R | nuScenes-50 | $\\underline{58.96}$ |\n| CroCo | nuScenes-50  | 92.98 |\n|  |  |  |\n| Alligat0R | ScanNet-all | **36.07** |\n| CroCo | ScanNet-all  | 56.70 |\n| Alligat0R | ScanNet-50 | $\\underline{40.25}$ |\n| CroCo | ScanNet-50 | 86.60 |\n|  |  |  |\n| CroCo v2 | 5 datasets | 51.55 |\n\nAs we can see, when trained on the same dataset, Alligat0R always outperforms CroCo for this task, and even beats CroCo v2, showcasing the generalization of our method. An interesting finding is that CroCo trained on all overlaps outperforms its counterpart trained on at least 50% overlap. We will add those results in the camera-ready version."}, {"review_id": "f2lVpWBHiS", "reviewer": "Reviewer_uMs9", "summary": "The authors propose a new multi-view pre-training paradigm for 3D based on covisibility segmentation.\nThe key contribution is to replace the inpainting objective of CroCo with a cross-entropy loss over segmentation labels (non-covisible, covisible, occluded).\nTo pre-train the model they collect a dataset of 5M pairs from the nuScenes and ScanNet datasets.\nFinally they introduce a relative-pose head that directly predicts a quaternion and metric translation from their backbone features.\nThe authors show that their approach leads to better relative pose estimates on the nuScenes and ScanNet datasets when compared both to CroCo but also to other SotA methods.", "questions": "1. It was not entirely clear to me how the pairs are sampled for ScanNet and nuScenes. Are they sampled uniformly from all scenes except the test scenes in RUBIK and ScanNet1500? If the authors could provide more details here I would be grateful.\n\nIf evaluation was conducted on a benchmark other than the training sets, I would likely increase my score to accept (even if the method is not SotA).", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 4, "strengths_and_weaknesses": "**S1 - Elegant objective:** Masked inpainting objectives such as croco have multiple issues. One is that they require masking patches at training time, which is misaligned with how the model will be used in downstream tasks. The second is that if there is no overlap it typically performs poorly. In contrast the proposed objective is very simple, and faces fewer such issues.\n\n**S2 - Strong results:** The results are in general strong, beating out CroCo with a good margin. When finetuned for relative pose they also beat out MASt3R in most cases. The caveat here is that the benchmarks are only conducted on their training datasets, which I go into more detail in W1.\n\n\n**W1 - Unclear generalization:** All benchmarks in the paper are conducted on the training datasets. While I assume (although I could not actually find this in the paper) that they have excluded these benchmarks from the training data, the fact is that the chosen datasets are not very diverse, and there are likely very similar scenes in the training data as in the test. One could for example investigate map-free relocalization as an alternative benchmark, as the task there is metric pose prediction, to investigate whether the model generalizes. As it stands I think including at least one benchmark on an unseen dataset is vital.\n\n**W2 - Missing/non-standard evaluations:** I was dissapointed to find that there were no comparisons to previous direct relative pose estimation methods, such as Reloc3r. Besides this, the authors have chosen metrics which do not align with the standard AUC metrics in the field, which makes comparisons difficult. I understand that metric relative pose can draw advantage in measuring metric translation errors, instead of angular, but not including any comparisons (except to CroCo) makes this problematic.", "was_revised": true, "final_justification": "The rebuttal clarified my questions regarding the data splits, and results on a benchmark outside of the training domain.\nThis, as well as the discussion with the other reviews (showing robustness to label noise, etc), is sufficient for me to be confident in this paper, and I have thus increase my score from 4->5.", "rebuttal": "We thank the reviewer for recognizing the elegance of our objective and the strength of our results. Below, we address the concerns related to generalization and evaluation methodology.\n\n## **Generalization beyond training datasets**\n\nWe would like to clarify that our evaluations are conducted on unseen data, and there is no scene overlap between the training and test sets:\n\n- RUBIK uses test scenes from nuScenes, while our Cub3 dataset uses only the training split of nuScenes, with disjoint scenes.\n- ScanNet1500 is derived from the standard ScanNet test split, whereas Cub3 uses scenes from the ScanNet training split exclusively.\n\nThus, both RUBIK and ScanNet1500 represent genuine generalization benchmarks in terms of scene disjointness. That said, we acknowledge that the test scenes are from domains that are similar to those seen during training (i.e., same dataset distributions), which may limit out-of-domain generalization.\n\nWe also provide non-metric pose regression results in Table 2 of the Supplementary Material, showing strong generalization:\n\n- Alligat0R achieves 20.5 AUC@5 on ScanNet (non-metric)\n- Compared to CroCo, which achieves 13.2 AUC@5\n\nFurthermore, to showcase the generalization of our method, we decided to use both our CroCo and Alligat0R models pre-trained on either nuScenes or ScanNet, and evaluate them for the task of zero-shot matching using the correlations from the decoder features on the ETH3D dataset. The metric we use is the Average EndPoint Error (AEPE), computed as the averaged Euclidian distance between estimated and ground-truth flow fields over all valid target pixels (please see *Cross-View Completion Models are Zero-shot Correspondence Estimators*, CVPR 2025). The results are shown in the table below:\n\n|  |  | ETH3D |\n|:----------|:-------------:|:-------------:|\n| **Method** | **Training Data** | **AEPE $\\downarrow$** |\n| Alligat0R | nuScenes-all | **43.82** | \n| CroCo | nuScenes-all | 77.61 |\n| Alligat0R | nuScenes-50 | $\\underline{58.96}$ |\n| CroCo | nuScenes-50  | 92.98 |\n|  |  |  |\n| Alligat0R | ScanNet-all | **36.07** | \n| CroCo | ScanNet-all  | 56.70 |\n| Alligat0R | ScanNet-50 | $\\underline{40.25}$ | \n| CroCo | ScanNet-50 | 86.60 |\n|  |  |  |\n| CroCo v2 | 5 datasets | 51.55 |\n\nAs we can see, when trained on the same dataset, Alligat0R always outperforms CroCo for this task, and even beats CroCo v2, showcasing the generalization of our method. An interesting finding is that CroCo trained on all overlaps outperforms its counterpart trained on at least 50% overlap. We will add those results in the camera-ready version.\n\n\n## **Data sampling protocol**\n\nWe pre-filter all possible pairs from all training scenes with at least 5% overlap for the \"all\" version of Cub3, and at least 50% for the \"50\" version. We then sample from all those pre-filtered pairs in order to extract the good number of samples and covisibility masks. The RUBIK benchmark creation is explained from the corresponding paper *RUBIK: A Structured Benchmark for Image Matching across Geometric Challenges*, CVPR 2025 and is extracted from test scenes, and ScanNet1500 benchmark was also created from test scenes of ScanNet and was first introduced in *SuperGlue: Learning Feature Matching with Graph Neural Networks*, CVPR 2020. We will add those details in the Supplementary Material."}, {"review_id": "NknHL2FhEn", "reviewer": "Reviewer_SujZ", "summary": "The paper defines a pretext task to pretrain a transformer network for geometric computer vision so that the resulting features are suitable for downstream task (e.g. relative pose regression demonstrated in the paper).\n\nGiven two images, the pretext task segments each image based on whether the pixel is visible / occluded / oustide of the other image.\nThe network is made of a self-attention encoder, cross-attentional decoder and a segmentation head.\n\nThe network is trained in a supervised manner i.e. each image pairs has an associated \"segmentation masks\" on two datasets: the indoor ScanNet1500 and the outdoor NuScenes dataset.\nThese datasets do not readily provide the \"segmentation\" masks used for training so the masks are generated automatically by leveraging the ground-truth data they provide (e.g. camera poses and depth maps, or camera poses and estimated depth maps).\n\nThe relevance of the pretraining is tested on the task of relative pose regression by augmenting the network with a pose regression head.\nThe network is evaluated on test splits of the ScanNet1500 and NuScenes datasets, which are standard pose estimation datasets.\nThe experiments compare the result between training the network from scratch, with the proposed pre-training and with another standard pre-training task Croco.\nThe results show that the proposed pre-training leads to the best performance, which support its relevance.\n\nThe contributions of the paper are:\n- the definition of a new pre-training task for 3D computer vision tasks\n- the pre-training task requires annotation and the paper contributes a training dataset derived from the existing NuScene and ScanNet1500\n- The experiments are sound and support the claim of the paper.", "questions": "- In the pose regression loss, how are the quaternion parametrized?\n\n- Semantically speaking, occluded and outside the field of view could be considered the same class. Was there any experiment pre-training on two classes only (covisible / not covisible) and was there any insight?", "limitations": "Yes, the authors have adequately addressed the limitations and potential negative societal impact of their work.\n\n\n### Misc. comments\n\nStrong or unjustified statements:\n- L25: \"Croco [43, 44] relies on pairs with at least 50% of covisible regions\". This is not a strict requirement but covisibility between [40,50]% leads to the best result as shown in Croco's ablation study.\n- L31-32: \"it [Alligator] aligns more [than Croco] directly with the correspondence reasoning required in downstream binocular vision task\". Cross view image completion implies that the network implicitly matches 1 pixel from the 1st image to its matching pixel in the 2nd image so it is also aligned with correspondence-based tasks.\n- L36: \" providing a more diverse training signal than previous approaches.\" This is hard to prove as it is hard to quantify a \"training signal\".\n- L90: \"Our symmetric design is not only more efficient but also better aligned with downstream binocular vision tasks, which typically process unmasked images.\" Why is it more efficient? Because there are less parameters to train? Why is it \"better aligned with downstream binocular vision tasks\"?\n\nMath typo:\n- Eq4: $c_{ij}$ should be outside the log\n\nConfusing writing:\n- \"confirming the results in the original CroCo paper\", which results?", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n\n- S1 -  The paper proposes a new pre-training task relevant for 3D computer vision task. Previous works have shown how complex yet important pre-training is in 3D vision so the contribution is useful to the community.\n- S2 - The paper derives a dataset to enable the proposed pre-training.\n- S3 - The paper is extremely well written and pleasant to read.\n- S4 - The pretraining and downstream task training are evaluated on both indoor and outdoor datasets.\n\n### Weaknesses\n\n- W1-  Some limitations in the experiments:\n  - a) There is no experiment with the off-the-shelf Croco or Croco v2 (trained on real indoor and outdoor images) with and without finetuning. There is no requirement for the Croco pretraining to be on the target dataset and such an experiment would be fairer to Croco as the Croco's author most likely tuned the network to get the best pretraining results.\n  - b) The downstream task is evaluated on the same dataset as the pre-training, which prevents appreciating whether the pre-training is generic, which is the case in Croco v1.\n  - c) The relevance of the pre-training is evaluated on a single downstream task (relative pose regression). Other tasks would have been relevant although it would have been more computationally demanding: monocular depth estimation, image matching, optical flow.\n\n- W2 - The formulation often argues that proposed pre-training \"better\" than Croco or addresses its limitation but it would be fairer to say that it is an alternative pretext task. For example:\n    - a) Croco's covisibility limitations (i): Croco is trained on image pairs with at least 50% overlap whereas Alligator can train on a pairs with covisibility ranging from [0,100]%. In practice, Croco can also be trained on the full range of covisibility but Croco's authors have run an ablation study showing that the pretraining is best when the covisibility ranges in [40,50]%. Raising the Croco's 50% covisibility requirement as a limitation would be sound only if similar study was run showing that Alligator is insensitive to the covisibility range.\n    - b) Croco's covisibility limitations (ii): the 50% covisibility requirement in Croco could be seen as a trade-off for the simpler self-supervision (Alligator requires annotation and at least ground-truth camera poses). Hence the suggestion to frame Alligator as an alternative pre-training method instead of a method better than croco.", "was_revised": true, "final_justification": "Thank you for the strong rebuttal, it is very much appreciated.\nThe rebuttal in general has addressed my concerns (data splits and datasets in the evaluation, the use of the pretraining for more tasks than relative pose regression and the framing of the contribution).\nThere can still be the argument that testing the proposed pretraining on more computer vision tasks would be useful, but the paper is already strong as is, and there is only so many experiments that can fit in a single paper, so the rating is raised to accept.", "rebuttal": "We thank the reviewer for their constructive feedback and address the main concerns below.\n\n## **Experimental limitations**\n\n**Off-the-shelf Croco v2 comparison:** We fine-tune the official CroCo v2 on our nuScenes Cub3-all dataset for pose regression and evaluate on RUBIK. We show the results below, keeping only the best Alligat0R model for comparison:\n\n|  |  |  | RUBIK |  |\n|:----------|:-------------:|:-------------:|:-------------:|:-------------:|\n| **Method** | **Backbone** | 5° / 0.5m | 5° / 2m | 10° / 5m |\n| Alligat0R | Frozen | $\\underline{24.0}$ | $\\underline{55.3}$ | **82.3** | \n| CroCo v2 | Frozen | 5.8 | 15.8 | 34.1 |\n| Alligat0R | Unfrozen | **24.6** | **60.3** | $\\underline{81.9}$ | \n| CroCo v2 | Unfrozen | 14.6 | 44.5 | 73.1 |\n\nWe can see that our method still outperforms the original CroCo v2 pre-trained on more datasets and fine-tuned for pose regression with our Cub3-all dataset, showcasing the strength of our method.\n\n**Other task evaluation:** To showcase the generalization of our method, we decided to use both our CroCo and Alligat0R models pre-trained on either nuScenes or ScanNet, and evaluate them for the task of zero-shot matching using the correlations from the decoder features on the ETH3D dataset. The metric we use is the Average EndPoint Error (AEPE), computed as the averaged Euclidian distance between estimated and ground-truth flow fields over all valid target pixels (please see *Cross-View Completion Models are Zero-shot Correspondence Estimators*, CVPR 2025). The results are shown in the table below:\n\n|  |  | ETH3D |\n|:----------|:-------------:|:-------------:|\n| **Method** | **Training Data** | **AEPE $\\downarrow$** |\n| Alligat0R | nuScenes-all | **43.82** | \n| CroCo | nuScenes-all | 77.61 |\n| Alligat0R | nuScenes-50 | $\\underline{58.96}$ |\n| CroCo | nuScenes-50  | 92.98 |\n|  |  |  |\n| Alligat0R | ScanNet-all | **36.07** | \n| CroCo | ScanNet-all  | 56.70 |\n| Alligat0R | ScanNet-50 | $\\underline{40.25}$ | \n| CroCo | ScanNet-50 | 86.60 |\n|  |  |  |\n| CroCo v2 | 5 datasets | 51.55 |\n\nAs we can see, when trained on the same dataset, Alligat0R always outperforms CroCo for this task, and even beats CroCo v2, showcasing the generalization of our method. An interesting finding is that CroCo trained on all overlaps outperforms its counterpart trained on at least 50% overlap. We will add those results in the camera-ready version.\n\n## **Framing and claims**\n\nWe agree with the reviewer that our method should be positioned as complementary rather than universally superior to CroCo, as it is supervised and targets different assumptions. Our goal is to enable learning from image pairs with arbitrary overlap, and our results confirm that Alligat0R is effective for both pose regression and dense matching under this paradigm. We will modify it.\n\n## **Specific questions**\n\n**Quarternion parameterization:** We use unit quaternions with L2 normalization, treated as 4D vectors in the homoscedastic loss.\n\n**Two-class experiment:** We actually have this experiment in Table 2 of the Supplementary Material. Results show 3-class formulation provides modest improvements. However, we believe that the model's knowledge about occluded regions might be beneficial for other tasks, especially for pointmap regression. That remains to be tested.\n\n## **Technical corrections**\n\nWe thank the reviewer for their comments and we will implement all corrections and adopt more balanced framing in the camera-ready version while maintaining our core technical contributions."}]

Author rebuttal:

Meta-review: This paper proposes Alligat0R, a pre-training scheme for binocular vision that enhances downstream applications, such as camera pose regression. The method's core idea is to predict a covisibility segmentation map for a second image, classifying pixels as covisible, occluded, or outside the field of view.
 
During the review process, the reviewers appreciated the simplicity and effectiveness of the proposed pre-training framework, noting the improved results and the high quality of the writing.
However, the reviewers also raised several concerns, including:
1. The method's sensitivity to errors in the training data.
2. The necessity of ground truth camera poses and depth maps to generate training data.
3. The evaluation focused on global relative pose estimation rather than local computer vision tasks like stereo matching.
4. The evaluation was conducted on test splits from the training domain rather than on out-of-domain data.

During the rebuttal and discussions, most of the concerns have been solved. The authors provided a strong rebuttal with new experiments, showing robustness to label noise and successfully demonstrating generalization to a new task and an out-of-domain dataset. Consequently, all the reviewers are positive about the paper with final scores: 5,5,5,4.
Therefore, I believe that this paper should be accepted to NeurIPS.
Decision: Accept (spotlight)