Title: Alligat0R: Pre-Training through Covisibility Segmentation for Relative Camera Pose Regression
Venue: NeurIPS 2025
Accepted: yes
Keywords: pre-training

co-visibility

3D

pose estimation

pose regression
Abstract: Pre-training techniques have greatly advanced computer vision, with CroCo’s cross-view completion approach yielding impressive results in tasks like 3D reconstruction and pose regression. However, cross-view completion is ill-posed in non-covisible regions, limiting its effectiveness. We introduce Alligat0R, a novel pre-training approach that replaces cross-view learning with a covisibility segmentation task. Our method predicts whether each pixel in one image is covisible in the second image, occluded, or outside the field of view, making the pre-training effective in both covisible and non-covisible regions, and provides interpretable predictions. To support this, we present Cub3, a large-scale dataset with 5M image pairs and dense covisibility annotations derived from the nuScenes and ScanNet datasets. Cub3 includes diverse scenarios with varying degrees of overlap. The experiments show that our novel pre-training method Alligat0R significantly outperforms CroCo in relative pose regression. Alligat0R and Cub3 will be made publicly available.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper defines a pretext task to pretrain a transformer network for geometric computer vision so that the resulting features are suitable for downstream task (e.g. relative pose regression demonstrated in the paper).\n\nGiven two images, the pretext task segments each image based on whether the pixel is visible / occluded / oustide of the other image.\nThe network is made of a self-attention encoder, cross-attentional decoder and a segmentation head.\n\nThe network is trained in a supervised manner i.e. each image pairs has an associated \"segmentation masks\" on two datasets: the indoor ScanNet1500 and the outdoor NuScenes dataset.\nThese datasets do not readily provide the \"segmentation\" masks used for training so the masks are generated automatically by leveraging the ground-truth data they provide (e.g. camera poses and depth maps, or camera poses and estimated depth maps).\n\nThe relevance of the pretraining is tested on the task of relative pose regression by augmenting the network with a pose regression head.\nThe network is evaluated on test splits of the ScanNet1500 and NuScenes datasets, which are standard pose estimation datasets.\nThe experiments compare the result between training the network from scratch, with the proposed pre-training and with another standard pre-training task Croco.\nThe results show that the proposed pre-training leads to the best performance, which support its relevance.\n\nThe contributions of the paper are:\n- the definition of a new pre-training task for 3D computer vision tasks\n- the pre-training task requires annotation and the paper contributes a training dataset derived from the existing NuScene and ScanNet1500\n- The experiments are sound and support the claim of the paper.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n- S1 -  The paper proposes a new pre-training task relevant for 3D computer vision task. Previous works have shown how complex yet important pre-training is in 3D vision so the contribution is useful to the community.\n- S2 - The paper derives a dataset to enable the proposed pre-training.\n- S3 - The paper is extremely well written and pleasant to read.\n- S4 - The pretraining and downstream task training are evaluated on both indoor and outdoor datasets.\n\n### Weaknesses\n\n- W1-  Some limitations in the experiments:\n  - a) There is no experiment with the off-the-shelf Croco or Croco v2 (trained on real indoor and outdoor images) with and without finetuning. There is no requirement for the Croco pretraining to be on the target dataset and such an experiment would be fairer to Croco as the Croco's author most likely tuned the network to get the best pretraining results.\n  - b) The downstream task is evaluated on the same dataset as the pre-training, which prevents appreciating whether the pre-training is generic, which is the case in Croco v1.\n  - c) The relevance of the pre-training is evaluated on a single downstream task (relative pose regression). Other tasks would have been relevant although it would have been more computationally demanding: monocular depth estimation, image matching, optical flow.\n\n- W2 - The formulation often argues that proposed pre-training \"better\" than Croco or addresses its limitation but it would be fairer to say that it is an alternative pretext task. For example:\n    - a) Croco's covisibility limitations (i): Croco is trained on image pairs with at least 50% overlap whereas Alligator can train on a pairs with covisibility ranging from [0,100]%. In practice, Croco can also be trained on the full range of covisibility but Croco's authors have run an ablation study showing that the pretraining is best when the covisibility ranges in [40,50]%. Raising the Croco's 50% covisibility requirement as a limitation would be sound only if similar study was run showing that Alligator is insensitive to the covisibility range.\n    - b) Croco's covisibility limitations (ii): the 50% covisibility requirement in Croco could be seen as a trade-off for the simpler self-supervision (Alligator requires annotation and at least ground-truth camera poses). Hence the suggestion to frame Alligator as an alternative pre-training method instead of a method better than croco.\n\nQUESTIONS:\n- In the pose regression loss, how are the quaternion parametrized?\n\n- Semantically speaking, occluded and outside the field of view could be considered the same class. Was there any experiment pre-training on two classes only (covisible / not covisible) and was there any insight?\n\nLIMITATIONS:\nYes, the authors have adequately addressed the limitations and potential negative societal impact of their work.\n\n\n### Misc. comments\n\nStrong or unjustified statements:\n- L25: \"Croco [43, 44] relies on pairs with at least 50% of covisible regions\". This is not a strict requirement but covisibility between [40,50]% leads to the best result as shown in Croco's ablation study.\n- L31-32: \"it [Alligator] aligns more [than Croco] directly with the correspondence reasoning required in downstream binocular vision task\". Cross view image completion implies that the network implicitly matches 1 pixel from the 1st image to its matching pixel in the 2nd image so it is also aligned with correspondence-based tasks.\n- L36: \" providing a more diverse training signal than previous approaches.\" This is hard to prove as it is hard to quantify a \"training signal\".\n- L90: \"Our symmetric design is not only more efficient but also better aligned with downstream binocular vision tasks, which typically process unmasked images.\" Why is it more efficient? Because there are less parameters to train? Why is it \"better aligned with downstream binocular vision tasks\"?\n\nMath typo:\n- Eq4: $c_{ij}$ should be outside the log\n\nConfusing writing:\n- \"confirming the results in the original CroCo paper\", which results?", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a new multi-view pre-training paradigm for 3D based on covisibility segmentation.\nThe key contribution is to replace the inpainting objective of CroCo with a cross-entropy loss over segmentation labels (non-covisible, covisible, occluded).\nTo pre-train the model they collect a dataset of 5M pairs from the nuScenes and ScanNet datasets.\nFinally they introduce a relative-pose head that directly predicts a quaternion and metric translation from their backbone features.\nThe authors show that their approach leads to better relative pose estimates on the nuScenes and ScanNet datasets when compared both to CroCo but also to other SotA methods.\n\nSTRENGTHS AND WEAKNESSES:\n**S1 - Elegant objective:** Masked inpainting objectives such as croco have multiple issues. One is that they require masking patches at training time, which is misaligned with how the model will be used in downstream tasks. The second is that if there is no overlap it typically performs poorly. In contrast the proposed objective is very simple, and faces fewer such issues.\n\n**S2 - Strong results:** The results are in general strong, beating out CroCo with a good margin. When finetuned for relative pose they also beat out MASt3R in most cases. The caveat here is that the benchmarks are only conducted on their training datasets, which I go into more detail in W1.\n\n\n**W1 - Unclear generalization:** All benchmarks in the paper are conducted on the training datasets. While I assume (although I could not actually find this in the paper) that they have excluded these benchmarks from the training data, the fact is that the chosen datasets are not very diverse, and there are likely very similar scenes in the training data as in the test. One could for example investigate map-free relocalization as an alternative benchmark, as the task there is metric pose prediction, to investigate whether the model generalizes. As it stands I think including at least one benchmark on an unseen dataset is vital.\n\n**W2 - Missing/non-standard evaluations:** I was dissapointed to find that there were no comparisons to previous direct relative pose estimation methods, such as Reloc3r. Besides this, the authors have chosen metrics which do not align with the standard AUC metrics in the field, which makes comparisons difficult. I understand that metric relative pose can draw advantage in measuring metric translation errors, instead of angular, but not including any comparisons (except to CroCo) makes this problematic.\n\nQUESTIONS:\n1. It was not entirely clear to me how the pairs are sampled for ScanNet and nuScenes. Are they sampled uniformly from all scenes except the test scenes in RUBIK and ScanNet1500? If the authors could provide more details here I would be grateful.\n\nIf evaluation was conducted on a benchmark other than the training sets, I would likely increase my score to accept (even if the method is not SotA).\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 4, "DATE": "2025-06-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Alligat0R, a pre-training method for binocular vision that departs from the established cross-view completion paradigm. The authors identify a limitation in prior work like CroCo: the task of reconstructing masked pixels is ill-posed for regions that are not mutually visible (i.e., occluded or outside the field of view). To address this, Alligat0R reframes the pre-training objective as a covisibility segmentation task. The model is trained to predict, for each pixel in one image, whether its corresponding 3D point is covisible, occluded, or outside the field of view in the second image. To support this new pre-training task, the authors have created Cub3, a large-scale dataset of 5 million image pairs with dense covisibility annotations, derived from the nuScenes and ScanNet datasets. The paper demonstrates through extensive experiments that pre-training with Alligat0R leads to significantly better performance on the downstream task of relative camera pose regression compared to a CroCo baseline, especially in challenging scenarios with low image overlap.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n\n1. The core idea is simple yet seems powerful. The paper correctly identifies a weakness in the cross-view completion pre-training task (e.g., CroCo), its inability to handle non-covisible regions gracefully. The proposed solution of replacing an ill-posed reconstruction task with a well-defined, multi-class segmentation task directly addresses the identified problem. \n\n2. The results are compelling. The head-to-head comparison with CroCo under identical conditions (Table 1) shows an impressive performance gain for Alligat0R. \n\n\nWeaknesses:\n\n1. The primary weakness of the method lies in its pre-training data requirements. Unlike self-supervised methods that can learn from raw image pairs, Alligat0R requires dense covisibility labels. Generating these labels necessitates access to ground-truth or high-quality estimated 3D information (camera poses and depth maps). Although the authors are transparent about their complex data generation pipeline for nuScenes, this dependency on 3D supervision limits the scalability of the pre-training to datasets where such geometric information is available, and it moves the method away from a purely self-supervised paradigm.\n\nA potential counter-point here is, Croco also requires covisibility in order to filter out image pairs with small overlap. However, it is worth noting that even without filtering, Croco's paradigm still works.\n\n\n2. The paper convincingly demonstrates the benefits of Alligat0R for relative pose regression. However, as mentioned in the limitation of this paper, the broader claim of pre-training for \"binocular vision\" would be strengthened by evaluating on other fundamental tasks like stereo matching or optical flow. This is important as Croco has shown the results on these tasks. It remains a question whether the features learned for covisibility segmentation are as effective for these dense correspondence tasks as they are for global pose estimation.\n\nQUESTIONS:\n1. Is it possible to use Dust3R/VGGT to provide pseudo depth and depth labels, and hence to generate Covisibility masks? Given that the Covisibility masks for nuScenes were generated by monocular depth estimation methods such as DepthAnything, the quality of Dust3R/VGGT should be sufficient for this case. (Dynamic pixels may be a potential problem, though)\n\n2. Do the authors have any other plan or idea to simplify the generation of Covisibility masks?\n\n\nThe main concerns is about Point 1 in Weaknesses, i.e., pre-training data requirements. The reviewer would be happy to change the rating if the authors could address these issues.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a pre-training scheme for 3D reconstruction and pose regression. The proposed scheme is based on covisibility segmentation where each pixel in one image is predicted as being covisible, occluded or outside the field of view in the second image. This is in contrast to an existing pretraining approach termed CroCro that is based on cross-view completion of image regions in a pair of images. The paper also presents a 5M image-pair dataset with dense covisibility annotations for training and validation of the proposed scheme.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: The proposed scheme is conceptually simple and intuitive. Unlike the CroCro scheme it does not require a high degree of overlap between the image pairs which is needed for cross-view completion. The proposed scheme can be trained on both, covisible and non-covisible regions in the image pairs unlike CroCro that requires training on regions that are covisible in both images in the image pair. Experimental results show that he proposed scheme performs better than CroCro on most cases in terms of pose regression accuracy and generalizability. The proposed scheme also shows higher pose regression accuracy compared to techniques that do not employ pre-training. \n\nWeaknesses: The proposed scheme would be expected to be sensitive to covisibility segmentation error. This point needs to be discussed accompanied by a formal sensitivity analysis. The notations in the cross-entropy loss function formulation (page 4, line 112) are confusing and need to be clarified/corrected.\n\nQUESTIONS:\nHow sensitive is the proposed scheme to errors in covisibility segmentation?\n\nLIMITATIONS:\nIn addition to the limitations mentioned by the authors, the dependence/sensitivity of the proposed technique on/to errors in segmentation should be discussed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}