GIM: Learning Generalizable Image Matcher From Internet Videos
Authors: Xuelun Shen
zhipeng cai
Wei Yin
Matthias Müller
Zijun Li
Kaixuan Wang
Xiaozhi Chen
Cheng Wang
Venue: ICLR 2024 spotlight (2024)
Keywords: Image Matching
Pose Estimation
3D Reconstruction
Primary area: applications to robotics, autonomy, planning
TL;DR: We propose the first framework to learn a single generalizable image matching model from internet videos.

Abstract: Image matching is a fundamental computer vision problem. While learning-based methods achieve state-of-the-art performance on existing benchmarks, they generalize poorly to in-the-wild images. Such methods typically need to train separate models for different scene types (e.g., indoor vs. outdoor) and are impractical when the scene type is unknown in advance. One of the underlying problems is the limited scalability of existing data construction pipelines, which limits the diversity of standard image matching datasets. To address this problem, we propose GIM, a self-training framework for learning a single generalizable model based on any image matching architecture using internet videos, an abundant and diverse data source. Given an architecture, GIM first trains it on standard domain-specific datasets and then combines it with complementary matching methods to create dense labels on nearby frames of novel videos. These labels are filtered by robust fitting, and then enhanced by propagating them to distant frames. The final model is trained on propagated data with strong augmentations. Not relying on complex 3D reconstruction makes GIM much more efficient and less likely to fail than standard SfM-and-MVS based frameworks. We also propose ZEB, the first zero-shot evaluation benchmark for image matching. By mixing data from diverse domains, ZEB can thoroughly assess the cross-domain generalization performance of different methods. Experiments demonstrate the effectiveness and generality of GIM. Applying GIM consistently improves the zero-shot performance of 3 state-of-the-art image matching architectures as the number of downloaded videos increases (Fig. 1 (a)); with 50 hours of YouTube videos, the relative zero-shot performance improves by 6.9% − 18.1%. GIM also enables generalization to extreme cross-domain data such as Bird Eye View (BEV) images of projected 3D point clouds (Fig. 1 (c)). More importantly, our single zero-shot model consistently outperforms domain-specific baselines when evaluated on downstream tasks inherent to their respective domains. The code will be released upon acceptance.

Published as a conference paper at ICLR 2024
GIM: LEARNING GENERALIZABLE IMAGE MATCHER FROM INTERNET VIDEOS
Xuelun Shen't, Zhipeng Cai?te, Wei Yin Matthias Müller?, Zijun Li", Kaixuan Wang Xiaozhi Chen Cheng Wang
Equal Contribution, Corresponding athuor (cwang eeraag@@xm.cer xmu.edu.cn, zhipeng, cai @el@@nntel @intel.com) Fujian Key Laboratory of Sensing and Computing for Smart Cities, Xiamen University, 361005, P.R. China
2 Intel Labs," 3 DJI Technology
55
GIMOKM
Match
CoFTR
50
GIM SuperGiee
49.4
48.3
46.2
47.4
Image gair
Point cloud
AUC@5
GIMpXM
40
(b) 3D reconstruction.
39.1
37.3
Manch
35
35.5
33.1
33.5
34.3
32.5
31.22
BEV point cloud pair
30
12.5
50.0
Video length (hours)
DKM
GIMDEM
(a) Zero-shot performance vs video length.
(c) BEV point cloud registration
Figure GIM overview. We propose an effective framework for learning generalizable image matching (GIM) from videos. (a): GIM can be applied to various architectures and scales well as the amount of data (i.e., internet videos) increases. (b): The improved performance transfers well to various downstream tasks such as 3D reconstruction. (c): Our best model GIMpKM also generalizes challenging out-of-domain data like Bird Eye View (BEV) images of projected point clouds.
ABSTRACT
Image matching a fundamental computer vision problem. While learning barning-bbseed methods achieve state-of-the-art performance on existing benchmarks, they gener- alize poorly to in-the-wild images. Such methods typically need to train separate models for different scene types (e.g., indoor vs. outdoor) and are impractical when the scene type is unknown in advance. One of the underlying problems is the limited scalability of existing data construction pipelines, which limits the di- versity of standard image matching datasets. To address this problem, we propose GIM, a self-training framework for learning single generalizable model based any image matching architecture using internet videos, abundant and diverse data source. Given an architecture, GIM first trains it on standard domain-specific datasets and then combines it with complementary matching methods to create dense labels on nearby frames of novel videos. These labels are filtered by robust fitting, and then enhanced by propagating them distant frames. The final model is trained on propagated data with strong augmentations. Not relying on complex 3D reconstruction makes GIM much more efficient and less likely fail than stan- dard SIM-and-MVSS based frameworks. We also propose ZEB, the first zero-shot evaluation benchmark for image matching. By mixing data from diverse domains, ZEB can thoroughly assess the cross domain generalization performance of differ- ent methods. Experiments demonstrate the effectiveness and generality GIM. Applying GIM consistently improves the zero-shot performance of state-of-the- art image matching architectures as number of downloaded videos increases (Fig. 1 (a)); with 50 hours of YouTube videos, the relative zero-shot performance improves by 6.9% - 18.1%. GIM also enables generalization to extreme cross- domain data such as Bird Eye View (BEV) images of projected 3D point clouds (Fig. (C)). More importantly, our single zero-shot model consistently outper- forms domain-specific baselines when evaluated downstream tasks inherent their respective domains. The source code, a demo, and the benchmark are avail- able hupp://suelmsshen.cmm//.

---

Published as a conference paper at ICLR 2024
INTRODUCTION
Image matching fundamental computer vision task, the backbone for many applications such as 3D reconstruction Ullman (1979), visual localization Sattler et al. (2018) and autonomous driv- ing (Yurtsever et al., 2020).
Hand-crafted methods (Lowe, 2004; Bay et al., 2006) utilize predefined heuristics to compute and match features. Though widely adopted, these methods often yield limited matching recall and den- sity challenging scenarios, such as oong-baselines and extreme weather. Learning-bar -based methods have emerged as a promising alternative with a much higher accuracy (Sarlin et al., 2020; Sun et al., 2021) and matching density (Edstedt et al., 2023). However, due the scarcity of diverse multi- view data with ground-truth correspondences, current approaches typically train separate indoor and outdoor models on ScanNet and MegaDepth respectively. Such domain-specitfe training limits their generalization to unseen scenarios, and makes them impractical for applications with unknown scene types. Moreover, existing data construction methods, which rely on RGBD seans (Dai et al., 2017) or contturr-frro-Moone efficiency and applicability, making them ineffective for scaling up the data and model training.
(SfM) + Multi-view Stereo (MVS) (Li & Snavely, 2018), have limited
To address these issues, we propose GIM, the first framework that can learn single image matcher generalizable to in-the-wild data from different domains. Inspired by foundation models for com- puter vision (Radford et al., 2021; Ranftl et al., 2020; Kirillov et al., 2023). GIM achieves zero- shot generalization by self ooafniraining (Grandvalet & Bengio, 2004; Arazo et al., 2020) on diverse and large-scale visual data. We use internet videos as they are easy to obtain, diverse, and practically unlimited. Given any image matching architecture, GIM first trains it on standard domain-specific datasets (Li & Snavely, 2018; Dai al., 2017). Then, the trained model is combined with multiple complementary image matching methods to generate candidate correspondences on nearby frames of downloaded videos. The final labels are generated by removing outlier correspondences using robust fitting. and propagating the correspondences to distant video frames. Strong data augmenta- tions are applied when training the final generalizable model. Standard SfM and MVS based label generation pipelines (Li & Snavely, 2018) have limited efficiency and are prone to fail on in mn-the-will videos (see Sec. 4.2 for details). Instead, GIM can efficiently generate reliable supervision signals on diverse internet videos and effectively improve the generalization of tfate-of-the-art models.
To thoroughly evaluate the generalization performance of different methods, we also construct the first zero-shot evaluation benchmark ZEB, consisting of data from 8 real-world and 4 simulated do- mains. The diverse cross-domain data allow ZEB to identify the in-the-wild generalization gap of ex- isting Nomain--pecifee models. For example, we found that advanced hand-crafie methods (Arand- jelović & Zisserman, 2012) perform better than recent saaming-bbsse methods Sarlin et al. (2020); Sun al. (2021) on several domains of ZEB.
Experiments demonstrate the significance and generality of GIM. Using 50 hours of YouTube videos, GIM achieves relative zero-shot performance improvement of 9.9%. 18.1% and 6.9% respectively for SuperGlue (Sarlin et al., 2020) LoFTR (Sun et al., 2021) and DKM (Edstedt et al., 2023). The performance improves consistently with the amount of video data (Fig. I (a)). De- spite trained only on normal RGB images, our model generalizes well to extreme cross-domain data such BEV images of projected 3D point clouds (Fig. (c)). Besides image matching robustness, a single GIM model achieves cross-the-boan performance improvements on various down stwn-stream tasks such as visual localization, homography estimation and 3D reconstruction, even comparing to in-domain baselines on their specific domains. In summary, the contributions of this work include:
GIM, the first framework that can learn generalizable image matcher from internet videos.
ZEB, the first zero-shot image matching evaluation benchmark.
Experiments showing the effectiveness and generality of GIM for both image matching and various downstream tasks.
2
RELATED WORK
Image matching methods: Hand-crafted methods (Lowe, 2004; Bay al., 2006; Rublee et al., 2011) use predefined heuristics to compute local features and perform matching. RootSIFT (Arand- jelović & Zisserman, 2012) combined with the ratio test has achieved superior performance (Jin
2

---

Published as a conference paper at ICLR 2024
et al., 2020). Though robust, hand-crafted methods only produce sparse key- key-point matches, which contain many outliers for challenging inputs such as low overlapping images. Many methods (Tian et al., 2017; Mishchuk et al.. 2017; Dusmanu et al., 2019; Revaud et al., 2019: Tyszkiewicz et al., 2020) have been proposed recently to learn better single-image local features from data. Sarlin et al. (2020) pioneered the use of Transformers (Vaswani et al., 2017) with two images as the input and achieved significant performance improvement. The output density has also been significantly im- proved by state-of-the-arr semi-dense (Sun et al., 2021) and dense matching methods (Edstedt al., 2023). However, existing learning- carning-baseed methods train indoor and outdoor models separately, mak- ing them generalize poorly on in-the-wild data. We find that RootSIFT performs better than recent learning-based methods (Sarlin et al., 2020; Sun et al., 2021) in many in-the-wild scenarios. We show that domain-speciff training and evaluation are the cause of the poor robustness, and propose novel framework GIM that can learn generalizable image matching from internet videos. Similar to GIM, SGP (Yang et al., 2021) also applied self- self-training (using RANSAC + SIFT). However, was not designed to improve generalization and still trained models on domain-specithi data. Em- pirical results (Sec. 4.2) show that simple robust fitting without further label enhancement cannot improve generalization effectively.
Existing image matching methods typically train separate models for indoor and outdoor scenes using MegaDepth (Li & Snavely, 2018) and ScanNet (Dai et al., 2017) spectively. These models are then evaluated on test data from the same domain. MegaDepth consists of 196 scenes reconstructed from million internet photos using COLMAP (Schönberger & Frahm, 2016). The diversity is limited since most scenes are of famous tourist attractions and hence revolve around a central object. ScanNet consists of 1613 different scenes reconstructed from RGBD images using BundlcFusion. ScanNet only covers indoor scenes in schools and it is difficult to use RGBD scans to obtain diverse images from different places of the world. In contrast, we propose to use internet videos, a virtually unlimited and diverse data source to complement the scenes not covered by existing datasets. The in-domain test data used in existing methods also limited since they lack cross-domain data with diverse scene conditions, such as aerial photography, outdoor natural environments, weather variations, and seasonal changes. To address this problem and fully measure the generalization ability of model, we propose ZEB, a novel zero-shot evaluation benchmark for image matching with diverse in-the-wild data.
Image matching datasets:
Learning generalizable models has been an important research topic recently. CLIP (Radford et al., 2021) was trained on 400 million image-text pairs collected from the internet. This massive corpus provided strong supervision, enabling the model to learn wide range of visual-textual concepts. Ranftl et al. (2020) mixed various existing depth estimation datasets and complementing them with frames and disparity labels from 3D movies. This allowed the depth estimation model to first time generalize across different environments. SAM (Kirillov et al., 2023) was trained on SA-IB containing over billion masks from 11 million diverse images. This training data was collected using a "data engine", a three-stage process involving assisted- manual, semi-automatic, and fully automatic annotation with the model in the loop. A common approach for all these methods is to efficiently generate diverse and large scale training data. This work applies similar idea to learn generalizable image matching. We propose GIM, a self-training framework to efficiently create supervision signals on diverse internet videos.
Zero-shot computer vision models:
3 METHODOLOGY
Training image matching models requires multi-view images and ground-truth correspondences. Data diversity and scale have been the key towards generalizable models in other computer vision problems (Radford et al., 2021; Ranftl et al., 2020; Kirillov et al., 2023). Inspired by this observa- tion. we propose GIM (Fig. 2), self-training framework utilizing internet videos to learn a single generalizable model based on any image matching architecture.
Though other video sources are also applicable, GIM uses internet videos since they are naturally diverse and nearly infinite. To experiment with commonly accessible data, we download 50 hours (hundreds of hours available) of tourism videos with the Creative Commons License from YouTube, covering 26 countries, 43 cities, various lightning conditions, dynamic objects and scene types. See Appendix D for details.
3

---

Published as a conference paper at ICLR 2024
Frame
Frame
Frama
Frame
Complememary
Muli-method matching
Outler
propagation
Figure 2: GIM framework, We start by downloading a large amount of internet videos. Then, given selected architecture, we first train it on standard datasets, and generate corresoondences between nearby frames by using the trained model with multiple complementary image matching methods. The self-training signal is then enhanced by 1) filtering outlier correspondences with robust fitting, 2) propagating correspondences to distant frames and 3) injecting strong data augmentations.
Standard image matching benchmarks are created by RGBD scans Dai et al. (2017) or COLMAP (SfM + MVS) (Schönberger & Frahm, 2016; Li & Snavely, 2018). RGBD scans require physical access to the scene, making it hard to obtain data from diverse environments. COLMAP effective for landmark-type scenes with dense view coverage, however, it has limited efficiency and often fails on in-the-wild data with arbitrary motions. As a result, although millions of images are available in these datasets, the diversity is limited since thousands of images come from one (small) scene. In contrast, internet videos are not landmark-centrrc. A one hour tourism video typically covers a range of several kilometers (e.g., a city), and has widely spread view points. As discussed later in Sec. 3.1, the temporal information in videos also allows us to enhance the supervision signal significantly.
3.1 SELF-TRAININ
A naive approach to learn from video data is to generate labels using the standard COLMAP- pipeline (Li & Snavely, 2018); however, preliminary experiments show that it inefficient and prone to fail on in-the-wild videos (see Sec. 4.2 for details). To better scale up video training, GIM relies on self-training (Grandvalet & Bengio, 2004), which first trains a model on standard labeled data and then utilizes the enhanced output (on videos) of the trained model to boost the generalization of the same architecture.
Multi-method matching: Given image matching architecture, GIM first trains it on standard (domain-specific) datasets (Li & Snavely, 2018; Dai et al., 2017), and uses the trained model as the 'base label generator'. As shown in Fig. 2, for each video, uniformly sample images every 20 frames to reduce redundancy. For each frame X, we generate base correspondences between {X, + 20}. [X, X + 40) and (X, X + 80). The base correspondences robust fitting Barath et al. (2019) on the output of the base label generator. We fuse these labels with the outputs of different complementary matching methods to significantly enhance the label density. These methods can either be hand-crafted algorithms, or other architectures trained on standard datasets; see Sec. 4 for details.
generated by running
Existing image matching methods typically require strong supervision signals from images with small overlaps (Sarlin et al., 2020). However, this cannot be achieved by multi- method matching since the correspondences generated by existing methods are not reliable beyond an interval of 80 frames, even with ofte-oofthhear robust fitting algorithms for outlier filtering. An important benefit of learning from videos is that the dense correspondences between video frame and different nearby frames often locate at common pixels. This allows us propagate the correspondences to distant frames, which significantly enhances the supervision signal (see Sec. 4.2 for analysis). Formally, we define CAB {0, 1)r">>""" xr the correspondence matrix of image A and 1 where y and B are the number of pixels in A and 1B A matrix element og means that pixel in LA has a corresponding pixel j in B Given the correspondences CAB and CBC to obtain the propagated correspondences CAC. for each cBB in CBB that is 1, if we can also BBC find a G/k 1 in CCC, and the distance between j and j' in image _ is less than pixel. we set AC 1 in CAC, Intuitively, this means that for pixel j (or j') image B it matches both pixel i in A and pixel k in 10 Hence image 1 and JC have correspondence at location ki,
Label propagation:
4

---

Published as a conference paper at ICLR 2024
To obtain strong supervision signals, propagate the correspondences far as possible long as we have more than 1024 correspondences between two images. The propagation is exccuted on each sampled frame (with 20 frame interval) separately. After each propagation step, double the frame interval for each image pair that has correspondences. As an example, initially we have base correspondences between every 20, 40 and 80 frames. After round of propagation, we propagate the base correspondences from every 20 frames to every 40 frames and merge the propagated corre- spondences with the base ones. Now we have the merged correspondences for every 40 frames, we perform the same operation to generate the merged correspondences for every 80 frames. Since we have no base correspondence beyond 80 frames, the remaining propagation rounds do not perform the merging operation and keep doubling the frame interval until we do not have more than 1024 correspondences The reason we enforce the minimum number of correspondences is to balance the difficulty of the learning problem, so that the model is not biased towards hard or easy sam- plcs. Though the standard approach of uniform sampling from different overlapping ratios (Sun et al., 2021) can also be applied, we find it more space and computation friendly to simply limit the number of correspondences and save the most distant image pairs as the final training data.
Strong data augmentation: To experiment with various existing architectures, apply the same loss used for Nomain-speciffe training to train the final GIM model, but only calculate the loss on the pixels with correspondences. Empirically, we find that strong data augmentations video data provide better supervision signals (see Sec. 4.2 for the effect). Specifically, for each pair of video frames, we perform random perspective transformations beyond standard augmentations used in existing methods. We conjecture that applying perspective transformation alleviates the problem where the camera model of two video frames is the same and the cameras are mostly positioned front-facing without too much "roll" rotation.
In practice, the major computation for generating video training data lies in running matching meth- ods, and the average processing time per frame does not increase significantly w.r.t. the input video length. The efficiency and generality allows GIM to effectively scale up training on internet videos. It can process 12.5 hours of videos per day using 16 A100 GPUs, achieving a non-trivial perfor- mance boost for various stateoof-the-aa architectures.
3.2 ZRB: ZERO-SHOT EVALUATION BENCHMARK FOR IMAGE MATCHING
Existing image matching frameworks (Sarlin et al., 2020; Sun al., 2021: Edstedt et al., 2023) typically train and evaluate models on the same in-domain dataset (MegaDepth (Li & Snavely, 2018) for outdoor models and ScanNet (Dai et al., 2017) for indoor models). To analyze the robustness of individual models on in-the-wild data, we construct a new evaluation benchmark ZEB by merging 8 real-world datasets and 4 simulated datasets with diverse image resolutions, scene conditions and view points (see Appendix E for details).
For each dataset, we sample approximately 3800 evaluation image pairs uniformly from 5 image overlap ratios (from 10% to 50%). These ratios are computed using ground truth poses and depth maps. The final ZEB benchmark thus contains 46K evaluation image pairs from various scenes and overlap ratios, which has a much larger diversity and scale comparing to the 1500 in-domain image pairs used existing methods.
Metries:
Following the standard evaluation protocol (Edstedt et al., 2023), we report the AUC of the relative pose error within 5-, where the pose error is the maximum between the rotation angular error and translation angular error. The relative poses are obtained by estimating the essential matrix using the output correspondences from an image matching method and RANSAC (Fischler Bolles, 1981 1). Following the zero-shot computer vision literature (Ranftl et al., 2020; Yin et al., 2023), we also provide the average performance ranking across the twelve cross-domain datasets.
4 EXPERIMENTS
We first demonstrate in Sec. 4.1 the effectiveness of GIM on the basic image matching task relative pose estimation. We evaluate different methods on both our zero-shot benchmark ZEB and the standard in-domain benchmarks (Sarlin et al., 2020). In Sec. 4.2, we validate our design choices with ablation studies. Finally, we apply the trained image matching models various
5

---

Published as a conference paper at ICLR 2024
Table 1: Zero-shot matching performance. GIM significantly improved the generalization of all state-of-the architectures. IN means indoor model and OUT means outdoor model.
Method
Mean
Mean
(U)))5(()
ETO
Real
SEA
MUL
SCE
Simulaze
JCL
Handerofoed
RIDTSIFT
7.1
31.8
43.5
33.6
49.9
48.7
35.2
21.4
44.1
14.7
33.4
7.6
14.8
35.1
Spurse Matching
SUPERGLUF (IN)
9.3
21.6
19.2
16.0
38.2
37.7
22.0
20.8
40.8
13.7
21.4
9.6
18.8
SUPERGLUE (OUT)
6.6
31.2
29.7
24.2
52.3
59.3
28.0
28.2
48.0
20.9
33.4
4.5
16.6
29.3
GIMsusaGuaE
5,9
34.3
43.2
34,2
58.7
61.0
29.0
28.3
48.4
18.8
34,8
2,8
15,4
36.5
Semi dense Matching
LOFTR (IN)
9.6
10.7
5.6
5.1
11.8
7.5
17.2
6.4
9.7
3.5
22.4
1.3
14.9
23.4
LOFTR (OUT)
5.6
33.1
29.3
22.5
51.1
60.1
36.1
29.7
48.6
19.4
37.0
13.1
20.5
30.3
GIMLoFTR
3.5
39.1
50.6
43.9
62.6
61.6
35.9
26.8
47.5
17.6
41.4
10.2
25.6
45.0
Matching
DKM (IN)
2.6
46.2
44.4
37.0
65.7
73.3
40.2
32.8
51.0
23.1
54.7
33.0
43.6
55.7
DKM (OUT)
2.3
45.8
45.7
37.0
66.8
75.8
41.7
33.5
51.4
22.9
56.3
27.3
37.8
52.9
GTMoXM
58.3
74.5
25.1
53.7
32.3
38.8
GlMMXM with 100 hours of video
GIMOKM
51.2
63.3
53.0
73,9
76.7
43.4
52.5
24.5
56.6
32,2
42.5
61.6
downstream tasks (Sec. 4.3). To demonstrate the generality of GIM, we apply it to 3 state-of-thh-ar image matching architectures with varied output density, namely, SuperGlue (Sarlin et al., 2020), LoFTR (Sun et al., 2021) and DKM (Edstedt et al., 2023).
Implementation details: We take the official indoor and outdoor models of each architecture as the baselines. Note that the indoor model DKM is trained on both indoor and outdoor data. For fair comparisons, only allow the of complementary methods that perform worse than the baseline during multi-method matching. Specifically, we use RootSIFT, RootSIFTTSSuperGiee and respectively to complement SuperGlue, LoFTR and DKM. We use the outdoor official model of each architecture as the base label generator in GIM. Unless otherwise stated, we use 50 hours of YouTube videos in all experiments, which provide roughly 180K pairs of training images. The GIM label generation on our videos takes 4 days on 16 A100 GPUs. To achieve the best in-domain and cross-domain performance with single model, we train all GIM models from scratch using a mixture of original in in-domain data and our video data (sampled with equal probabilities). The training code and hyper-parameters of GIM strictly follow the original repositories of the individual architectures.
4.1
MAIN RESULTS
We use the proposed ZEB benchmark to evaluate the zero-shot general- ization performance. For all three architectures (Tab. 1), applying GIM produces single zero-shot model with a significantly better performance compared to the best in-domain baseline. Specifically, the AUC improvement for SuperGlue, LoFTR and DKM is respectively 31.2 2 + 34.3, 33.1 - 39.1 and 46.2 + 49.4. GIMsuperGlue performs even better than LoFTR (IN)/OUT), despite using a less advanced architecture. Interestingly, the hand-crafted method RootSIFT (Arandjelović & Zisser- man, 2012) performs better or on-par with the in-domain models on non-trivial number of ZEB subsets, e.g., GI.3, BLE, KIT and GTA. GIM successfully improved the performance on these sub- sets, resulting in a significantly better robustness across the board. Note that the performance of GIM did not saturate yet (Fig. 1), and further improvements can be achieved by simply download- ing more internet videos. For example, using 100 hours of videos (Table 1, last row) we further improved the performance of GIMPKM to 51.2% AUC.
Zero-shot generalization:
Two-view geometry: Qualitatively, GIM also provides much better two-view match- ing/reconstruction on challenging data. As shown in Fig. 3, the best in-domain baseline DKM (IN) failed to find correct matches data with large view changes small overlaps (both indoor and outdoor), resulting in erroneous reconstructed point clouds. Instead, GIMpKM finds a large number of reliable correspondences and manages to reconstruct dense and accurate 3D point clouds. In- terestingly, the robustness GIM also allows to be applied to inputs completely unseen during training. In Fig. 4, we apply GIMPKM to Bird Eye View (BEV) images generated by projecting

---

Published as a conference paper at ICLR 2024
Image pair
DRM
DAM
GiMMoOM
CiMMOR
Figure 3: Two-view reconstruction. DKM returns many incorrect matches (red lines) on challeng- ing scenes resulting in erroneous reconstruction. Applying GIM to the same architecture signifi- cantly improves matching and reconstruction quality.
Image pulr
BIV
Muscles
DKM
GIMOKM
Figure 4: Point cloud BEV image matching. GIMpKM even successfully matches BEV images projected from point clouds despite never being trained for it.
the top-down view of two point clouds into 2D RGB images. The data comes from real mapping application where want to align the point clouds of different building levels in the same hor- izontal plane. Unlike the best baseline DKM (IN) that fails catastrophically, our model GIMpKM successfully registers all three pairs of point clouds even though BEV images of point clouds were never seen during training. Due to the space limit, we show the qualitative results for the other architectures in Appendix G.
GIM also performs well for multi-view reconstruction. To demon- strate the performance in-the-wild data, download internet videos for both indoor and outdoor scenes, extract roughly 200 frames for each video, and run COLMAP (Schönberger & Frahm, 2016) reconstruction but replace the SIFT matches with the ones from our experimented models. As shown
Multi-view Reconstruction:

---

Published as a conference paper at ICLR 2024
Sumple frames
DKM (UN)
GIMMOM
Figure 5: Multi-view reconstruction. GIM significantly improved the reconstruction coverage and accuracy.
Table Ablation study.
Table 3: Homography estimation.
Models
Method
AUC (%)
@3px
@5px
@ @10px
GIMPRM
49.4
SUPERGLUE (OUT)
53.9
68.3
81.7
replace 5üh video to 25h
48.3
GIMsipeGlle
68.8
82.3
replace
3) only use RootSIFT to generate labels
49.3
LoFTR (aUT)
65.9
75.6
4) wio data augmentation
49.2
GIM
70.6
79.8
88.0
5) woo propagation
47.1
6) use COLMAP labels, same computation and time 2)
46.5
DKM (OUT)
71.3
80.6
88.5
7) w/o video
46.2
GIMDKM
71.5
80.9
89.1
in Fig. 5, applying GIM allows DKM to reconstruct much larger portion of the captured scene with denser and less noisy point clouds.
In-domain performance: We also evaluate different methods on the standard in-domain datasets. Due the space limit, we report the result in Appendix Though the improvement is as significant as in ZEB because individual baselines overfit well to the in-domain data, GIM still performs the best on average (over the indoor and outdoor scenes). This result also shows the importance of ZEB for measuring the generalization capability more accurately.
4.2 ABLATION STUDY
To analyze the effect of different GIM components, we perform an ablation study on our best- performing model GIMDKM- As shown in row 1, and of Tab. 2, the performance of GIM consis- tently decreases with the reduction of the video data size. Meanwhile, adding only a small amount (12.5h) of videos already provides reasonable improvement compared the baseline (46.2% to 47.4%). This shows the importance of generating supervision signals on diverse videos. Using only RootSIFT Arandjelović & Zisserman (2012) generate video labels, the performance of GIM duces slightly. Comparing the performance between rows 3 and 1, we can see that generating labels on more diverse images more important than having advanced base label generators. Removing label propagation reduces the performance more than lack of data augmentations and base label generation methods. Specifically, using hours of videos without label propagation performs even worse than using the full GIM method on only 12.5 hours of videos.
We also experiment with the standard COLMAP-based label generation pipeline (Li & Snavely, 2018) (row 6). Specifically, uniformly sample 200 frames for label generation. We apply the same GPU and time (roughly day) row 2 run COLMAP SEM+MVS. COLMAP only manages to process 3.9 hours of videos, and fails to reconstruct 44.3% of them, resulting in only 2.2 hours labeled videos (vs. 12.5 hours from GIM), and low performance improvement of 46. 2% 46.5%.
separate the downloaded videos into clips of 4000 frames and
8

---

Published as a conference paper at ICLR 2024
Table 4: Outdoor visual localization. Unit: %
Table 5: Indoor visual localization. Unit: of correctly localized queries (1)
of correctly localized queries (1).
Method
Day
Night
Method
DUCI
DUC2
(0.25m,2"))
(0.5m,5")/ (1.0m,10')
(0.25m, 10") (0.5m, (00mmm00"/((/) (L.0m (1.0m,10)) 10")
SUPERGLUE (OUT)
1.8/96.1/999
77.0 90.6 100.0
SUPERGLUU (IN)
49.0 68.7 80.8
53.4/77.1//
GIMSUSROUUF
90.3 96.4 /9..4
78.0 90.6 100.0
GIMsppratus
53.5 / 76.8 86,9
61.8 / 85,5 87.8
LOFTR (OUT)
88.7/95.6/ /9.1
78.5 / 99.0
LOFTR (IN)
47.5/ 84.8
54.2/ 85.5
GIMLIFTR
90.0/9..2/ 99.4
79.1 /1.6 100.0
GLMLOFTR
54.5 87.4
63.4/83.22 87.0
DKM (OUT)
84.8/ 1.8/99.7/97.1
70.2
99.1 97.4
DKM (IN)
51.5/75.3 86,9
63.4 82.4 87.8
GIMDKM
89.7 99.2
77.0 90.1/ 99.5
GIMDKM
57.1/78.. 8884
70.2 91.6 92.4
4.3 APPLICATIONS
Homography estimation: As classical down-stream application (Edstedt et al., 2023), we con- duct experiments on homography estimation. We use the widely adopted HPatches dataset, which contains 52 outdoor sequences under significant illumination changes and 56 sequences that exhibit large variation in viewpoints. Following previous methods Dusmanu et al. (2019), we use OpenCV to compute the homography matrix with RANSAC after the matching procedure. Then, we com- pute the mean reprojection error of the four corners between the images warped with the estimated and the ground-truth homography as a correctness identifier. Finally, we report the area under the cumulative curve (AUC) of the corner error up to 3, 5, and 10 pixels. We take the numbers from the original paper for each baseline.
As illustrated in Tab. 3, the GIM models consistently outperform the baselines, even though the baselines are trained for outdoor scenes already. Among all architectures, GIM achieves the most pronounced improvement
LoFTR, achieving absolute performance increase of 4.7%, 4.2%, and 3.4% in the three metrics.
Visual localization: Visual localization is another important down-stream task of image matching. The goal is to estimate the 6-DoF poses of an image with respect to 3D scene model. Following standard approaches (Sun al., 2021), we evaluate matching models Term Visual Localization benchmark, namely, the Machen-Day-Night v1.1 dataset (Sattler et al., 2018) for outdoor scenes and the InLoc dataset (Taira et al., 2018) for indoor scenes. We use the standard localization pipeline HLoc (Balntas et al., 2017) with the matches extracted by correspond- ing models to perform visual localization. We take the numbers from the original paper for each baseline. Since DKM did not report the result on the outdoor case, we use the outdoor baseline to obtain the performance number.
tracks of the Long-
With single model, GIM consistently and significantly out-performs the domain-specific baselines for both indoor (Tab. 5) and outdoor (Tab. 4) scenes. For example, we improve the absolute pose accuracy of DKM by >> 5% for the (0.25m, 2") metric in both indoor and outdoor datasets. For indoor scenarios, GIMpKM reaches a remarkable performance of 57.1 78.8 / 88.4 on DUCI and 70.2 / 91.6 92.4 on DUC2. These results show that without the need of domain- training, single GIM model can be effectively deployed to different environments.
5 CONCLUSION
We have introduced a novel approach GIM, that leverages abundant internet videos to learn gen- eralizable image matching. The key idea is perform trainiraining where we use the enhanced output of domain-specific models to train the same architecture, and improve generalization by con- suming large amount of diverse videos. We have also constructed novel zero-shot benchmark ZEB that allows thorough evaluation of an image matching model in in-the-wild environments. We have successfully applied GIM increases steadily with the video data size. The improved image matching performance also benefits various downstream tasks such as visual localization and 3D reconstruction. single GIM model generalizes to applications from different domains.
state-of-the-art architectures. The performance improvement

---

Published as a conference paper at ICLR 2024
REFERENCES
Relja Arandjelović and Andrew Zisserman. Three things everyone should know to improve object retrieval. In 2012 IEEE conference on computer vision and pattern recognition, pp. pp.2911-22 2911-2918. IEEE, 2012.
Eric Arazo, Diego Ortego, Paul Albert, Noel E. O'Connor, and Kevin McGuinness. Pscudo-labeliee labeling and confirmation bias in deep semi-supervised learning. In International Joint Conference on Neural Networks, pp. 1-8. IEEE, 2020.
Vassileios Balntas, Karel Lene, Andrea Vedaldi, and Krystian Mikolajczyk. Hpatches: A benchmark and evaluation of handcrafted and learned local descriptors. In CVPR, 2017.
Daniel Barath, Jiri Matas, and Jana Noskova. MAGSAC: marginalizing sample consensus. In Conference on Computer Vision and Pattern Recognition, 2019.
Herbert Bay. Tinne Tuytelaars, and Luc Van Gool. Surf: Speeded up robust features. In Aleš Leonardis, Horst Bischof, and Axel Pinz (eds.), Computer Vision ECCV 2006, pp. 404-417, Berlin, Heidelberg. 2006. Springer Berlin Heidelberg. ISBN 978-3-540-33883-4
Angela Dai, Angel X. Chang, Manolis Savva, Macicj Halber, Thomas A. Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. CVPR, pp. 2432-2443, 2017.
Mihai Dusmanu, Ignacio Rocco, Tomás Pajdla, Mare Pollefeys, Josef Sivic, Akihiko Torii, and Torsten Sattler. D2-net: trainable enn for joint description and detection of local features. CVPR, pp. 8084 00438009,,10 2019.
Johan Edstedt, Joannis Athanasiadis, Märten Wadenbäck, and Michael Felsberg. Dkm: Dense kernelized feature matching for geometry cstimation. In Proceedings of the IEEE//VV Conference on Computer Vision and Pattern Recognition, pp. 17765-17778 2023.
Martin A. Fischler and Robert C. Bolles. Random sample consensus: paradigm for model fitting with appli- cations image analysis and automated cartography. Commun. ACM, 24:381-333 335, 1981.
Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In Conference on Computer Vision and Pattern Recognition (CVPR), 2012.
Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization. Advances in neural information processing systems, 17, 2004.
A. Handa, T. Whelan, J.B. McDonald, and AJ.. Davison. A benchmark for RGB-D visual odometry, 3D reconstruction and SLAM, IEEE Iul. Conf. on Robotics and Automation, ICRA, Hong Kong, China, May 2014.
Yuhe Jin, Dmytro Mishkin, Anastasia Mishchuk, Jiri Matas. Pascal Fua, Kwang Moo Yi, and Eduard Trulls. Image Matching across Wide Baselines: From Paper Practice. International Journal Computer Vision, 2020.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, et al. Segment anything. arXiv preprint arXiv:2304.0266 2023.
Zhengqi Li and Noah Snavely. Megadepth: Learning single-view depth prediction from internet photos. CVPR, pp. 2041-2050, 2018.
David Lowe. Distinctive image features from scale-invariant keypoints. IJCV, 60:91-100,, 2004.
Will Maddern, Geoff Pascoe, Chris Linegar, and Paul Newman. 1 Year, 1000km: The Oxford RobotCar Dataset. The International Journal of Robotics Research (IJRR), 36(1):3-15, 2017.
John McCormae, Ankur Handa, Stefan Leutenegger, and Andrew J. Davison. Scenenet rgh-d: Can 5m synthetic images beat generic imagenet pre-training indoor segmentation? In ICCV, 2017.
A. Mishchuk, Dmytro Mishkin, Filip Radenović, and Jiri Matas. Working hard to know your neighbor's margins: Local descriptor learning loss. In NeurlPS, 2017.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763 PMIR, 2021.
10

---

Published as a conference paper at ICLR 2024
René Ranfil, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE transactions on pattern analysis and machine intelligence, 44(3):1623-166 2020.
Jérôme Revaud, César Roberto Souza, M. Humenberger, and Philippe Weinzaepfel. R2d2: Reliable and repeatable detector and descriptor. In NeurlPS, 2019.
Ethan Rublee, Vincent Rabaud, Kurt Konolige, and Gary Bradski. Orb: An efficient alternative sift or surf. doi: 10.1 10911
In 20/1 International Conference on Computer Vision, pp. 2564-2571, 2011.
6126544.
Paul-Edoue houl-Edoaard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. SuperGlue: Learning feature matching with graph neural networks. In CVPR, 2020.
Torsten Sattler, Will Maddern, Carl Toft, Akihiko Torii, I Lars Hammarstrand, Erik Stenborg, Daniel Safari, Masatoshi Okutomi, Mare Pollefeys, Josef Sivic, Fredrik Kahl, and Tomas Pajdla. Benehmarking 6DOF Outdoor Visual Localization in Changing Conditions. In Conference on Computer Vision and Puttern Recog- nition (CVPR), 2018.
Johannes L. Schönberger and Jan-Michad Frahm. Stracture-foommmttiinn revisited. CVPR, pp. 4104-4113, 2016.
Thomas Schöps. Johannes L. Schönberger. Silvano Galliani, Torsten Sattler, Konrad Schindler. Marc Pollefeys, and Andreas Geiger. A multi-view stereo benchmark with high nigh-resolution images and multi-camera videos. In Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Tianwei Shen, Zixin Luo, Lei Zhou, Runze Zhang, Siyu Zhu, Tian Fang, and Long Quan. Matchable image retrieval by learning from surface reconstruction. In The Asian Conference on Computer Vision (ACCV, 2018.
Jiaming Sun, Zehong Shen, Yuang Wang, Hujun Bao, and Xiaowei Zhou. Loftr: Detector local feature matching with transformers. CVPR, 2021.
Hajime Taira, Masatoshi Okutomi, Torsten Sattler, Mircea Cimpoi, Mare Pollefeys, Josef Sivie, Tomas Pajdla, and Akihiko Torii. InLoc: Indoor visual localization with dense matching and view synthesis. In CVPR, 2018.
Yurun Tian, Bin Fan, and F. Wu, 1.2-net: Deep learning of discriminative patch descriptor euclidean space. CVPR, 2017.
Michal J. Tyszkiewicz, Fua, and Eduard Trulls. Disk: Learning local features with policy gradient. NeurlPS, 2020.
Shimon Ullman. The interpretation of structure from motion. Proceedings of the Royal Society of London. 1979.
Seriex B. Biological Sciences, 203(1
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Fukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances neural information processing systems, 30, 2017.
Kaixuan Wang and Shaojie Shen. Flow-motion and depth network for monocular stereo and beyond. IEEE Robotics Autom. Lett. 5(2):3307-333) 2020.
Heng Yang, Wei Dong, Luca Carlone, and Vladlen Koltun. Self-supervised geometric perception. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14350-14361, 2021.
Yao Yao, Zixin Luo. Shiwei Li. Jingyang Zhang, Yufan Ren, Lei Zhou, Tian Fang, and Long Quan. Blend- edmvs: A large-scale dataset for generalized nult stereo networks. Computer Vision and Pattern Recognition (CVPR), 2020.
Wei Yin, Chi Zhang, Hao Chen, Zhipeng Cai, Gang Yu, Kaixuan Wang, Xiaozhi Chen, and Chunhua Shen. Metric3d: Towards zero-shot metric 3d prediction from single image. In ICCV, 2023.
Ekim Yurtsever, Jacob Lambert, Alexander Carballo, and Kazuya Takeda. A survey of autonomous driving: Common practices and emerging technologies. IEEE access, 1558443-58469 2020.
Zichao Zhang, Henri Rebecq, Christian Forster, and Davide Scaramuzza. Benefit of large field-of-view cameras for visual odometry. In 2016 IEFF International Conference on Robotics and Automation, ICRA 2016, Stockholm, Sweden, May 16-21, 2016, pp. 801-808. IEFF, 2016.
11

---

Published as a conference paper at ICLR 2024
A TRAINING DETAILS
Given 8 A100 GPUs, takes about 4, and 7.5 days train GuMeuperilue LFTR, and GIMDKM respectively. For small memory GPUs, we have tried 4 RTX-3090. It takes 15 days to generate training signals on 50 hours of videos. And for training. it takes 9.5, 12, and 19 days to train Superuuu LFML. and GIM respectively. Gradient accumulation is needed when using the RTX-3090 GPU to ensure the same training batch size as on the A100 GPU.
B PREVENT OVERFITTING
Preventing overfitting is crucial when training GIM with large-scale video data. Overfitting can occur if the training videos not diverse enough and biased towards certain types of scenes. To address this issue, we employ an effective and implicit regularization technique in GIM by carefully selecting video data from diverse seo-locations and scene conditions. We focus YouTube tourism videos and describe our search and filter strategies in Sec. 3 of the paper. Table provides detailed statisties of the selected videos, showcasing the diversity of the training data.
Another factor that helps prevent overfitting is the continually growing amount and diversity of inter- videos. As the training data expands, the risk of overfitting is effectively reduced. Additionally, we apply strong data augmentations during GIM training to make the model resistant to noise. The effectiveness of this implicit regularization approach is demonstrated by our experimental results in the Zhoo Evaluation Benchmark (ZEB).
Although not used in our video selection process, evaluating the trained model on the ZEB bench- mark can also serve as verification tool to determine whether a set of candidate videos are po- tentially biased or noisy. By assessing the model's performance on the diverse and challenging scenarios present in the ZEB, we can gain insights into the generalization capability of GIM and identify any potential overfitting issues.
C HANDLING NOISY AND LOW-QUALITY DATA
When selecting internet videos for GIM, we recommend choosing videos with high resolution, clear images, long duration, and small number of scene transitions. To ensure these criteria met, we focus on tourism videos, which are generally 0.5-2 hours long and feature a person holding the camera and recording their travel, mostly without transitions. We also remove the first and the last 5 minutes of each video. This preprocessing step is performed because the beginning and the end of tourism videos often contain quick previews or summaries, which can include non-smooth scene transitions. These strategies helps to minimize the impact of noise and low low-quality data on the training process.
GIM also exhibits some inherent robustness to transitions in the video. Drastic changes from one frame to another make very difficult for matches survive after robust fitting. Even if few matches manage to survive, it is challenging for the label propagation to continue. This property of GIM helps to mitigate the effects of sudden scene changes and transitions in the training videos.
Furthermore, GIM incorporates strong data augmentations during training, which introduces noise to make the model more resistant to various sources error the input data. By exposing the model to wide range of augmentations,, such as random cropping. flipping, and color jittering, can improve its robustness to noise and low-quality data that may be present the internet videos.
Despite these measures, it is important to acknowledge that the quality of the training data still have an impact on the performance of GIM. While our video selection strategy and the model's 's in- herent robustness help to mitigate the effects of noise and low-quality data, it is essential to strive for high-quality and diverse training videos whenever possible to ensure the best possible performance of GIM.
12

---

Published as a conference paper at ICLR 2024
Figure 6: Sample video data and the generated labels (for GIMnKM).
Table 6:
The downloaded videos cover wide range of scene types from 26 countries around the world, ensuring the diversity the training data in GIM.
Video statistics.
Country (26)
City (43)
Scenario (39)
Italy, China, Korea, Bosnia, Greece, Poland, Turkey, Germany, Vietnam, Romania, Croatia, Austria, Albania, Hungary, America. Cambudia, Slovenia, Slovakia, Bulgaria, Thailand, Lithuania, Singapore. Montenegro. Herzegovina, Switzerland, Czech Republic
Rome, Scoul, Gdynia, Sopot, Gdask, Karpacz, Krakow, Trento, Tropea, Myslecinek, Bangkok. Santorini, Hoi An, Travnik, Singapore, Side. Brasov, Kumpot, Bautzen, Ljubljana. Rovinj. Salzburg. Hue, Cottbus. Shkodr, Kemer, Bem, Prague, ilina, Budva, Debreeen, Vilnius, Kotor, Bodrum, Geneva. Varna, Shanghai, Milano. Dusscldorf, Busan, Los Angeles, Las Vegas, Irvine
Daytime, Driving, Suburhs, From Day to Night, Beach, Cave, Market, Sunny Day. Dock. Mountainous Area, Evening. Coast, Lights, Night, Park, Outskirts, Planetarium, Indoor and Outdoor Trunsition. Wilderness, Indoor, Storm rain, Lakeside, Chinatown, Street, Factory. Outdoor. Mountain Climbing.
Mountain Road, City, Building, Shopping Mall,
Small Town, Forest, Heavy Rain, Historic Building. Hollywood. Overcast Day,
Historical Relics, Subway Station
D DETAILS OF VIDEO DATA
In this section, we show details of our video data. Tab. shows the diverse geo-locations and scene types of our downloaded videos. Fig. 6 provides example training data (images and correspon-
13

---

Published as a conference paper at ICLR 2024
dences) generated video data, which covers both indoor and outdoor scenes, urban and natural environments, various illumination conditions.
E
DETAILS OF ZEB
Image Type
Dataset Name
Scenario
Image Size
GL33 Shen al. (2018)
aerial / wild
1000 1000
BlendedMVS Yao et al. (2020)
objects
1000 x 1000
ETH3D Indoor Schöps
(2017)
basement corridor
6000 4136
ETH3D Outdoor Schöps al. (2017)
school / park
6000 x 4136
Real Images
KITTI Geiger et al. (2012)
driving
1226 370
KobotcarWeather Maddemn et al. (2017)
weather changes
1280 x 960
RobotcarSeason Maddemn et al. (2017)
scasonal changes
1280 x 960
RobotcarNight Maddem et al. (2017)
sunlight changes
1280 960
Multi-FoV Zhang (2016)
driving
640 480
SceneNet RGB-D McCormac et al. (2017)
living house
320 x 240
Simulated Images
ICI-NUIM Handa (2014)
hotel office
640
480
UTA-SIM Wang & Shen (2020)
aerial / wild
640
x 480
Table 7: Datasets used construct our zero-shot evaluation benchmark ZEB. They contain varied image resolutions and scene conditions, with challenging view points (e.g., aerial images). They also cover both real and simulated images.
Figure 7: Sample images of our zero-shot evaluation benchmark ZEB. Various scene types, view points and lightning conditions are included to ensure thorough evaluation of the matching robustness.
This section shows the details of the proposed ZEB benchmark. Specifically, Tab. shows the 12 datasets used to construct ZEB, and the diverse scene conditions and images resolution covered by these datasets. We also show in Fig. sampled image pairs in ZEB, covering varied scene types, view points and lightning conditions.
F IN-DOMAIN EVALUATION RESULT
As mentioned Sec. 4.1, we also compared GIM with basclines on standard in-domain evaluation data, i.e., MegaDepphh-1506 (Edstedt et al., 2023) and ScanNet-1500 (Edstedt al., 2023). The
14

---

Published as a conference paper at ICLR 2024
Table In-domain results (t). GIM still achieved the best overall performance on in-domain data.
MegaDepth- 1500
ScanNet-
Mean
Method
AUC
@5
@ 10
@20
@5'
@ 10"
@20"
SUPERGLUE (IN)
31.9
46.4
57.6
16.2
33.8
51.8
39.62
SUPERGLUE (OUT)
42.2
61.2
76.0
15.5
32,9
49.9
46.28
GIMSUPERGLUE
41.3
60.7
75.9
15.8
33.1
51.2
46.33
LOFTR (IN)
4.0
9.3
18.4
22.1
40.8
57.6
25.37
LOFTR (OUT)
52.8
69.2
81.2
18.0
34.6
50.5
51.05
GIMLOFTR
51.3
68.5
81.1
19.5
37.3
55.1
52.13
DKM (IN)
59.2
74.1
84.7
29.4
50.7
68.3
61.07
DKM (OUT)
60.4
74.9
85.1
26.4
46.6
63.7
59.52
GIMpKM
60.7
75.5
85.9
27.6
49.5
67.7
61.15
evaluation metric follows existing methods (Edstedt et al., 2023; Sun et al., 2021), and we take the numbers from the paper for each in-domain baseline. As shown in Tab. 8, though in-domain baselines already overfitted well their trained domains, GIM still achieved the best average per- formance over indoor and outdoor scenes. The smaller performance gap comparing to the zero-shot scenario also shows the importance of the proposed ZEB benchmark, which elearly reflect the generalization performance.
G FURTHER QUALITATIVE RESULTS
Imigie
LOFTR
LOFTR
SUPERGLL (OUT)
Figure 8: Two-view reconstruction of other baselines. We take the in-domain baseline that per- forms the best on ZEB for qualitative evaluation. Both LoFTR and SuperGlue generalize poorly on challenging in-the-wild data.
In Sec. 4.1, we only have space to show baseline results for the best architecture DKM. Here we provide the ones also for LoFTR and SuperGlue. Fig. 8 shows the two-view reconstruction results in-the-wild images. Similar to DKM. the in-domain LoFTR and SuperGlue models also generalizes poorly challenging in-the-wild data. Fig. 9 shows the results on BEV point cloud registration.
15

---

Published as a conference paper at ICLR 2024
BEV
Image
LOFTR
LOFTR Warp
SUPIRG SUPERCLHE Warp
Figure 9: Point cloud BEV image matching of other baselines. The in-domain models of Su- perGlue and LoFTR also failed to find reliable correspondences, resulting in wrong point cloud warping.
The in-domain LoFTR and SuperGlue models failed to find reliable matches and the correct relative transformations between two point clouds.
16


Reviews:
[{"review_id": "aqS2wJGwTa", "reviewer": "Reviewer_sN7h", "summary": "This paper proposes a self-supervised method for image correspondence learning from easily accessible internet videos. The proposed method first trains an image-matching network on a standard dataset with GT supervision and then combines it with complementary image-matching methods to generate candidate correspondences between nearby frames of internet videos. Then, robust fitting is applied to remove outliers in the generated candidate correspondences. The remaining correspondences are used to re-train the image-matching network. By using this self-supervised training scheme, the performance improvement on existing image-matching models is impressive.", "questions": "Please refer to the weakness section above.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Many of the current deep networks suffer from poor generalization ability to unknown data distributions when the amount and diversity of training data are limited. Fine-tuning the model on the target data distribution with a small amount of data from the target domain with GT supervision is a natural way. However, obtaining GT information of the data from the target domain might not always be easy, especially in correspondence matching, pose estimation, 3D reconstruction, etc. \n\nTo address this challenge, this paper introduces a self-supervision strategy for image matchers using readily available internet videos. There is no need to run an SfM or COMAP pipeline, which is usually computationally expensive and un-robust to in-the-wild images, to obtain the GT correspondences. The performance improvement on three standard image-matching networks, diverse evaluation images, and various downstream tasks is impressive.", "weaknesses": "My comments below are more like questions instead of weaknesses. \n\n(1) The proposed method combines a baseline image-matching network (e.g., SuperGlue, LoFTR, DKM) trained on a standard dataset and complementary image-matching methods to generate candidate correspondences. From the experiments section, the complementary image matching methods perform inferiorly than the baseline network. I have two questions here. \n     a. Since the performance is inferior, why are they needed? Will this increase the number of estimated correspondences between two images and thus improve the performance?\n     b. The introduction says \"multiple\" complementary image matching methods are used. While in the experiments section, it seems that only one complementary image matching method is used for each baseline method (SuperGlue, LoFTR & DKM). Will the number of complementary methods affect the performance?\n\n(2) Does batch size affect the self-supervision performance? From the implementation details, the GIM label generation on 50 hours of YouTube videos takes 4 days on 16 A100 GPUs. How long will it take to re-train a baseline network using the generated labels (and on which type and how many GPUs)? Will the proposed method still work on an RTX 3090, which is commonly used in universities?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YbJWFMvG2c", "reviewer": "Reviewer_UvwB", "summary": "This paper proposed a Generalizable Image Matcher. It generalizes across training sets, view points, it works with BEV and even point cloud.\n\nThey also proposed ZEB, the first zero-shot evaluation benchmark for image matching. The claim it mixes data from diverse domains,\nby using it one can assess the cross-domain generalization performance.\n\nExtensive experiments on state-of-the-art baselines are compared.", "questions": "Please share more about the training details. By domain specific training is it just swapping the GT or there are more details that's not being covered in the paper?", "limitations": "", "rating": 10, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The overview image in page.1 is impressive already. The method works on three strongest baseline (DKM, SuperGlue, and LoFTR) and improves them further more. It surprises me the method works with such huge view point differences and it also works with BEV pointcloud.\n\nThe training is using internet videos which prevents the COLMAP (SfM + MVS) bias for a single scene.\n\nThe proposed GIM is essentially a point matching ground-truth reinvention by using the label propagation through video with strong augmentation. However, it's so effective on every method by using the same training according to section 3.1.\n\nI consider the simplicity not a weakness, but as a strength. If the proposed method is reproducable, I believe it would be the new standard of image matching.\n\nThe reconstruction results in Fig.3,4,5 are very impressive especially when you know it's training on sequence video instead of SfM alike scenario.", "weaknesses": "This paper is very impressive, I think the only thing left is just some implementation details becuase the self-training part is very short and only about the ground-truth instead of the training itself. \n\nThe only thing left is just open-sourcing the proposed code of the label propagation and training data to verify it's accuracy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nLsNGVlTUI", "reviewer": "Reviewer_jxyk", "summary": "Observing that existing datasets for learning image-matching algorithm lacks diversity, the authors propose to learn image matchers from diverse tourism videos available on the internet. To obtain (pseudo-)labels for training, the authors propose to aggregate predictions from multiple matchers trained on small datasets and use a label propagation algorithm to propagate labels beyond nearby frames. Training on the pseudo-labeled data, existing image matching techniques demonstrate strong generalizability to unseen image domains, significantly outperforming image matchers trained on traditional datasets. In addition, the authors demonstrated that the proposed self-training framework showed strong scaling behavior, promising stronger image matchers for future work.", "questions": "Questions:\n1. Will the performance continue to improve if the self-training is repeated multiple times or do the authors expect the models to start degrading due to noisy pseudo-labels?\n2. Current work focuses on tourism videos. Would the approach work for other types of videos such as egocentric videos?\n\n\nSuggestions:\n1. (Related Work) Image Matching Datasets: MegaDepth is outdoor and ScanNet is indoor?\n2. Section 3.1: “Multi-method Matching” was a little confusing. Would it make sense to use “Pseudo-matches Generation through Multi-method Matching”?\n3. Table 1 KIT: GIM_LoFTR is slightly worse that LoFTR (out). \n4. Table 2 w/o video: The reviewer assumes this is DKM (IN). It would be nice to specify this in the table. \n\n\nPre-rebuttal Rating: Overall, this is a good paper that presents an alternative to learning strong image matcher. The zero-shot investigation is insightful, and the framework’s strong zero-shot performance is encouraging despite its simplicity. The reviewer recommends accepting the paper prior to the rebuttal stage.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Simple and scalable framework: The proposed self-training framework is simple and scalable. \n2. Strong zero-shot generalizability: Compared to image matchers trained on the traditional datasets, image matchers trained using self-training demonstrated stronger zero-shot generalizability, yielding more robust and performant image matchers. \n3. Comprehensive experiments: Experiments include large collections of datasets and downstream tasks, showing the superiority of self-trained image matchers.", "weaknesses": "1. Lacking real indoor datasets in the benchmark: This is a nitpick but it would be great to have more real indoor datasets in the benchmark. Right now, most of the real datasets are driving-related and the indoor dataset only covers basements and corridors.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "04kZjNSp6L", "reviewer": "Reviewer_XccY", "summary": "The paper introduces the GIM framework, aiming to address the challenge of generalizing image matching to diverse real-world scenarios. Taking cues from established computer vision models, GIM adopts a zero-shot generalization approach, leveraging the vast and varied nature of internet videos. The methodology involves a two-step training process: initial training on domain-specific datasets followed by an integration with several image matching techniques. This combined model seeks to identify potential correspondences in close video frames, with outlier detection mechanisms ensuring data quality. Notably, while traditional methods such as SfM and MVS have exhibited limitations in handling in-the-wild videos, GIM claims to efficiently provide reliable supervision signals for these scenarios, thereby promising to push the boundaries of current state-of-the-art models.", "questions": "-  While the use of internet videos is innovative, how did you ensure that the videos used for training represent a diverse range of scenarios, especially given the potential for internet content to have biases?\n\n- How does the GIM model handle noise and varying quality within internet videos? Were any preprocessing steps or filters applied to ensure data quality?\n\n- How did you address concerns of overfitting, especially with the consistent improvement seen with increasing video data?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ The introduction of the GIM framework is a significant advancement, marking the first attempt to leverage internet videos in training a universally applicable image matcher. This initiative promises to address generalization challenges across multiple real-world contexts.\n\n+ The formulation of the ZEB benchmark is a noteworthy achievement. As a pioneering zero-shot evaluation benchmark integrating real-world and simulated data, ZEB is poised to become an instrumental tool in gauging the generalization capacities of existing models.", "weaknesses": "- The heavy reliance on internet videos for training might introduce biases or noise. The generalization capability of GIM, when trained on other diverse datasets, remains an unanswered question.\n\n- With the consistent performance improvement with increased video data, there might be concerns about potential overfitting. Addressing this, perhaps with regularization techniques or other measures, would be crucial.\n\n- Internet videos can be noisy, and their quality can vary. How resistant is GIM to such noise, and how does it handle low-quality data?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)