ETO:Efficient Transformer-based Local Feature Matching by Organizing Multiple Homography Hypotheses
Authors: Junjie Ni
Guofeng Zhang
Guanglin Li
Yijin Li
Xinyang Liu
Zhaoyang Huang
Hujun Bao
Venue: NeurIPS 2024 poster (2024)
Keywords: local feature matching
3d vision
pose estimation
Primary area: machine_vision
TL;DR: We reduce the time comsuming for local feature matching by reducing the units that participate in transformer, while using more complicated homography hypothese to maintain the accuracy.

Abstract: We tackle the efficiency problem of learning local feature matching.Recent advancements have given rise to purely CNN-based and transformer-based approaches, each augmented with deep learning techniques. While CNN-based methods often excel in matching speed, transformer-based methods tend to provide more accurate matches. We propose an efficient transformer-based network architecture for local feature matching.This technique is built on constructing multiple homography hypotheses to approximate the continuous correspondence in the real world and uni-directional cross-attention to accelerate the refinement. On the YFCC100M dataset, our matching accuracy is competitive with LoFTR, a state-of-the-art transformer-based architecture, while the inference speed is boosted to 4 times, even outperforming the CNN-based methods.Comprehensive evaluations on other open datasets such as Megadepth, ScanNet, and HPatches demonstrate our method's efficacy, highlighting its potential to significantly enhance a wide array of downstream applications.

Transformer-based Local Feature Matching by Organizing Multiple Homography Hypotheses
Junjie Ni
Guofeng Zhang'*
Guanglin Li
Yijin Li
Xinyang Liu'
Zhaoyang Huang?
Hujun Bao
State Key Lab of CAD&CG, Zhejiang University
CCUH MMLab
Abstract
Recent developments have led to the emergence of transformer-based approaches for local feature matching, resulting in enhanced accuracy of matches. However, the time required for transformer-base feature enhancement is excessively long, which limits their practical application. In this paper, propose methods reduce the computational load of transformers during both the coarse matching and refinement stages. During the coarse matching phase, organize multiple homography hypotheses to approximate continuous matches. Each hypothesis encompasses several features to be matched, significantly reducing the number of features that require enhancement via transformers. In the refinement stage, we reduce the bidirectional self-attention and cross-attention mechanisms unidirectional cross-attention, thereby substantially decreasing the cost of computation. Overall, our method demonstrates least times faster compared to other transformer- based feature matching algorithms. Comprehensive evaluations on other open datasets such as Megadepth, YFCC100M, ScanNet, and HPatches demonstrate our method's 's efficacy, highlighting its potential to significantly enhance wide array downstream applications.
Introduction
Local feature matching is a fundamental problem in the field of computer vision and plays significant role in downstream applications, including but not limited to SLAM (38), 3D re- construction 9 10 visual localization [THH33] and object pose estimation [44 15]. However, traditional CNN-based methods [16, [17 often fail under extreme conditions due to the lack of
global receptive field, thus meeting failure under dramatic changes in scale, illumination, viewpoint, teakly-textured scenes.
Recently, some methods [8-20 forgo traditional CNN-based approaches and base on Trans- former 21 for better modeling the long-range dependencies. However, Transformer is widely known for its high computational complexity especially when is applied in vision tasks where the computational complexity grows quadratically in the number of the input image tokens (i.e., patches). To reduce the inherent complexity associated with the Transformer, these methods generally adopt the coarse-to-fine strategy and incorporate more computationally efficient variants of the Transformer, such as the Linear Transformer [22] Nevertheless, the computational overhead remains substantial and severely hinders the application demanding low-latency operations, such tracking (23-25), or those requiring the processing of extensive datasets, such as large-scale mapping [26].
*Corresponding author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Ratotvve Prsse Accurscy [sux35]
Quadtree ASpanForrger
Ours
0 LoFTR
Source
Homography
View
Hypotheses
LightGlue
SuperGlue
Predicted
10-
Paicch
12
Lightllue
18
Target
Uniform
View
Hypotheses
SuperPoints ointt Search
20 20
20
Image Pairs Per Second
Figure ETO goes beyond the Pareto curve between accuracy and Efficiency. This figure shows the performance of different state-of-the- art methods YFCC100M. We take into ac- count the time for the extraction of feature and their description. LightGlue and LightGlue* are different settings of LightGlue.
Figure As demonstrated in the figure, there exists correspondence between two red regions on the sphere. In contrast to uniform hypothe- ses, homography hypotheses approximate the correspondence function better, which allows for more precise matching results with fewer com- putational resources.
In this paper, propose solve the efficiency problem of transformer- based local feature matching. Our insights are twofold. First, we propose to introduce the homography hypothesis in the pipeline. The homography hypothesis is kind of piece-wise smooth prior to the scene that has long been explored in the vision tasks [27]. It allows us to create larger patches and reduce the tokens number that need to be processed in the Transformer. However, it non-trivial since the regular shape introduced by the homography hypothesis can bring significant errors, especially along the boundary. Besides, how to supervise the training of multiple homography hypotheses with the absence of ground truth remains a problem. Second, We empirically find it is redundant that the previous methods employ multiple self- and cross-attention in their fine-level stage since the coarse-level stage has conducted sufficient propagation. As result, the computation complexity can be further reduced.
Specifically, propose ETO, the Efficient Transformer-bast based Local Feature Matching by Organizing Multiple Homography Hypotheses. ETO follows previous methods (20) [9] and employs a two- stage coarse-to fo-nee pipeline. It first establishes matches at the patch level and then refines the matches to the sub-pixel level. In the first stage, ETO learns to predict a set of hypotheses, each encompassing multiple patches to be matched. We approximately assume that each patch to be matched within one hypothesis is on the same plane, and thus describe these matches under the homography transformation, as illustrated in Fig. The homography hypotheses allow us to reduce the image tokens (patches) that are fed to the Transformer. For a typical image with resolution of 640 480, Previous methods feed 80 60 tokens to the transformer with 1/8 resolution, while we only need to feed 20 x 15 with 1/32 resolution, which brings significant speed up. To reduce the possible error due to the regular shape of the homography hypotheses, ETO subdivides the patches into multiple sub-patcher problem of selecticon as a segmentation problem (28). After that, ETO refines the matches in the second stage. Unlike previous methods that employ multiple self- and cross-attention, ETO only conducts one cross-attention, and the size of query tokens it use is much smaller than previous methods. We call it uni-directional cross-attention. Empirically we find uni-directional cross attention converges significantly faster at training while providing much higher efficiency. As shown in Fig. [ ETO outpaces existing methods, achieving 4-5 times faster than LoFTR [20] and 2-3 times more rapid than LightGlue
and re-selects the correct hypothesis for each sub-patch, We model the
29 while maintaining a comparable accuracy with them.
Our contributions can be summarized as follows. 1) We introduce multiple homography hypotheses for the local feature matching problem, which can greatly compress the number of tokens involved in the transformer. 2) We introduce uni-directional coss-attention in the refinement stage. This structure provides fast inference efficiency while maintaining accuracy. 3) Our method not only matches the performance of other transformer-based approaches on diverse open-source datasets such as Megadepth 30), YFCC100 (3I), ScanNet (32), and HPatches (33, but it also operates at a significantly higher speed, outpacing all compared methods.
2
Related Works
Transformer-based Local Feature Matching, To find sparse correspondence between two images
under diverse viewpoint movement conditions, traditional hand-crafted [4 2 or early learning- based approaches methods [17] 35 usually match keypoints [36] 37] with their descriptors after

---

detecting them [16]. These kinds of matching pipelines are highly dependent on the description of unique feature points, and they fail naturally at dramatic viewpoint changes or poorly textured scenes. Thanks to transformer [21), researchers now have a toolbox to enhance the feature descriptors with global information. Earlier researches [36] integrate the information that where other key points are modeling the mapping relationship as a continuous 2-d function. Sun et al. [20] constructs a global matching pipeline for each unit on the feature map with Transformer [21 40.42. The following works refine this pipeline with optical flow [19 or more efficient attention structure 43). There exists other attempts. [4 follow another technical route without transformer. They try to merge the gap between optical flow [47 41 and local feature matching with the concept of confidence. Our approach is based on similar feature extractor of (2),, while parameterize more information for the units on the feature map, and finally extend the correspondence relationship to the homography relationship between them. The concept of parameterized units on the feature map for local feature matching is introduced by Ni et al. 18 but they parameterize the units only with scale.
to each descriptors of key points. Then, 38 39
Paramerterization in Local Feature Matching. Conventional techniques, as demonstrated in previ- ous works 33 48 35), when confronted with appearance differences due to changes in viewpoint, try to construct feature descriptors which are invariant to these changes, involving scale, normal and rotation, etc. For rule-based methods [34], they create hand-crafted descriptors with scale space anal- ysis [49]. For learning- beased methods [16]. they input many image pairs with different viewpoints, letting the neural network learn about the invariance of these changes in appearance. Nevertheless, to fully mitigate the impact of these appearance changes, it is essential to accurately estimate their effects. Recent efforts [38] 18 50 have attempted to directly estimate the scale differences between images and resize the images enhance the refinement stage of feature matching. In similar route, [51] has focused on directly estimating rotation to calibrate the local features. Unlike the methods aimed solely at discovering more invariant feature descriptors, we argue that parameterized local appearance changes not only pose a challenge in finding accurate matches but also provide direct avenue to achieve more accurate and more efficient matches over a broader spatial extent. It allows us to locally parameterize the image to many planes, thus creating multiple homography relationships, not just point-to-point correspondence.
Acceleration in Local Feature Matching. In the realm of local feature matching based on the Transformer architecture, three predominant strategies have been mainly employed to enhance computational efficiency in the past. The first approach involves substituting the Softmax function for the Optimal Transport algorithm introduced by (6).. The second approach seeks to replace the full Transformer with a linear Transformer (22). LoFTR (20 incorporates both of these strategies, however, it introduces a larger number of matching units compared to SuperGlue (36), resulting in considerably slower performance than [36]. The third strategy focuses on reducing the number of
layers and units within the Transformer. LightGlue (29] introduces early termination [52] 53 and
progressive unit selection strategies to accelcrate computation, yielding significant improvements in speed. However, the performance of LighiGlue heavily relies on SuperPoint [16]. which puts ceiling on its acceleration. In contrast, ETO relies on a more precise parameterized model, achieving higher coarse matching accuracy with feature map whose resolution 16 times smaller than LoFTR 20), which distinguishes ETO from another concurrent work Efficient LoFTR (54) that accelerates the algorithm with adaptive token selection on the feature map of the same resolution. Furthermore, during the fine matching stage, we introduce uni-directional cross-attention mechanism, allowing us to achieve higher matching speed while sacrificing only a minimal amount of fine stage accuracy.
3 Method
Fig. presents our comprehensive feature matching process, organized into three structured modules. These modules are interconnected through feature extractors inspired by U-Net (55] and local attributes generated by neural networks. For each unit on a H/32 x W/32 resolution feature map M,, we estimate the attributes of homography hypotheses H; For each unit j on H/8 x W/8 resolution feature map M2. it re-selects the optimal homography hypotheses Íi, from nearby 9 hypotheses to minimize the projection errors. For the chosen unit k; on H/2xW/2 resolution feature map M3. we fix its center point P7 at the source image and refine the coordinates of its projected point P3 at target image according to H, then get the final matches pt+ We introduce the feature extractor in Sec. Sec. details the estimation of the hypotheses. In Sec. 3.3 we describe the segmentation
3

---

Hypothesis
Segmentation
Refinement
Two-Stage Training
Source
Step 2
Step
D
L
L
L
H
Module
Midule
Cross Attention
Module
H
H
P
+ AP Pj
H
14
11
H
Target
Figure 3: Given the source image S and target image T. we first use a U-Net like feature extractor to get images' feature map different resolution: M, (H/32 x W/32), M2 (H/8 x W/8) and M3 (H/2 x W/2). We use local 3 3 patches to illustrate method: (a) We estimate homography hypotheses H, for every feature after performing transformer. (b) We segment the map from these hypotheses to minimize projection errors. With the applied homography matrix Hy, we can project the chosen source point P7 to target point Pf (c) We update the P/ to Pf* after uni-directional cross attention. The training process is split into two parts, the coarse and the fine. We train the coarse part with LH, while training the fine part with L, and Ly.
of the feature map, Sec. 3.4 delves into refining the matches, while Sec. 3.5 states supervision methodology.
3.1 Feature Extraction
Following 220]. we use R8 [56] as the basic feature extractor to get the feature map with the resolution of N H/32 W /32, while we assume that the resolution of the source image and target image is the same. Here we get N = H/32 x W W/32 features and then perform stacked self-attention and cross attention layers between these N tokens compute the feature map M1. Although there will be N (more than 9) possible patches, our method is mainly performed on locally adjacent patches, so we will omit N in the future and take the local 3x3 patches illustrate our method (as shown in Fig. 3 Then, we follow [8] to upsample the feature map Mi with U-Net (55) like structure. So, we can obtain the feature maps M2 and M3 1/8 and 1/2 scale, respectively.
3.2 Hypothesis Estimation
Traditional semi-dense feature matching methods 20 19 often divide an image into thousands of units. For each unit, they perform bipartite graph matching [57]. Contrarily, argue that bipartite graph matching can be extended to the local homography transformation as hypotheses that cover multiple units to be matched. This approach's merit lies two folds: achieving more precise matches estimated during the first stage and reducing the number of units which are involved in transformer.
For each unit i on Mj of source images, it is equipped with feature f/ confidence score C,, and a set of hypothesis homography parameters H, (including the source positions p E R², target positions p R², rotation r, R scale s; RR and perspective q R²). And the unit target images is indicated by a.
Homography Matrix. Initially, we outline the methodology for estimating each unit's local attributes (pt Pi. rs. Sy qs. G.) and subsequently use these attributes formulate the local homography matrix. Among these local attributes, the scale 8;, rotation r;, and perspective qi are more related to the feature itself and are regressed directly from features f; on the source image through an MLP network. In contrast, target coordinates p/. and confidence scores are more related with the feature map of target images. They are acquired by first identifying the unit o; with maximal similarity among all target units, and the similarity is defined as the cosine similarity of initial features f on the source image and target image. Following this, construct new features f by executing the group-wise correlation (58) within the neighborhood of the target units on M..
+ 5 SENeighbor(a;)
(< fl f. >g).
(I)
where < +, * >g the group-wins wise correlation 588 and the group size our method is 8, Neiihbor represents the 5x5 neighborhood of unit af, C indicates the operation of concatenation. With the new features f. we an MLP process it get the target position pf and confidence 6, With these attributes. in order to compute the homography matrix, we establish four target points B that correspond to four predetermined reference points:
B P + 2
84.
(2)

---

Extracted from UNet's
Extracted from UNet's
Feature Map M
Feature Map M2
32 pixel
H
H
H ; = argmin - HER
H4
Hg
H6
pixel
* (Ai,)i M₂)
H
H8
Figure 4: Any unit j on M2 should be classified for hypotheses in H to minimize projection error.
Each H; describes plane.
Here Bf are four imaginary points source image, while B are the corresponding target points of B. on the target image. Bi computed from B& by following operation: Al is the operation of rotation with parameter ri, 9e is the perspective transformation operation with parameter 9:- These operations allow each variable within the homography matrix H, to be deduced from projection equations of 4 correspondence B H. Details regarding the specific implementation methods for rotation and perspective transformations will be included the supplementary materials.
3.3 Segmentation
To propagate the homography hypotheses predicted by M1 to more detailed resolution. We introduce a segmentation operation at the feature map M2 with the resolution H/8 x W/8. Segmentation is per-unit classification task, and we predict class for each unit j on M2. Here, we only consider locally adjacent 3 x 3 patches, and all possible classes is defined as H {H,li = 1...9). This classification (segmentation) involves that, for each unit selecting hypothesis from H. that minimizes the projection error at the center of unit j. After our proposed segmentation stage, each sub-unit can find the hypotheses H, that make its error smallest in all possible hypotheses H. We illustrate the intuitive process of segmentation Fig.
Our proposed segmentation differs from traditional semantic segmentation. Instead of aiming for specific semantic category, it targets a dynamic geometric relationship. To find the relationship, we introduce a new cosine similarity matrix Cj between the local feature ʃ3 on M2 and all features fi Mi, Positional encoding is employed during this phase to enhance local features, which is indispensable here because the hypothesis in H are not equivalent. To predicting the class Hi by finding the maximum Cj, we generate the computed groundtruth H, as follows:
H, argmin H.EH || Pj H,P
where Hj is the optimal hypothesis that minimize the projection error. Then, use focal loss (59) T.g to minimize the segmentation error between predicted H, and the computed groundtruth H, The probability of focal loss is set Cy.
3.4 Refinement
Following 20), to enhance efficiency, only one of the points within each unit j is selected for refinement, which we denote the source and target point as P5 and Pt. Given PJ = HH the refinement stage finds the offset A of each target point Pj relative to fixed source point P5. With the feature f3 from Ms. conventional techniques unfold features f from local regions in both source image and target image, followed by self-attention and cross- attsstantnnion. We claim that this process is unnecessarily slow. Here we eliminate self-attention and reduce cross-attention from bi-directional process to a uni-di uni-directional one. Specificallyy, the feature f is computed by querying the features fü within the neighborhood of the original target point Pj. and become the final feature vectors f for the point P3 on the images. We illustrate this process in supplementary materials. With the proof of experiments in Sec. 4.4, we find that refining single feature in the local region of M2 is enough to get expected results. Our findings indicate that this approach can largely diminish the computational load of attention mechanisms while still preserving highly accurate matching outcomes.
Finally, following 28 we process one fixed element of the final feature vector f² via cross-attention
as the confidence score ey for the corresponding set of matches. supervised by if the error final match is larger than threshold. The coordinates for the matched pairs on the source and target
5

---

images are (Pj. Pj)) Therefore we can define the supervision of refinement as:
L, = -Pj - P BCE())))
(4)
where BCE is binary cross entropy classification problems. Here we use BCE to recognize reliable matches. Here P, is the ground truth value of ps*
60 which is a commonly utilized loss function for binary
3.5 Supervision
Indirect supervision for the homography hypothesis. Instead supervising the attributes of hypothesis directly, our approach employs indirect supervision by monitoring the correspondences of sampling points that are linked via the homography transformation. This design offers the advantage of leveraging excessive number of ground truth matches to efficiently train network focused on estimating a set of homography parameters. Using the ground truth camera pose and depth from datasets, we can get real matched points T in source and F' in target images. We sample the select only 3x3 adjacent hypotheses
matched points train our method. For each points p in P = (H,,...... Hg] around it on M where Hs is in the hypothesis region's center and represents the hypothesis of the region containing point p. Similar to the necessity of segmentation stated in Sec. B.3 direct supervision which applies Hş to every p could result in avoidable errors, which arise from the mismatch between the irregular boundaries of planes in the real world and the grid- structurddd unit i on the source image. However, given that these sampling points are merely an auxiliary tool for the loss function, we can directly utilize the ground truth coordinates of the matches supervise and eliminate classifying each sampling point at this stage. For each point p. we assume that satisfies a certain homography transformation p. and Hp satisfies the following defination:
H ar orggmnn H,p" -p" p' HEEH
(5)
We denote in the source image as pd and target image as p', we use the following error to optimize method:
Ep = 1Hp - p'11.
where #|1 is the L1 norm error.
Classificatton or Correspondence Loss for Hypotheses. H, calculated on the base identifying the matched unit of for unit i in target image. Therefore, if the estimated a significantly deviates from the ground truth aj. H₃ would be entirely incorrect. In such conditions, loss to enhance the feature similarity between at and ai In the opposite case, directly supervising the point correspondences calculated through H, yields better results. The methodology is detailed follows:
classification
Q1 {il0] < la 0, oo
Q2 = {il0 2 la - b,
CosSim faa))
eiE q1.
Lg
Cp
i Q2:
PEP:
where oc denotes the computation of infinity norm, CosSim is the cosine similarity of two features in the feature map M₁ P: is the set of sampled points that apply H; as H.
Two-stage Training Process. In the entire feature matching process, we divide the training process into two stages, the coarse stage and the fine stage. The fine stage will freeze all the parameters of the coarse stage during training. The coarse stage includes the homography hypotheses estimation, while the fine stage includes segmentation and refinement. The losses used in these two parts are:
Leourse LH
(8)
Lfine = La + Ly.

---

Average Corner Error
Point Accuracy
time
Methods
1px(5)
3px(%)
Spx(%)
Ipx(%)
mis
LoFTR [20
46
77
86
63
218
SP 16 +LG 22
44
73
85
51
101
SP 16
+SG
41
SP 16 +search
38
68
81
32
81
Ours
42
72
82
52
53
Table 1: Evaluation HPatches
[33] for homography estimation.
3.6 Implementation Details
For feature extracting, We implement mni-iircttioall tross-attention once in the process of refinement. We train our outdoor model and indoor model respectively. The outdoor model is trained on the Megadepth [30] dataset, while the indoor model is trained on a mixed dataset of Megadepth and ScanNet [32]. The training process divided into three stages: the first stage is training on data of 640x480 resolution; in the second stage, the longer side is scaled to 640, and some images are rotated by 90 degrees for adaptation training for the coarse; the third stage involves training the fine on 640x480 data. The learning rate used in the first stage is le-4. In the second stage is 5e-5, and in the third stage is 3e-4. Both models are trained using three RTX 3090 for 80 hours, with a batch size of 24 the first stage and 16 in both the second and third stages. We perform all inferences using PyTorch, merely following the implementation of LightGlue to pre-compile the transformer in the coarse stage with PyTorch.
Resnet-18 [36. then we perform transformer 21 five times Mi-
Experiment
We conducted these evaluations on four different datasets for outdoor and indoor relative pose estimation and homography estimation. These experiments demonstrate superior performance various downstream tasks.
4.1 Homography Estimation
As our first experiment, we evaluate our quality correspondences and the ability to fit the homogra- phy matrix for planar scenes the HPatches 33 dataset.
Experimental Setup. We conducted comparative experiments using the image matching toolbox pro- posed by [61. Our experiments were configured to replicate the settings outlined for SuperPoint[16]
SuperGlue [36], and LoFTR (20) as shown in this toolbox. For LightGlue [29], we follow their open-source code settings. To estimate the homography, we employed the RANSAC algorithm with a threshold 0.25 pixels, leveraging the OpenCV library. To comprehensively assess the performance of each method, we considered three key metrics: the proportion of matched points with an error within a l-pixel threshold, the average corner distance for estimated homography matrices measuring less than 1/3/5 pixels, and the average computational time. These metrics were chosen to simultaneously evaluate the matching accuracy. homography estimation precision, and computational efficiency of the methods. We perform this experiment acceleration options for pytorch implementations, such as flash attention and precompilation. In order get as close as possible real usage scenario, here, do not warm-up operation when measuring the computing speed.
RTX2070 GPU, and turn off all
Dataset. HPatches [33] contains 52 sequences under significant illumination changes and 56 se- quences that exhibit large variations viewpoints. All images are resized with longer dimensions equal to 640.
Results. We compare ETO with SuperPoint [16). SuperGlue 36) LightGlue 29 and LoFTR 20 According to Table. | our experimental results demonstrate that our method excels in homography estimation accuracy compared to SuperGlue, achieving lower errors within 1 l-pixel threshold when compared to both SuperGlue and LightGlue. Furthermore, our approach is significantly faster, outpacing other methods several times in the evaluation.
4.2 Outdoor Pose Estimation
We assess the efficacy of our approach relative pose estimation in the same setting using two distinct datasets: YFCC100M [3 and Megadepth (30) for outdoor scenes.

---

Megadepth
YFCC100M
Methods
@
T0
@020
ms
I0
@20
ASpanFormer
19
58.6
72.2
81.7
158.5
44.5
63.5
78.1
155.5
Quadtree
[43
58.6
72.1
81.5
147.6
44.7
63.9
78.2
159,4
LoFTR 20
57.5
71.2
80.8
93.2
44.7
63.6
78.3
96.3
SP
16
+L.G
20
51.5
67.7
78.9
64.2
36.1
56.2
73.1
60.8
SP 16
+I.
(2
47.1
64.0
77.3
26.9
29.2
48.8
67.0
27.2
SP
16
+SG
36
43,2
60.0
72.8
43.9
29.7
49.6
67.9
48.7
SP
16
+search
28.8
43,4
56,6
23.7
14.0
27.0
42.2
24.6
RoMa
62
64.8
77.4
86.1
689
Tiny-RoMa 62
36.2
53.6
67.5
29.0
Ours
51.7
66.6
77.4
21.0
44.8
64.0
78.8
22.1
Table 2: Evaluation on Megadepth [30] and YFCC100M [31 for outdoor pose estimation.
Experimental setup. We report the pose accuracy in terms of AUC metric at multiple thresholds (5 "9,100,201 )) and runtime for every approach, and the RANSAC threshold here is set as 0.25 pixel for all methods. All of the evaluations here are conducted on a RTX2080ti. We turn on flash-attention for LightGlue and turn on the pre-compilation to accelerate the transformer for LightGluee and ETO. Here LightGlue (29) slower than SuperGlue [36] for the reason that following the default configuration LightGlue extracts 2048 keypoints and resizes the resolution of images to 1024, while SuperGlue extracts only 1024 keypoints and keep the resolution of images as the same. LightGlue* apply the setting of SuperGlue. It is imperative to highlight that our method encompasses 4800 points to be matched here, which is the same as LoFTR (20), ASpanFormer [9), and Quadtree 4 ensure
an accurate representation of the actual computation speed, initiate a warm-up phase for each method, consisting of 10 iterations, prior to conducting mcasurements.
Dataset. YFCC100M 33 encompasses an extensive repository comprising 100 million media assets. For our evaluation, we following (36] and focus on a subset of YFCC100M, specifically four handpicked image collections featuring prominent landmarks, in accordance with the criteria outlined in [36] and [20]. MegaDepth, on the other hand, comprises dataset containing one million Internet-sourced images depicting 196 distinct outdoor scenes. To ensure the integrity of our evaluation protocol, in line with the guidelines presented in [8] we randomly select 1000 image pairs, guaranteeing that none of these pairs have been used in the training processes of any existing methods. All images in Megadepth and YFCC100M are resized with a resolution equal to 640*480.
Results. We compare ETO with SuperPoint [16], SuperGlue (36). LightGlue 229] LoFTR (20).
ASpanFormer [19] and Quadtree 43. According the results shown in Table on the easier outdoor cases in MegaDepth, the accuracy of our method for pose estimation is lower than advanced detector-free method but is higher than any detector- based approaches, while our runtime is at most 23% of the detector- free methods, 81% % of the detector- methods. While on the more difficult outdoor cases in YFCC100M, the performance our model much better than detector-based methods and is comparable with detector-free methods. And still, our superiority on runtime is preserved.
methods and 90% of the CNN-based
4.3 Indoor Pose Estimation
We evaluate our method for indoor pose estimation with ScanNet- 32 following 36 20
Experimental Setup. Just like outdoor cases, we report the pose accuracy in terms of the AUC metric at multiple thresholds 10"00000000 .220)) and runtime for every approach. However, here we set all of the RANSAC thresholds 0.5 pixels. All of the images resized with longer dimensions equal to 640. This evaluation is conducted on RTX2080ti. We have done a warm-up here in measuring the efficiency.
Dataset. The ScanNet dataset represents a comprehensive indoor RGB-D collection encompassing 1,613 distinct sequences that cumulatively offer 2.5 million unique views. Each view within this dataset is meticulously annotated with a corresponding ground truth camera pose and depth map. We follow the same training and testing split used by [36].
Results. We compare our approach with SuperPoint 16). SuperGlue [6] LoFTR 20 ASpan-
Former 19 and Quadtree [43. The results are demonstrated in Table 3 We find that our results are comparable with LoFTR and are superior to SuperPoint+search and SuperPoint--SuperGine, while much faster than any other methods.
8

---

Pose estimation AUC
average time
Methods
@
@ 10
@22
ASpanFormer [19
24.5
45.0
62.8
160.0
Quadtree [43
23.9
43.0
60.2
145.9
LoFTR (20
21.4
40.3
57.2
94.2
SuperPoint 16
SSpeerGluee
36
13.7
29.8
47.2
63.1
Superpoint 16 +scarch
8.0
18.3
29.8
27.1
Ours
20.1
40.4
59.8
24.2
Table 3: Evaluation on Scannet
[32]
for indoor pose estimation.
Pose estimation
AUC
time
Ablation
@5
@ 10
@20
Base32 w/o Homography
21.3
36.3
8.5
Base8 w/o Homography (LoFTR coarse)
25.7
41.8
57.7
58.9
Base32 w/ Homography
28.5
46.2
61,4
8.5
Basic Refinement w/ segmentation
51.0
66.6
77.6
32.8
Uni-directional w/o segmentation
42.1
59.0
72.0
21.2
Full
51.7
66.6
77.4
22.0
Table 4: Ablation study based on Megadepth [30] for outdoor pose estimation.
4.4
Ablation Studies
To evaluate the impact of each design component the overall structure, we perform ablation study using the MegaDepth dataset. We systematically add each design element one at a time. The quantitative results presented Table 4
Base32 w/o Homography We match the units on M1 at H/32 W/32 resolution and permit the target of unit centroid to be continuous, and the unit are still fixed. We output four virtual correspondences as matches. While it offers rapid processing, it does not achieve high level of accuracy.
compute it as Section. 3.2 while other parameters for
Base8 w/o Homography We set this ablation experiment as the coarse matching of LoFTR (2)] Here we match every possible 8 units. We output the center of corresponding units as matches. It more accurate but too slow.
Base32 w/ Homography. Following Section. 3.2 we estimate the whole homography matrix and output four virtual correspondences matches. performs better than the coarse matching of LoFTR 20 while providing higher efficiency at the same time,
Basic Refinement w/ Segmentation. Following [20]. we set a layer of transformer between 25 tokens on these two images and try to refine our results while the transformer is trained for 12 hours, which is the same as the training time of our uni-directional attention for the refinement stage. While full attention execution speed is considerably slower than that of uni-directional attention, its accuracy merely comparable with the latter.
miidiirettinnll w/o Segmentation. Here we directly choose the homography hypotheses Hs for each unit j which is in the center. Then, we conduct the refinement as the same. The results show that the segmentation stage significantly improves the accuracy.
Conclusion and Limitations.
In this paper, we propose Efficient Transformer-basse Local Feature Matching by Organizing Multiple Homography hypotheses (ETO). ETO tries to approximate continuous corresponding function with multiple homography hypotheses with fewer tokens fed to the transformer. Multiple datasets demonstrate that ETO delivers nearly comparable performance in relative pose estimation and homography estimation with other transformer-basee them by large margin. However, there remains significant space for improvement in ETO's matching accuracy. Next, we could explore end-to-end training mode, which would allow for further enhancement of the feature extractor at fine- ine-grained level. Moreover, we believe that intermediate- features can provide not only segmentation information but also data conducive to more precise matching. Finally, the form of parametric scheme we present here may not be optimal and complete for homography transformation, so we will continue to explore better parametric schemes. These strategies are expected to enable our method to compete with approaches like PATS 18] and DKM [45] in terms of matching precision, without considerably compromising speed. Acknowledgements. This work was partially supported by NSF of China (No. 61932003).
methods, while its speed surpasses all of

---

References
[1] Ethan Rublee, Vincent Rabaud, Kurt Konolige, and Gary R. Bradski. ORB: an efficient alternative to SIFT SURF. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 2564-2571. IEEE, 2011.
[2] Herbert Bay, Andreas Ess, Tinne Tuytelaars, and Luc Van Gool. Speeded-uy up robust features (SURF). Comput. Vis. Image Underst., 11(3))346-359) 2008.
[3] Raul Mur-Artal, J. M. M. Montiel, and Juan D. Tardós. ORB-SLAM: A versatile and accurate monocular SLAM system. IEEE Trans. Robotics, 31(5): 147-1163, 2015.
[4] Xingrui Yang, Hai Li, Hongjia Zhai, Yuhang Ming, Yuqian Liu, and Guofeng Zhang. Vox- fusion: Dense tracking and mapping with voxel-based neural implicit representation. In IEEE International Symposium Mixed and Augmented Reality, pages 499-507. IEEE, 2022.
[5] Xinyang Liu, Yijin Li, Yanbin Teng, Hujun Bao, Guofeng Zhang, Yinda Zhang, and Zhaopeng Cui. Multi-modal neural radiance field for monocular dense slam with a light-weight tof sensor. In Proceedings of the ieee/cvf international conference computer vision, pages 1-11, 2023.
[6] Jiarui Hu, Xianhao Chen, Boyin Feng, Guanglin Li, Liangjing Yang, Hujun Bao, Guofeng Zhang. and Zhaopeng Cui. Cg-slam: Efficient dense rgb-d slam a consistent uncertainty-aware 3d gaussian field. arXiv preprint arXiv:2403. 1609,, 2024.
[7] Jiarui Hu, Mao Mao, Hujun Bao, Guofeng Zhang, and Zhaopeng Cui. Cp-slam: Collaborative neural point-based slam system, Advances in Neural Information Processing Systems, 36, 2024.
[8] Hongjia Zhai, Gan Huang, Qirui Hu, Guanglin Li, Hujun Bao, and Guofeng Zhang. Nis-slam: Neural implicit semantic rgb-d slam for 3d consistent scene understanding. IEEE Transactions on Visualization and Computer Graphics, pages 1-11, 2024.
[9] Shuo Chen, Mao Peng, Yijin Li, Bing-Feng Ju, Hujun Bao, Yuan-Liu Chen, and Guofeng Zhang. Multi-view neural 3d reconstruction of micro-and nanostructures with atomic force microscopy. Communications Engineering, 3(1):131, 2024.
[10] Bangbang Yang. Yinda Zhang, Yijin Li, Zhaopeng Cui, Sean Fanello, Hujun Bao, and Guofeng Zhang. Neural rendering in a room: amodal 3d understanding and free-viewpoint rendering for the closed scene composed of pre-captured objects. ACM Transactions Graphics (TOG), 41 (4):1-10, 2022.
[11] -dou-dd Sarlin, Cesar Cadena, Roland Siegwart, and Marcin Dymezyk. From coarse to fine: Robust hierarchical localization at large scale. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 12725, 2019.
[12] Zhaoyang Huang, Han Zhou, Yijin Li, Bangbang Yang, Yan Xu, Xiaowei Zhou, Hujun Bao, Guofeng Zhang, and Hongsheng Li. Vs-net: Voting with segmentation for visual localization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 6101-6111, 2021
[13] Hongjia Zhai, Xiyu Zhang, Zhao Boming, Hai Li, Yijia He, Zhaopeng Cui, Hujun Bao, and Guofeng Zhang. Splatloc: 3d gaussian aplatting-based visual localization for augmented reality. arXiv preprini arXiv:2409. 1444099.4067,. 2024.
[14] Yan Xu, Kwan- Yee Lin, Guofeng Zhang, Xiaogang Wang, and Hongsheng Li. Rnnpose: Recurrent 6-dof object pose refinement with robust correspondence field estimation and pose optimization. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14880-14890, 2022.
[15] Guanglin Li, Yifeng Li. Zhichao Yc. Qihang Zhang, Tao Kong. Zhaopeng Cui, and Guofeng Zhang. Generative cetegory-level shape and pose estimation with semantic primitives. In Conference Robot Learning, pages 1990-1400. PMLR, 2022.
[16] Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. Superpoint: Self- interest point detection and description. Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 224-236, 2018
10

---

[17] Mihai Dusmanu, Ignacio Rocco, Tomás Pajdla, Mare Pollefeys, Josef Sivic, Akihiko Torii, and Torsten Sattler. D2-net: A trainable CNN for joint description and detection of local features. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8092-8101, 2019.
[18] Junjie Ni, Yijin Li, Zhaoyang Huang, Hongsheng Li, Hujun Bao, Zhaopeng Cui, and Guofeng Zhang. Pats: Patch area transportation with subdivision for local feature matching. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17776-17786, 2023.
[19] Hongkai Chen, Zixin Luo, Lei Zhou, Yurun Tian, Mingmin Zhen, Tian Fang, David Mckinmon, Yanghai Tsin, and Long Quan. Aspanformer: Detector-free image matching with adaptive span transformer. In European Conference on Computer Vision, 2022.
[20] Jiaming Sun, Zehong Shen. Yuang Wang, Hujun Bao, and Xiaowei Zhou. Loftr: Detector-free local feature matching with transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 8922 8931, 2021.
[21] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems, 30, 2017.
[22] Angelos Katharopooloss, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are mms:: Fast autoregressive transformers with linear attention. In International conference on machine learming, pages 5156-5165, PMLR, 2020.
[23] Adam W Harley, Zhaoyuan Fang, and Katerina Fragkiadaki. Particle video revisited: Tracking through occlusions using point trajectories. In European Conference on Computer Vision, pages 59-75. Springer. 2022.
[24] Yijin Li, Yichen Shen, Zhaoyang Huang, Shuo Chen, Weikang Bian, Xiaoyu Shi, Fu-Yun Wang, Keqiang Sun, Hujun Bao, Zhaopeng Cui, Guofeng Zhang, and Hongsheng Li. Blinkvision: benchmark for optical flow, scene flow and point tracking estimation using rgb frames and events. In European conference computer vision. Springer, 2024.
[25] Weikang Bian, Zhaoyang Huang, Xiaoyu Shi, Yitong Dong, Yijin Li, and Hongsheng Li. Context-pips: Persistent independent particles demands context features. Advances in Neural Information Processing Systems, 36, 2024.
[26] Paul-Fdouard Sarlin, Cesar Cadena, Roland Siegwart, and Marcin Dymezyk. From coarse to fine: Robust hierarchical localization large scale. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12716-12725, 2019.
[27] Connelly Barnes, Eli Shechtman, Adam Finkelstein, and Dan Goldman. Patchmatch: A randomized correspondence algorithm for structural image editing. ACM Trans. Graph., 28(3): 24, 2009.
[28] Joseph Redmon and Ali Farhadi. Yolov3: An incremental improvement. arXiv preprint 1oX10:1804.02767, 2018.
[29] Philipp Lindenberger, Paul-Edouard Sarlin, and Mare Pollefeys. LightGlue: Local Feature Matching at Light Speed. In ICCV, 2023.
[30] Zhengqi Li and Noah Snavely. Megadepth: Learning single-view depth prediction from internet photos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018.
[31] Bart Thomee, David A. Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. YFCC100M: the new data in multimedia research. Commun. ACM, 59(2):64-73 2016.
[32] Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas A. Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In Pro- ceedings of the IEEF/VVF Conference on Computer Vision and Pattern Recognition, pages 2017.
11

---

[33] Vassileios Balntas, Karel Lenc, Andrea Vedaldi, and Krystian Mikolajczyk. Hpatches: A benchmark and evaluation handerafted and learned local descriptors. In CVPR, 2017.
[34] David G. Lowe. Distinctive image features from 60(2):91-110, 2004.
keypoints. Int. J. Comput. Vis.,
[35] Jérôme Revaud, César Roberto de Souza, Martin Humenberger, and Philippe Weinzaepfel. R2D2: reliable and repeatable detector and descriptor. Advances Neural Information Pro- cessing Systems, 32, 2019.
[36] Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewiez, and Andrew Rabinovich. Superglue: Learning feature matching with graph neural networks. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 1938-4947,2 2020.
[37] Kwang Moo Yi, Eduard Trulls, Yuki Ono, Vincent Lepetit, Mathieu Salzmann, and Pascal Fua. Learning to find good correspondences. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2666-2674, 2018
[38] Wei Jiang, Eduard Trulls, Jan Hosang, Andrea Tagliasacchi, and Kwang Moo Yi. COTR: correspondence transformer for matching across images. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 171-180. Springer, 2021.
[39] Dongli Tan, Jiang-Jiang Liu, Xingyu Chen, Chao Chen, Ruixin Zhang, Yunhang Shen, Shouhong Ding, and Rongrong Ji. Eco-tr: Efficient correspondences finding via coarse-to-fin refinement. In European Conference Computer Vision, pages 317-334. Springer, 2022.
[40] Yijin Li, Xinyang Liu, Wenqi Dong, Han Zhou, Hujun Bao, Guofeng Zhang, Yinda Zhang, and Zhaopeng Cui. DELTAR: depth estimation from lightt tof sensor and RGB image. In European Conference on Computer Vision, pages 619-636. Springer, 2022.
[41] Zhaoyang Huang, Xiaoyu Shi, Chao Zhang, Qiang Wang, Yijin Li, Hongwei Qin, Jifeng Dai, Xiaogang Wang, and Hongsheng Li. Flowformer: A transformer architecture and its masked cost volume autoencoding for optical flow. arXiv preprint arXiv:2306.. 05442, 2023.
[42] Yijin Li, Zhaoyang Huang, Shuo Chen, Xiaoyu Shi, Hongsheng Li, Hujun Bao, Zhaopeng Cui, and Guofeng Zhang. Blinkflow: A dataset to push the limits of event-based optical flow estimation. In 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 3881 3881-3888. IEEE, 2023.
[43] Shitao Tang, Jiahui Zhang, Siyu Zhu, and Ping Tan. Quadtree attention for vision transformers. In The International Conference on Learning Representations. OpenReview.net, 2021.
[44] Prune Truong, Martin Danelljan, Radu Timofte, and Lue Van Gool. Pdc-nett: Enhanced probabilistic dense correspondence network. arXiv preprint arXiv:2109. 2021.
[45] Johan Edstedt, Ioannis Athanasiadis, Märten Wadenbäck, and Michael Felsberg. Dkm: Dense kernelized feature matching for geometry estimation. In Proceedings of the IEÉE/CVF Confer- ence on Computer Vision and Pattern Recognition, pages 17765-17775, 2023.
[46] Songyan Zhang, Xinyu Sun, Hao Chen, Bo Li, and Chunhua Shen. Rgm: A robust generalist matching model. arXiv preprint arXiv:2310.11755 2023.
[47] Zachary Teed and Jia Deng. RAFT: recurrent all all-pairs field transforms for optical flow. In European Conference on Computer Vision, pages 402-419. Springer, 2020.
[48] Zixin Luo, Lei Zhou, Xuyang Bai, Hongkai Chen, Jiahui Zhang, Yao Yao, Shiwei Li, Tian Fang, and Long Quan. Aslfeat: Learning local features of accurate shape and localization. In Proceedings of the JEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 6589-6598. 2020.
[49] Tony Lindeberg. Feature detection with automatic scale selection. Int. Comput. Vis., 30(2): 79-116, 16, 1998.
12

---

[50] Axel Barroso Barroso-Laguna, Laguna, Yurun Tian, and Krystian Mikolajezyk. Scalenet: A shallow architecture for scale estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 12808-12811 2022.
[51] Miao Fan, Mingrui Chen, Chen Hu, and Shuchang Zhou. Occ" 2net: Robust image matching based on 3d occupancy estimation for occluded regions. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 9652-96623 2023.
[52] Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser, Uni- versal transformers. arXiv preprint arXiv:/807.0388 03809, 2018.
[53] Maha Elbayad, Jiatao Gu, Edouard Grave, and Michael Auli. Depth-adaptive transformer. arXiv preprint UrXiv:1910.10077, 2019.
[54] Yifan Wang, Xingyi He, Sida Peng, Dongli Tan, and Xiaowei Zhou, Efficient loftr: Semi-dense local feature matching with sparse-like speed. arXiv preprint arXiv:2403.04 04.665 2024.
[55] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assiste CntvvvettiooMMMCCCA 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pages 234-241. Springer, 2015.
[56] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and paltern recognition, pages 770-778, 2016.
[57] Marco Cuturi.
Sinkhorn distances: Lightspeed computation of optimal transport. Advances in Neural Information Processing Systems, 26, 2013.
[58] Xiaoyang Guo, Kai Yang, Wukui Yang, Xiaogang Wang, and Hongsheng Li. Group-wise correlation stereo network. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 3273-3282, 2019.
[59] Tsung-Yl Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pages 2980-2988, 2017.
[60] Irving John Good, Rational decisions. Journal of the Royal Statistical Society: Series B (Methodological), 14(1):107-114, 1952.
[61] Qunjie Zhou, Torsten Sattler, and Laura Leal-Taixé. Patch2pix: Epipolar-guides pixel-level correspondences. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 4669-4678 2021.
[62] Johan Edstedt, Qiyu Sun, Georg Bökman, Märten Wadenbäck, and Michael Felsberg. Roma: Robust dense feature matching. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 19990-19800,00 2024.
13

---

Checklist
1. Claims: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Claims in the paper should match theoretical and ex- perimental results in terms of how much the results can be expected to generalize. The paper's contributions should be clearly stated in the abstract and introduction, along with any important assumptions and limitations. It is fine include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. Enter yes, no, or an explanation if appropriate. Answers are visible to reviewers. Claims the paper should match theoretical and experimental results in terms of how much the results can be expected to generalize. [Ys]] The main claims are clearly stated in the abstract and introduction.
2. Limitations: The authors are encouraged create a separate "Limitations" section in their paper. The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification asymptotic approximations only holding locally). [Yes] Limitations are discussed in Sec.
3. Theory, Assumptions and Proofs: If you are including theoretical results, did you state the full set of assumptions of all theoretical results, and did you include complete proofs of all theoretical results? All assumptions should be clearly stated referenced in the statement of any theorems. The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, authors are encouraged to provide a short proof sketch to provide intuition. Enter yes, no, n/a, or an explanation if appropriate. Answers are visible to reviewers. [Yes] We provide simple proof for using homography hypotheses in supplementary materials.
4. Experimental Result Reproducibility: If the contribution a dataset or model, what steps did you take to make your results reproducible or verifiable? Depending reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility also be provided via detailed instructions for how to replicate the results, access to hosted model (e.g., in the case of a large language model), release a model checkpoint, or other means that appropriate your research. Enter yes, no, n/a, or an explanation if appropriate. Answers are visible to reviewers. [Yes]
the contribution,
5. Open Access to Data and Code: If you ran experiments, did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as URL)? Please see the NeurIPS code and data submission guidelines for more details. While we encourage release of code and data, we understand that this might not be possible, so no is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g.. for new open-source benchmark). At submission time, to preserve anonymity, remember to release anonymized versions. Enter yes, no, n/a,
explanation if appropriate. Answers are visible to reviewers. [No]
6. Experimental Setting/ Details: If you ran experiments, did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? The full details can be provided with the code, but the important details should be in the main paper, and information about how hyperparameters were selected should appear either in the paper or supplementary materials. Enter yes, no, n/a, or an explanation if appropriate. Answers are visible to reviewers. [Yes]
7. Experiment Statistical Significance: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
[No]
8. Experiments Compute Resource: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments? [Yes]
14

---

9. Code Of Ethics: Have you read the NeurIPS Code of Ethics and ensured that your research conforms it? Enter yes, no, or an explanation if appropriate. Answers are visible to reviewers. [Yes]
10. Broader Impacts: If appropriate for the scope and focus of your paper, did you discuss potential negative societal impacts of your work? Please the Paper Checklist Guidelines for detailed instructions and examples of points that you may choose to discuss. Enter yes, no, n/a, or an explanation appropriate. Answers are visible reviewers. [No]
11. Safeguards: Do you have safeguards in place for responsible release of models with high risk for misuse (e.g., pretrained language models)? Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use the model, for example by requiring that users adhere to usage guidelines or restrictions access the model. Enter yes, no, n/a, or an explanation if appropriate. Answers visible to reviewers, [Yes]
12. Licenses: If you are using existing assets (e.g., code, data, models), did you cite the creators and respect the license and terms of use? Cite the original paper that produced the code package or dataset. possible, include a URL. Be sure to check the original license and respect its conditions. Enter yes, no, n/a, or explanation if appropriate. Answers visible to reviewers. [Yes]
13. Assets: you are releasing new assets, did you document them and provide these details alongside the assets? Researchers should communicate the details of the dataset or the model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. Enter yes, no, n/a, an explanation if appropriate. Answers are visible to reviewers. [NA]
14. Crowdsoureing and Research with Human Subjects: If you used crowdsoureing or con- ducted research with human subjects, did you include the full text of instructions given to participants and sereenshots, if applicable, as well as details about compensation (if any)? Including this information in the supplemental material is fine, but if the main contribution your paper involves human subjects, then we strongly encourage you to include as much detail possible in the main paper. According to the NeurlPS Code of Ethics, you must pay workers involved data collection, curation, or other labor at least the minimum wage in your country. Enter yes, no, n/a, explanation if appropriate. Answers visible to reviewers. [NA]
15. IRB Approvals: Did you describe any potential participant risks and obtain Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements your institution), applicable? Depending on the country which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this the paper. For initial submissions, do not include any information that would break anonymity, such the institution conducting the review. Enter yes, no, n/a, or an explanation if appropriate. Answers are visible to reviewers. [NA]
15


Reviews:
[{"review_id": "yd26T1IUW1", "reviewer": "Reviewer_yFZw", "summary": "This paper propose an efficient framework to reduce the computational load of transformer-based matching approaches.  It is a coarse-to-fine two-stage solution. During the coarse matching phase, multiple homography hypotheses are estimated to approximate continuous matches. Each hypothesis encompasses few features to be matched, reducing the number of features that require enhancement via transformers. In the refinement stage, the unidirectional cross-attention is proposed to replace the classical bidirectional self-attention and cross-attention mechanism, which further decreases the cost of computation. Comprehensive evaluations on other open datasets such as Megadepth, YFCC100M, ScanNet, and HPatches are presented to demonstrate its efficacy.", "questions": "1. I think some descriptions in Sec 3.4 need further clarification, e.g., Line 192-193 $\\hat{f^3_j}$ is computed by querying the $f3_k$ with …, is the query done by nearest neighboring searching?\n2. How is $\\triangle P_j^t$ in Line 188 obtained or is it predicted?\n3. In Figure 3, is there an MLP after cross-attention block? If yes, what is the input/output.  Also, the presence of “Module 3” is confusing, should it contain the cross attention?\n4. In Table 4, it seems the usage of bidirectional cross-attention performs worse than that of uni-directional cross-attention at 5. Is that reasonable, please give some analysis. \n5. From Table 2, the performance of lightGlue is much worse than ASpanFormer, LoFTR. Interestingly, ASpanFormer is better than LoFTR with a small margin in the Megadepth dataset, similar to the results reported in LightGlue. The performance gap between LightGlue and LoFTR is much larger than that in LightGlue, could you please explain? Moreover, why LightGlue is not reported in the dataset of Scannet?", "limitations": "1. Some of the network architectures are not depicted in figures, making it difficult to know the overall diagram of the method.\n2. The two-stage training process may not be easily trained.\n3. It would be better to present some failure case studies.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The whole idea makes sense to me. The experimental studies are quite comprehensive.", "weaknesses": "Some parts of the description are quite vague to me, which may hinder its reproducibility. Some of the experimental results may need further clarification.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and valuable comments.\n\n1. Some descriptions in Sec 3.4 need further clarification.\n\nYes. Due to space limitations, we have included a more intuitive description and illustrations of this step in Sec.2 in the supplementary materials. Specifically, since the corresponding point $P_j^s$ on the source image is self-chosen,  following LoFTR and for computational convenience, we select a point $P_j^s$ at the center of each specific feature $\\hat{f}_j^3$ on the $M_3$ feature map. For the target image, after calculating the initial $P_j^t$ coordinates through the accepted $H$ hypotheses, we query the 7*7 features around $P_j^t$ with $\\hat{f}_j^3$. We then perform cross attention between this single feature from the source image and the 49 features from the target image. If we were to use traditional methods to perform a complete transformer, we would need a 49*49 transformer to achieve such a large receptive field, but here we only need a 1*49 transformer.\n\n2. How is $\\Delta P_j^t$ in Line 188 obtained or is it predicted?\n\nAfter refining $\\hat{f}_j^3$ using the above method, we follow traditional methods (such as LoFTR) to correlate this feature with the 49 features on the target image, we obtain a local heatmap representing the matching probability for each feature. By computing the expectation over this probability distribution, we can obtain the final offset for matches.\n\n3. In Figure 3, is there an MLP after cross-attention block? If yes, what is the input/output. Also, the presence of “Module 3” is confusing, should it contain the cross attention?\n\nHere, Module 3 refers to the entire refinement part, including the cross-attention part. The MLP shown in the figure follows traditional methods(like PATS, ASpanFormer or LoFTR) in the module of transformer, passing the feature through an MLP module before computing the expectation. Its function is to further enhance the $\\hat{f}_j^3$ mentioned in Section 3.4.\n\n4. In Table 4, it seems the usage of bidirectional cross-attention performs worse than that of uni-directional cross-attention at 5. Is that reasonable, please give some analysis.\n\nThis is because we are using the same training time rather than the same number of training samples as the comparison criterion, which makes a small but not significant difference to the effectiveness of the model. If we had used the same training samples, auc@5 would have achieved 52.3, which is better ours though while still much slower.\n\n5. From Table 2, the performance of lightGlue is much worse than ASpanFormer, LoFTR. Interestingly, ASpanFormer is better than LoFTR with a small margin in the Megadepth dataset, similar to the results reported in LightGlue. The performance gap between LightGlue and LoFTR is much larger than that in LightGlue, could you please explain? Moreover, why LightGlue is not reported in the dataset of Scannet?\n\nIt is important to note that the LightGlue employs two RANSAC methods in their paper: Lo-RANSAC and OpenCV's RANSAC. We use OpenCV's RANSAC exclusively. In the experiments of the LightGlue, it is evident that under OpenCV's RANSAC, LightGlue's performance significantly decreases. Additionally, to better demonstrate the impact of sub-pixel precision, we use a RANSAC threshold of 0.25 in outdoor scenarios. Under this threshold, LoFTR, ASpanFormer, and our method ETO achieve smaller errors within this precision range, further widening the gap with LightGlue. The reason LightGlue was not tested on ScanNet is that its model was not trained on indoor scenes, which would be unfair.\n\n6. Some of the network architectures are not depicted in figures, making it difficult to know the overall diagram of the method.\n\nDue to space limitations, we have placed the illustrations of the refinement steps in the supplementary materials.\n\n7. The two-stage training process may not be easily trained.\n\nYes, an end-to-end pipeline may improve our method. But we don't have enough computing resources to complete it.\n\n8. It would be better to present some failure case studies.\n\nThere exists a shortcoming of our approach, which is that in the case of very complex planar assemblages, we provide too few planar proficiencies, and it is also more difficult to distinguish. It can be found in the Figure.1 of our Author Rebuttal.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2ZbUpzi7ER", "reviewer": "Reviewer_uPv2", "summary": "This paper presents a local feature matching method based on the multiple homography hypotheses. The paper explicitly introduces the homography hypotheses for coarse matching, combined with homography segmentation, cross-attention, and sub-pixel refinement to obtain fine matching results. The introduction of the homography hypotheses significantly reduces the number of input tokens for the attention mechanism, and with further model modification, the algorithm proposed in this paper has achieved a substantial increase in inference speed while maintaining performance as much as possible. Experiments on multiple datasets have proven the effectiveness and efficiency of the proposed method.", "questions": "-In Section 3, it is 2-stage training adopted, but in Implementation Details, the training phase contains 3 stages. Why there is a joint training stage at first? Moreover, the joint training is usually conducted for the final finetuning, why it is the first stage in your strategy?\n\n-For Basic Refinement w/ Segmentation in Section 4.4, why it is the same training hours instead of the same training samples? Early stopping may lead to performance drop.\n\n-In the introduction, the authors claim that the multiple self- and cross-attention in the fine-level stage are redundant, is there any numerical results? What if you stack several uni-directional cross-attention in your method?", "limitations": "see Weaknesses", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Clear motivation, reasonable structural design, convincing experiments", "weaknesses": "-The pre-compiled transformer model may have some impact on inference speed, can additional tests be conducted on the efficiency of a normal transformer module?\n\n-Lack of the latest state-of-the-art comparison: Some of the latest methods, such as RoMA[1] and EfficientLoFTR[2], have been made public at an early stage, and thus it is necessary to compare and analyze with these methods.\n\n[1] Edstedt, Johan, et al. \"RoMa: Robust dense feature matching.\" CVPR 2024.\n[2] Wang, Yifan, et al. \"Efficient LoFTR: Semi-dense local feature matching with sparse-like speed.\"  CVPR 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and valuable comments.\n\n1. The pre-compiled transformer model may have some impact on inference speed, can additional tests be conducted on the efficiency of a normal transformer module?\n\nWithout the pre-compiled transformer model, the runtime of ETO will be delayed from 21.0 ms to 32.8 ms for the images of 640*480. While the runtime is almost the same for LightGlue.\n\n2.  Lack of the latest state-of-the-art comparison.\n\nTo be mentioned that our paper is contemporaneous with the Efficient LoFTR. The accuracy of  Efficient LoFTR is a bit better than LoFTR, while the runtime of Efficient LoFTR is 56.9 ms, much faster than LoFTR, which is 93.2 ms, however,  it is much slower than our 21 ms . And while PATS gets extremely good results, it is almost 100 times slower than our method. More detailed results on Megadepth dataset can be found in the supplementary table inside Author Rebuttal.\n\n3. In Section 3, it is 2-stage training adopted, but in Implementation Details, the training phase contains 3 stages. Why there is a joint training stage at first? Moreover, the joint training is usually conducted for the final finetuning, why it is the first stage in your strategy?\n\nOur purples is to handle multi-resolution images better, which is of great significance for real-world applications, although it is not demonstrated in standard experimental data in this paper.This step is placed in the first stage rather than the second, this is because the different resolutions of the images mean that in the coarse matching stage one has to deal with data inputs of variable sizes, which has a relatively large impact on the results. Correspondingly, the fine matching stage can use inputs of fixed size from the coarse stage and is therefore inherently more robust to resolution changes.\n\n4. For Basic Refinement w/ Segmentation in Section 4.4, why it is the same training hours instead of the same training samples? Early stopping may lead to performance drop.\n\nWe try to demonstrate our superiority in training efficiency. While using the same training samples, the accuracy of this method will be 52.3/67.1/77.9 (auc@5/auc@10/auc@20), and the runtime is still 32.8 ms. While our accuracy is 51.7/66.6/77.4 (auc@5/auc@10/auc@20), and the runtime is 21.0 ms.\n\n5. In the introduction, the authors claim that the multiple self- and cross-attention in the fine-level stage are redundant, is there any numerical results? What if you stack several uni-directional cross-attention in your method?\n\nThe numerical results are in given in Q.4, only a simple version of cross attention can get similar accuracy as complete transformer with self-attention and cross attention, while much faster.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ag7yvbH52v", "reviewer": "Reviewer_aEzL", "summary": "This paper proposes a novel framework for efficient transformer-based local feature matching. Transformer-based local feature matching usually contains two stages: a coarse matching stage which applies self-attention and cross-attention on coarse-level features (usually H/8 x W/8) to obtain coarse matches, and a refinement stage that refines the coarse matches in a local window based on fine-grained features. This paper proposed two methods to improve the efficiency of this pipeline. First, the coarse matching is performed on a even coarser level (H/32 x W/32) to reduce the computational cost of the costly attention operations. Additionally, homography hypothesis for each patch is estimated at this level. Leveraging the piece-wise smooth prior, the matching at H/8 x W/8 resolution is directly approximated by selecting the most probable homography hypothesis in local windows using a proposed segmentation technique. Second, in the refinement stage, the bi-directional cross attention is reduced to uni-direction one. Experiments show that the proposed framework achieves comparable performance with less inference time compared with state-of-the-art transformer-based and local feature matching algorithms.", "questions": "1. Do you have any plan to reorganize the symbols?\n2. If the framework is trained end-to-end, will it yield acceptable performance?", "limitations": "The authors have adequately addressed the limitation of this work in the conclusion and limitations section of the paper. Enabling the network to be trained end-to-end, allowing for dense matching and further improving accuracy are my suggestions for directions of improvement.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The idea of using piece-wise smooth prior to accelerate transformer-based feature matching is novel and promising, and the supervision and the homography hypothesis re-selection algorithms are carefully designed.\n2. Comprehensive ablation studies show that the key design features, including homography hypothesis proposal, homography hypothesis re-selection and uni-directional cross attention have positive effects on the performance and are necessary.", "weaknesses": "1. The symbols used in the method part are too complicated, making it hard to read.\n2. The symbol $\\mathscr{H}$ used in figure 4 is undefined in the text.\n3. The framework adopts two-stage training, possibly making it hard to train and limiting the accuracy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and valuable comments.\n\n1. The symbols used in the method part are too complicated, making it hard to read.\n\nThank you for pointing this out and I'm so sorry for it.\n\n2. The symbol  $\\mathscr{H}$ used in figure 4 is undefined in the text.\n\nThanks for pointing this out, the $\\mathscr{H}$  here refers to the output of segmentation, which is the set of homography transformations that each unit on $M_2$ is ultimately assigned to, we will add this explanation.\n\n3. The framework adopts two-stage training, possibly making it hard to train and limiting the accuracy.\n\nYes, an end-to-end pipeline may improve our method. But we don't have enough computing resources to complete it. We will try this idea later in the next work.\n\n4. Do you have any plan to reorganize the symbols?\n\nMaybe I can directly add $s$ and $t$ to the features, from $f_k^3$ to $f_k^{3s}$ and $f_k^{3t}$ , it will be more clear.\n\n5. If the framework is trained end-to-end, will it yield acceptable performance?\n\nThen the batch size will be very small and make the process of training very slow, maybe an end-to-end pipeline will improve our method, but I don't have enough resources to complete it.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4k2P2x5Af3", "reviewer": "Reviewer_7KzZ", "summary": "The authors propose a local feature matching method that leverages homography to accelerate the transformer-based feature matching pipeline. Additionally, they employ unidirectional cross-attention in the refinement stage to further reduce computational overhead. Experimental results demonstrate the efficiency and effectiveness of this approach.", "questions": "No question to the authors.", "limitations": "The authors have addressed the limitations in their paper.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written and easy to understand.\n2. Integrating homography as theoretical guidance into the transformer pipeline is a commendable approach, enhancing the transformer-based pipeline with theoretical support.\n3. Experimental results demonstrate that the proposed method achieves much smaller time usage, validating its efficiency in practice.", "weaknesses": "1. Though the time usage is decrease, the accuracy is also decrease from Tab.1&2&3.\n2. The paper lacks significant citations in feature matching methods, such as Efficient LoFTR[1], RoMa[2]. Some of these works also focus on improving efficiency in feature matching and should be referenced to provide a comprehensive background. Including these methods in the experiments would offer a more thorough comparison of the proposed approach's performance.\n3. Though the authors acknowledge that some other methods (e.g., [14, 38]) are better in certain aspects (line356-357), it would be beneficial to include a comparative analysis with these methods. This would provide a clearer understanding of the strengths and weaknesses of the proposed approach.\n\n[1]. Wang, Yifan, et al. \"Efficient LoFTR: Semi-dense local feature matching with sparse-like speed.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n[2] Edstedt, Johan, et al. \"RoMa: Robust dense feature matching.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and valuable comments.\n\n1. Though the time usage is decrease, the accuracy is also decrease from Tab.1&2&3.\n\nYes, you are right. But we claim that the huge improvement in runtime is valuable in realtime applications such as robotics or SLAM.\n\n2. The paper lacks significant citations in feature matching methods, such as Efficient LoFTR, RoMa. Some of these works also focus on improving efficiency in feature matching and should be referenced to provide a comprehensive background. Including these methods in the experiments would offer a more thorough comparison of the proposed approach's performance.\n\nWe add a couple of experiments on Megadepth dataset in Author Rebuttal materials.  Here, to be mentioned that our paper is contemporaneous with the Efficient LoFTR. The accuracy of Efficient LoFTR is a bit better than LoFTR, while the runtime of Efficient LoFTR is 56.9 ms, much faster than LoFTR, which is 93.2 ms, however, it is much slower than our 21 ms . And while PATS gets extremely good results, it is almost 100 times slower than our method.\n\n3. Though the authors acknowledge that some other methods (e.g., [14, 38]) are better in certain aspects (line356-357), it would be beneficial to include a comparative analysis with these methods. This would provide a clearer understanding of the strengths and weaknesses of the proposed approach.\n \nThe same as Q.2", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7rZqv8IZe8", "reviewer": "Reviewer_LEiZ", "summary": "This paper proposes an efficient transformer-based local feature matching approach named ETO. ETO consists of three steps: hypothesis estimation, segmentation and refinement. The first and second steps can obtain coarse matching results whose resolution is similar to LoFTR’s coarse results. The third step can provide fine-level matching results. The hypothesis estimation performs the Transformer computation on the 1/32 resolution (rather than the 1/8 resolution in LoFTR), which is the core design that reduces the inference time. Experimental results demonstrate that ETO provides a good balance between accuracy and speed.", "questions": "Please provide more discussions and experimental results to address the above weaknesses.", "limitations": "The authors have discussed the limitations.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "1. The motivation of this paper is clear and valuable. Transformer-based approaches are prevalent in the area of feature matching. Nonetheless, it is still an open problem how to design an efficient and accurate transformer-based model.\n2. The technical design of ETO is novel. The manner of predicting the local homography parameters on a very coarse resolution (and then refining them) differs from most of the existing coarse-to-fine transformer-based approaches.\n3. The accuracy of ETO is acceptable, considering its superiority in inference speed.", "weaknesses": "**1. This paper lacks some important technical details.**\n\n1.1 What is the computation manner of the Transformer before the hypothesis estimation step? In line 237, the authors state that \"then we perform transformer five times at M_1\". Does the transformer contain some cross-attention processes? Or does it only involve the self-attention?\n\n1.2 The computation process of the hypothesis estimation step is confusing. The authors seem to use the notations \"M_1, M_2, M_3, f_i^1\" to represent the features of both the source and target images. Such definitions make many statements in Sec. 3.2 hard to understand. For example, the meaning of the variable M_1 seems inconsistent in the statements \"For each unit i on M_1\" (line 144) and \"within the neighborhood of the target units on M_1\" (line 155).\n\n1.3 The computation architecture of the segmentation step is unclear. The authors should provide the computation details on how to predict the classification confidence from the intermediate features. The statement in lines 177-180 is too brief to understand.\n\n1.4 Why is the classification label in the segmentation step termed as the \"pseudo\" ground truth (line 181)? To my understanding, the classification label of every unit should be obtained from the ground truth camera pose and depth. It should be the \"normal ground truth\" rather than the \"pseudo ground truth\" if the above understanding is correct. The authors should clarify this detail.\n\n**2. Some details should be further clarified.**\n\n2.1 The statement \"while we only need to feed 300\" (line 50-51) was not clarified in the subsequent text. Is it the unit number on the 1/32 resolution for a 640x480 input image?\n\n2.2 In the segmentation step, what is the size of output segmentation results for a local input window whose size is 3x3? Is it 4x4? Or 12x12?\n\n2.3 The statement in lines 304-305 should be further discussed. The authors just state that ETO is better on the more difficult YFCC100M without discussing the probable reason.\n\n2.4 The title \"Hypotheses Estimation\" (line 137) should be \"Hypothesis Estimation\".\n\n**3. Some intermediate results should be visualized.**\n\nSome real intermediate results after the hypothesis estimation/segmentation steps should be visualized to show how these steps provide appropriate predictions. The virtual intermediate \"results\" in Figure 3 are helpful to understand, but the actual results are still necessary.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and valuable comments.\n\n1.1 What is the computation manner of the Transformer before the hypothesis estimation step? In line 237, the authors state that \"then we perform transformer five times at M_1\". Does the transformer contain some cross-attention processes? Or does it only involve the self-attention?\n\nNo, we only use this strategy for the fine level, because this approach cannot cope with the need to iterate the transformer multiple times, and it only manages to improve the efficiency without compromising the accuracy if it is only iterated once. This drawback stems from the fact that it only needs to optimize the feature that is used to estimate the refined bias, which is only one out of every 16 features on the source image on the $M_3$ feature map according to our method, and the features on the target image are not optimized in this uni-directional attention operation. Therefore, the scope of this uni-directional attention can only be used in refinement stage, and it is not suitable for the scenarios that require multiple iterations such as coarse level.\n\n1.2  The computation process of the hypothesis estimation step is confusing.\n\n We are trying here to use $a$ and $i$ to represent the units on the target images and source images respectively, and then $a_i$ to describe the unit on the target image that corresponds to unit $i$ on the source image. Since you find this confusing, we will add an $s$ or $t$ annotation in the upper right corner, for example: $f_k^{3s}$ and $f_k^{3t}$\n\n1.3  The computation architecture of the segmentation step is unclear. The authors should provide the computation details on how to predict the classification confidence from the intermediate features. The statement in lines 177-180 is too brief to understand.\n\nSegmentation refers to the classification of each unit, where we determine which homography hypothesis should be adopted for unit j on $M_2$ through classification. The way to obtain the classification result is by comparing the classification score matrix $C_j$ of unit $j$ for different hypotheses $H_i$, where the largest one is the result of our classification operation. This classification uses the concept of multi-label classification, a method widely applied in detection problems. Therefore, we refer to DETR and use focal loss to optimize segmentation here. We can describe the process of obtaining the classification score matrix $C_j$ in the form of a formula: $C_{ji} = (T(f_i) + P(i), f_j)$, where $C_{ji}$ refers to the matching score of unit j for hypothesis i. $T$ refers to the function that converts the feature dimension of i (256 dimensions) to the feature dimension of j (128 dimensions); here, we use a 2D CNN to perform $T$. $P$ refers to positional embedding, which directly represents the relative position of the unit corresponding to the hypotheses i in the local 3*3 units. And ( * ,  *)  indicates the inner product. We will add this part to the supplementary materials later.\n\n1.4 Why is the classification label in the segmentation step termed as the \"pseudo\" ground truth (line 181)? To my understanding, the classification label of every unit should be obtained from the ground truth camera pose and depth. It should be the \"normal ground truth\" rather than the \"pseudo ground truth\" if the above understanding is correct. The authors should clarify this detail.\n\nHere we use the term 'pseudo ground truth' merely because the annotations are computed in real-time. However, considering that the term 'pseudo-labels' is generally used for labels obtained from the predictions of neural networks, this usage is indeed incorrect and your are right. We will correct this term to 'computed groundtruth'. \n\n2.1 The statement \"while we only need to feed 300\" (line 50-51) was not clarified in the subsequent text. Is it the unit number on the 1/32 resolution for a 640x480 input image?\n\n300 indeed refers to the number of units obtained at 1/32 resolution on a $640 \\times 480$ resolution image. We will change this to: 'Previous methods feed $80 \\times 60$ tokens to the transformer with 1/8 resolution, while we only need to feed $20 \\times 15$ with 1/32 resolution.' This will be easier to understand.\n\n2.2 In the segmentation step, what is the size of output segmentation results for a local input window whose size is 3x3? Is it 4x4? Or 12x12?\n\nThe output of the segmentation part is an H/8×W/8 array, with values ranging from 0 to 8. This number determines which local hypothesis should be adopted to calculate the input for the refinement step.\n\n2.3 The statement in lines 304-305 should be further discussed. The authors just state that ETO is better on the more difficult YFCC100M without discussing the probable reason.\n\nAll of the methods we tested achieved lower metrics on the YFCC dataset. From this perspective, we consider the YFCC dataset to be more challenging.\n\n2.4 The title \"Hypotheses Estimation\" (line 137) should be \"Hypothesis Estimation\".\n\nThank you for pointing out the typos. We will correct the typos later.\n\n3.1 Some real intermediate results after the hypothesis estimation/segmentation steps should be visualized to show how these steps provide appropriate predictions. The virtual intermediate \"results\" in Figure 3 are helpful to understand, but the actual results are still necessary.\n\nWe provide the visualization results in the pdf of  \"Author Rebuttal\".", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Thank you for your review and valuable comments.We will correct the problems mentioned by the reviewers. 

Here we add some comparative experiments on some recent methods, compared with these methods, ETO still has a high advantage in efficiency. We then provide a graphical representation of how our segmentation and homography hypotheses work.

Meta-review: The paper presents an innovative framework that significantly enhances the efficiency of transformer-based local feature matching by introducing a multi-stage approach that includes coarse matching with homography hypotheses and a refinement stage using unidirectional cross-attention. All reviewers rated the paper positively, recognising its technical soundness, comprehensive experiments, and effective design. The authors convincingly addressed most reviewer concerns in their rebuttal, particularly regarding methodological details and experimental comparisons. However, the need for further experiments and improved clarity was highlighted. To ensure the paper meets the highest standards, the AC recommends accepting the paper with the condition that the authors include additional comparisons, specifically with the latest methods such as RoMa, and enhance the clarity of the submission in the final version. This decision has been approved by Senior AC. Congratulation!!
Decision: Accept (poster)