Title: ETO:Efficient Transformer-based Local Feature Matching by Organizing Multiple Homography Hypotheses
Venue: NeurIPS 2024
Accepted: yes
Keywords: local feature matching

3d vision

pose estimation
Abstract: We tackle the efficiency problem of learning local feature matching.Recent advancements have given rise to purely CNN-based and transformer-based approaches, each augmented with deep learning techniques. While CNN-based methods often excel in matching speed, transformer-based methods tend to provide more accurate matches. We propose an efficient transformer-based network architecture for local feature matching.This technique is built on constructing multiple homography hypotheses to approximate the continuous correspondence in the real world and uni-directional cross-attention to accelerate the refinement. On the YFCC100M dataset, our matching accuracy is competitive with LoFTR, a state-of-the-art transformer-based architecture, while the inference speed is boosted to 4 times, even outperforming the CNN-based methods.Comprehensive evaluations on other open datasets such as Megadepth, ScanNet, and HPatches demonstrate our method's efficacy, highlighting its potential to significantly enhance a wide array of downstream applications.

ETO:Efficient Transformer-based Local Feature Matching by Organizing Multiple Homography Hypotheses

Abstract

Recent developments have led to the emergence of transformer-based approaches for local feature matching, resulting in enhanced accuracy of matches. However, the time required for transformer-based feature enhancement is excessively long, which limits their practical application. In this paper, we propose methods to reduce the computational load of transformers during both the coarse matching and refinement stages. During the coarse matching phase, we organize multiple homography hypotheses to approximate continuous matches. Each hypothesis encompasses several features to be matched, significantly reducing the number of features that require enhancement via transformers. In the refinement stage, we reduce the bidirectional self-attention and cross-attention mechanisms to unidirectional cross-attention, thereby substantially decreasing the cost of computation. Overall, our method demonstrates at least 4 times faster compared to other transformerbased feature matching algorithms. Comprehensive evaluations on other open datasets such as Megadepth, YFCC100M, ScanNet, and HPatches demonstrate our method's efficacy, highlighting its potential to significantly enhance a wide array of downstream applications.

1 Introduction

Local feature matching [1, 2] is a fundamental problem in the field of computer vision and plays a significant role in downstream applications, including but not limited to SLAM [3-8], 3D reconstruction [9, 10], visual localization [11-13], and object pose estimation [14, 15]. However, traditional CNN-based methods [16, 17] often fail under extreme conditions due to the lack of a global receptive field, thus meeting failure under dramatic changes in scale, illumination, viewpoint, or weakly-textured scenes.

Recently, some methods [18-20] forgo traditional CNN-based approaches and base on Transformer [21] for better modeling the long-range dependencies. However, Transformer is widely known for its high computational complexity especially when it is applied in vision tasks where the computational complexity grows quadratically in the number of the input image tokens (i.e., patches). To reduce the inherent complexity associated with the Transformer, these methods generally adopt the coarse-to-fine strategy and incorporate more computationally efficient variants of the Transformer, such as the Linear Transformer [22]. Nevertheless, the computational overhead remains substantial and severely hinders the application demanding low-latency operations, such as tracking [23-25], or those requiring the processing of extensive datasets, such as large-scale mapping [26].

50

Relative Pose Accuracy[auc@5]

Quadtree

Ours

45

ASpanFormer

-QLoFTR

40

！

LightGlue

35

SuperGlue

LightGlue*

25

20

15

SuperPoint+Search

0

10

20

30

40

50

ImagePairsPerSecond

Source

View

Homography

Hypotheses

Uniform

Hypotheses

Target

View

Predicted

Patch

In this paper, we propose to solve the efficiency problem of transformer-based local feature matching. Our insights are twofold. First, we propose to introduce the homography hypothesis in the pipeline. The homography hypothesis is a kind of piece-wise smooth prior to the scene that has long been explored in the vision tasks [27]. It allows us to create larger patches and reduce the tokens number that need to be processed in the Transformer. However, it is non-trivial since the regular shape introduced by the homography hypothesis can bring significant errors, especially along the boundary. Besides, how to supervise the training of multiple homography hypotheses with the absence of ground truth remains a problem. Second, We empirically find it is redundant that the previous methods employ multiple self- and cross-attention in their fine-level stage since the coarse-level stage has conducted sufficient propagation. As a result, the computation complexity can be further reduced.

Specifically, we propose ETO, the Efficient Transformer-based Local Feature Matching by Organizing Multiple Homography Hypotheses. ETO follows previous methods [20, 19] and employs a twostage coarse-to-fine pipeline. It first establishes matches at the patch level and then refines the matches to the sub-pixel level. In the first stage, ETO learns to predict a set of hypotheses, each encompassing multiple patches to be matched. We approximately assume that each patch to be matched within one hypothesis is on the same plane, and thus describe these matches under the homography transformation, as illustrated in Fig. 2. The homography hypotheses allow us to reduce the image tokens (patches) that are fed to the Transformer. For a typical image with a resolution of 640 × 480, Previous methods feed 80 × 60 tokens to the transformer with 1/8 resolution, while we only need to feed 20 × 15 with 1/32 resolution, which brings a significant speed up. To reduce the possible error due to the regular shape of the homography hypotheses, ETO subdivides the patches into multiple sub-patches and re-selects the correct hypothesis for each sub-patch. We model the problem of re-selection as a segmentation problem [28]. After that, ETO refines the matches in the second stage. Unlike previous methods that employ multiple self- and cross-attention, ETO only conducts one cross-attention, and the size of query tokens it use is much smaller than previous methods. We call it uni-directional cross-attention. Empirically we find uni-directional cross attention converges significantly faster at training while providing much higher efficiency. As shown in Fig. 1, ETO outpaces existing methods, achieving 4-5 times faster than LoFTR [20] and 2-3 times more rapid than LightGlue [29] while maintaining a comparable accuracy with them.

Our contributions can be summarized as follows. 1) We introduce multiple homography hypotheses for the local feature matching problem, which can greatly compress the number of tokens involved in the transformer. 2) We introduce uni-directional cross-attention in the refinement stage. This structure provides fast inference efficiency while maintaining accuracy. 3) Our method not only matches the performance of other transformer-based approaches on diverse open-source datasets such as Megadepth [30], YFCC100M [31], ScanNet [32], and HPatches [33], but it also operates at a significantly higher speed, outpacing all compared methods.

2 Related Works

Transformer-based Local Feature Matching. To find sparse correspondence between two images under diverse viewpoint movement conditions, traditional hand-crafted [34, 1, 2] or early learningbased approaches methods [17, 35] usually match keypoints [36, 37] with their descriptors after

detecting them [16]. These kinds of matching pipelines are highly dependent on the description of unique feature points, and they fail naturally at dramatic viewpoint changes or poorly textured scenes. Thanks to transformer [21], researchers now have a toolbox to enhance the feature descriptors with global information. Earlier researches [36] integrate the information that where other key points are to each descriptors of key points. Then, [38, 39] modeling the mapping relationship as a continuous 2-d function. Sun et al. [20] constructs a global matching pipeline for each unit on the feature map with Transformer [21, 40-42]. The following works refine this pipeline with optical flow [19] or more efficient attention structure [43]. There exists other attempts. [44-46] follow another technical route without transformer. They try to merge the gap between optical flow [47, 41] and local feature matching with the concept of confidence. Our approach is based on a similar feature extractor of [20], while we parameterize more information for the units on the feature map, and finally extend the correspondence relationship to the homography relationship between them. The concept of parameterized units on the feature map for local feature matching is introduced by Ni et al. [18], but they parameterize the units only with scale.

Paramerterization in Local Feature Matching. Conventional techniques, as demonstrated in previous works [34, 48, 35], when confronted with appearance differences due to changes in viewpoint, try to construct feature descriptors which are invariant to these changes, involving scale, normal and rotation, etc. For rule-based methods [34], they create hand-crafted descriptors with scale space analysis [49]. For learning-based methods [16], they input many image pairs with different viewpoints, letting the neural network learn about the invariance of these changes in appearance. Nevertheless, to fully mitigate the impact of these appearance changes, it is essential to accurately estimate their effects. Recent efforts [38, 18, 50] have attempted to directly estimate the scale differences between images and resize the images to enhance the refinement stage of feature matching. In a similar route, [51] has focused on directly estimating rotation to calibrate the local features. Unlike the methods aimed solely at discovering more invariant feature descriptors, we argue that parameterized local appearance changes not only pose a challenge in finding accurate matches but also provide a direct avenue to achieve more accurate and more efficient matches over a broader spatial extent. It allows us to locally parameterize the image to many planes, thus creating multiple homography relationships, not just point-to-point correspondence.

Acceleration in Local Feature Matching. In the realm of local feature matching based on the Transformer architecture, three predominant strategies have been mainly employed to enhance computational efficiency in the past. The first approach involves substituting the Softmax function for the Optimal Transport algorithm introduced by [36]. The second approach seeks to replace the full Transformer with a linear Transformer [22]. LoFTR [20] incorporates both of these strategies, however, it introduces a larger number of matching units compared to SuperGlue [36], resulting in a considerably slower performance than [36]. The third strategy focuses on reducing the number of layers and units within the Transformer. LightGlue [29] introduces early termination [52, 53] and progressive unit selection strategies to accelerate computation, yielding significant improvements in speed. However, the performance of LightGlue heavily relies on SuperPoint [16], which puts a ceiling on its acceleration. In contrast, ETO relies on a more precise parameterized model, achieving higher coarse matching accuracy with a feature map whose resolution is 16 times smaller than LoFTR [20], which distinguishes ETO from another concurrent work Efficient LoFTR [54] that accelerates the algorithm with adaptive token selection on the feature map of the same resolution. Furthermore, during the fine matching stage, we introduce a uni-directional cross-attention mechanism, allowing us to achieve higher matching speed while sacrificing only a minimal amount of fine stage accuracy.

3 Method

Fig. 3 presents our comprehensive feature matching process, organized into three structured modules. These modules are interconnected through feature extractors inspired by U-Net [55] and local attributes generated by neural networks. For each unit i on a H/32 × W/32 resolution feature map M 1 , we estimate the attributes of homography hypotheses H i . For each unit j on H/8 × W/8 resolution feature map M 2 , it re-selects the optimal homography hypotheses ˆ H j from nearby 9 hypotheses to minimize the projection errors. For the chosen unit k j on a H/2 × W/2 resolution feature map M 3 , we fix its center point P s j at the source image and refine the coordinates of its projected point P t j at target image according to ˆ H j , then get the final matches P t ∗ j . We introduce the feature extractor in Sec. 3.1, Sec. 3.2 details the estimation of the hypotheses. In Sec. 3.3 we describe the segmentation

̂

Hypothesis

Segmentation

Refinement

H

H

1

H

2

H

3

H

4

H

5

H

6

H

9

H

7

H

8

Δ

P t

j

=

P t

*

j

LH

Ls

Step 2

Step 1

H

1

H

4

H

5

H

8

H

9

H

3

P t

j

P s

j

Module 1

Module 2

1

32

1

8

1

2

i

j k

ℋ

̂ ℋ

Module 3

MLP

+

M

1

M

2

M

3

Hj

Cross Attention

Source

Target

Two-Stage Training

i

j

?

of the feature map, Sec. 3.4 delves into refining the matches, while Sec. 3.5 states our supervision methodology.

3.1 Feature Extraction

Following [20], we use ResNet-18 [56] as the basic feature extractor to get the feature map with the resolution of N = H / 32 × W / 32 , while we assume that the resolution of the source image and target image is the same. Here we get N = H / 32 × W / 32 features and then perform stacked self-attention and cross attention layers between these N tokens to compute the feature map M 1 . Although there will be N (more than 9) possible patches, our method is mainly performed on locally adjacent patches, so we will omit N in the future and take the local 3 × 3 patches to illustrate our method (as shown in Fig. 3). Then, we follow [18] to upsample the feature map M 1 with a U-Net [55] like structure. So, we can obtain the feature maps M 2 and M 3 at 1 / 8 and 1 / 2 scale, respectively.

3.2 Hypothesis Estimation

Traditional semi-dense feature matching methods [20, 19] often divide an image into thousands of units. For each unit, they perform bipartite graph matching [57]. Contrarily, we argue that bipartite graph matching can be extended to the local homography transformation as hypotheses that cover multiple units to be matched. This approach's merit lies in two folds: achieving more precise matches estimated during the first stage and reducing the number of units which are involved in transformer.

For each unit i on M 1 of source images, it is equipped with a feature f 1 i , a confidence score c i , and a set of hypothesis homography parameters H i (including the source positions p s i ∈ R 2 , target positions p t i ∈ R 2 , rotation r i ∈ R 1 , scale s i ∈ R 1 and perspective q i ∈ R 4 ). And the unit on target images is indicated by a .

Homography Matrix. Initially, we outline the methodology for estimating each unit's local attributes ( p s i , p t i , r i , s i , q i , c i ) and subsequently use these attributes to formulate the local homography matrix. Among these local attributes, the scale s i , rotation r i , and perspective q i are more related to the feature itself and are regressed directly from features f i on the source image through an MLP network. In contrast, target coordinates p t i , and confidence scores c i are more related with the feature map of target images. They are acquired by first identifying the unit a ∗ i with maximal similarity among all target units, and the similarity is defined as the cosine similarity of initial features f on the source image and target image. Following this, we construct new features ˆ f by executing the group-wise correlation [58] within the neighborhood of the target units on M 1 .

where < ∗ , ∗ > g is the group-wise correlation [58], and the group size in our method is 8, Neighbor represents the 5 × 5 neighborhood of unit a ∗ i , ⊕ indicates the operation of concatenation. With the new features ˆ f , we use an MLP to process it to get the target position p t i and confidence c i . With these attributes, in order to compute the homography matrix, we establish four target points B t i that correspond to four predetermined reference points:

ExtractedfromUNet's

Extracted fromUNet's

Feature Map

M

Feature Map

M2

32

H

pixell

3

H=²

argminIIP} - H;P§II

8

H

H5

H6

HEX

4

pixel

Here B s i are four imaginary points on source image, while B t i are the corresponding target points of B s i on the target image. B t i is computed from B s i by following operation: R is the operation of rotation with parameter r i , P is the perspective transformation operation with parameter q i . These operations allow each variable within the homography matrix H i to be deduced from 8 projection equations of 4 correspondence B t i = H i B s i . Details regarding the specific implementation methods for rotation and perspective transformations will be included in the supplementary materials.

3.3 Segmentation

To propagate the homography hypotheses predicted by M 1 to a more detailed resolution. We introduce a segmentation operation at the feature map M 2 with the resolution H / 8 × W / 8 . Segmentation is a per-unit classification task, and we predict a class for each unit j on M 2 . Here, we only consider locally adjacent 3 × 3 patches, and all possible classes is defined as H = { H i | i = 1 ... 9 } . This classification (segmentation) involves that, for each unit j , selecting a hypothesis from H that minimizes the projection error at the center of unit j . After our proposed segmentation stage, each sub-unit j can find the hypotheses ˆ H j that make its error smallest in all possible hypotheses H . We illustrate the intuitive process of segmentation at Fig. 4.

Our proposed segmentation differs from traditional semantic segmentation. Instead of aiming for a specific semantic category, it targets a dynamic geometric relationship. To find the relationship, we introduce a new cosine similarity matrix C j between the local feature f 2 j on M 2 and all features f 1 i on M 1 , Positional encoding is employed during this phase to enhance local features, which is indispensable here because the hypothesis in H are not equivalent. To predicting the class ˆ H j by finding the maximum C j , we generate the computed groundtruth H j as follows:

where H j is the optimal hypothesis that minimize the projection error. Then, we use focal loss [59] L s to minimize the segmentation error between predicted ˆ H j and the computed groundtruth H j . The probability of focal loss is set to C j .

3.4 Refinement

Following [20], to enhance efficiency, only one of the points within each unit j is selected for refinement, which we denote the source and target point as P s j and P t j . Given P t j = ˆ H j P s j , the refinement stage finds the offset ∆ P t j of each target point P t j relative to a fixed source point P s j . With the feature f 3 from M 3 , conventional techniques unfold features f 3 k from local regions in both source image and target image, followed by self-attention and cross-attention. We claim that this process is unnecessarily slow. Here we eliminate self-attention and reduce cross-attention from a bi-directional process to a uni-directional one. Specifically, the feature ˆ f 3 j is computed by querying the features f 3 k within the neighborhood of the original target point P t j , and become the final feature vectors ˆ f 3 j for the point P j on the images. We illustrate this process in supplementary materials. With the proof of experiments in Sec. 4.4, we find that refining a single feature in the local region of M 2 is enough to get expected results. Our findings indicate that this approach can largely diminish the computational load of attention mechanisms while still preserving highly accurate matching outcomes.

Finally, following [28], we process one fixed element of the final feature vector ˆ f 3 j via cross-attention as the confidence score c j for the corresponding set of matches. It is supervised by if the error final match is larger than a threshold. The coordinates for the matched pairs on the source and target

images are ( P s j , P t ∗ j ) . Therefore we can define the supervision of refinement as:

where BCE is binary cross entropy [60], which is a commonly utilized loss function for binary classification problems. Here we use BCE to recognize reliable matches. Here P t j is the ground truth value of P t ∗ j .

3.5 Supervision

Indirect supervision for the homography hypothesis. Instead of supervising the attributes of the hypothesis directly, our approach employs indirect supervision by monitoring the correspondences of sampling points that are linked via the homography transformation. This design offers the advantage of leveraging an excessive number of ground truth matches to efficiently train a network focused on estimating a set of homography parameters. Using the ground truth camera pose and depth from datasets, we can get real matched points P s in source and P t in target images. We sample the matched points to train our method. For each points p in P s , we select only 3 × 3 adjacent hypotheses H = { H 1 , ..., H 9 } around it on M 1 . where H 5 is in the hypothesis region's center and represents the hypothesis of the region containing point p . Similar to the necessity of segmentation stated in Sec. 3.3, direct supervision which applies H 5 to every p could result in avoidable errors, which arise from the mismatch between the irregular boundaries of planes in the real world and the grid-structured unit i on the source image. However, given that these sampling points are merely an auxiliary tool for the loss function, we can directly utilize the ground truth coordinates of the matches to supervise and eliminate classifying each sampling point at this stage. For each point p , we assume that it satisfies a certain homography transformation H p , and H p satisfies the following defination:

We denote p in the source image as p s and target image as p t , we use the following error to optimize our method:

where | ∗ | 1 is the L1 norm error.

Classification or Correspondence Loss for Hypotheses. H i is calculated on the base of identifying the matched unit a ∗ i for unit i in target image. Therefore, if the estimated a ∗ i significantly deviates from the ground truth a ∗ i , H i would be entirely incorrect. In such conditions, we use classification loss to enhance the feature similarity between a ∗ i and a ∗ i . In the opposite case, directly supervising the point correspondences calculated through H i yields better results. The methodology is detailed as follows:

where | ∗ | ∞ denotes the computation of infinity norm, CosSim is the cosine similarity of two features in the feature map M 1 . P i is the set of sampled points p that apply H i as H p .

Two-stage Training Process. In the entire feature matching process, we divide the training process into two stages, the coarse stage and the fine stage. The fine stage will freeze all the parameters of the coarse stage during training. The coarse stage includes the homography hypotheses estimation, while the fine stage includes segmentation and refinement. The losses used in these two parts are:

3.6 Implementation Details

For feature extracting, we use Resnet-18 [56], then we perform transformer [21] five times at M 1 . We implement uni-directional cross-attention once in the process of refinement. We train our outdoor model and indoor model respectively. The outdoor model is trained on the Megadepth [30] dataset, while the indoor model is trained on a mixed dataset of Megadepth and ScanNet [32]. The training process is divided into three stages: the first stage is training on data of 640x480 resolution; in the second stage, the longer side is scaled to 640, and some images are rotated by 90 degrees for adaptation training for the coarse; the third stage involves training the fine on 640x480 data. The learning rate used in the first stage is 1e-4. In the second stage is 5e-5, and in the third stage is 3e-4. Both models are trained using three RTX 3090 for 80 hours, with a batch size of 24 in the first stage and 16 in both the second and third stages. We perform all inferences using PyTorch, merely following the implementation of LightGlue to pre-compile the transformer in the coarse stage with PyTorch.

4 Experiment

We conducted these evaluations on four different datasets for outdoor and indoor relative pose estimation and homography estimation. These experiments demonstrate superior performance on various downstream tasks.

4.1 Homography Estimation

As our first experiment, we evaluate our quality of correspondences and the ability to fit the homography matrix for planar scenes on the HPatches [33] dataset.

Experimental Setup. We conducted comparative experiments using the image matching toolbox proposed by [61]. Our experiments were configured to replicate the settings outlined for SuperPoint[16], SuperGlue [36], and LoFTR [20] as shown in this toolbox. For LightGlue [29], we follow their open-source code settings. To estimate the homography, we employed the RANSAC algorithm with a threshold of 0.25 pixels, leveraging the OpenCV library. To comprehensively assess the performance of each method, we considered three key metrics: the proportion of matched points with an error within a 1-pixel threshold, the average corner distance for estimated homography matrices measuring less than 1/3/5 pixels, and the average computational time. These metrics were chosen to simultaneously evaluate the matching accuracy, homography estimation precision, and computational efficiency of the methods. We perform this experiment on a RTX2070 GPU, and we turn off all acceleration options for pytorch implementations, such as flash attention and precompilation. In order to get as close as possible to a real usage scenario, here, we do not use a warm-up operation when measuring the computing speed.

Dataset. HPatches [33] contains 52 sequences under significant illumination changes and 56 sequences that exhibit large variations in viewpoints. All images are resized with longer dimensions equal to 640.

Results. We compare ETO with SuperPoint [16], SuperGlue [36], LightGlue [29] and LoFTR [20]. According to Table. 1, our experimental results demonstrate that our method excels in homography estimation accuracy compared to SuperGlue, achieving lower errors within a 1-pixel threshold when compared to both SuperGlue and LightGlue. Furthermore, our approach is significantly faster, outpacing all other methods several times in the evaluation.

4.2 Outdoor Pose Estimation

We assess the efficacy of our approach for relative pose estimation in the same setting using two distinct datasets: YFCC100M [31] and Megadepth [30] for outdoor scenes.

Experimental setup. We report the pose accuracy in terms of AUC metric at multiple thresholds ( 5 ◦ , 10 ◦ , 20 ◦ ) and runtime for every approach, and the RANSAC threshold here is set as 0.25 pixel for all methods. All of the evaluations here are conducted on a RTX2080ti. We turn on flash-attention for LightGlue and turn on the pre-compilation to accelerate the transformer for LightGlue and ETO. Here LightGlue [29] is slower than SuperGlue [36] for the reason that following the default configuration LightGlue extracts 2048 keypoints and resizes the resolution of images to 1024, while SuperGlue extracts only 1024 keypoints and keep the resolution of images as the same. LightGlue* apply the setting of SuperGlue. It is imperative to highlight that our method encompasses 4800 points to be matched here, which is the same as LoFTR [20], ASpanFormer [19], and Quadtree [43]. To ensure an accurate representation of the actual computation speed, we initiate a warm-up phase for each method, consisting of 10 iterations, prior to conducting measurements.

Dataset. YFCC100M [31] encompasses an extensive repository comprising 100 million media assets. For our evaluation, we following [36] and focus on a subset of YFCC100M, specifically four handpicked image collections featuring prominent landmarks, in accordance with the criteria outlined in [36] and [20]. MegaDepth, on the other hand, comprises a dataset containing one million Internet-sourced images depicting 196 distinct outdoor scenes. To ensure the integrity of our evaluation protocol, in line with the guidelines presented in [18], we randomly select 1000 image pairs, guaranteeing that none of these pairs have been used in the training processes of any existing methods. All images in Megadepth and YFCC100M are resized with a resolution equal to 640*480.

Results. We compare ETO with SuperPoint [16], SuperGlue [36], LightGlue [29], LoFTR [20], ASpanFormer [19] and Quadtree [43]. According to the results shown in Table 2, on the easier outdoor cases in MegaDepth, the accuracy of our method for pose estimation is lower than advanced detector-free method but is higher than any detector-based approaches, while our runtime is at most 23% of the detector-free methods, 81% of the detector-based methods and 90% of the CNN-based methods. While on the more difficult outdoor cases in YFCC100M, the performance of our model is much better than detector-based methods and is comparable with detector-free methods. And still, our superiority on runtime is preserved.

4.3 Indoor Pose Estimation

We evaluate our method for indoor pose estimation with ScanNet-1500 [32] following [36, 20].

Experimental Setup. Just like outdoor cases, we report the pose accuracy in terms of the AUC metric at multiple thresholds ( 5 ◦ , 10 ◦ , 20 ◦ ) and runtime for every approach. However, here we set all of the RANSAC thresholds as 0.5 pixels. All of the images are resized with longer dimensions equal to 640. This evaluation is conducted on RTX2080ti. We have done a warm-up here in measuring the efficiency.

Dataset. The ScanNet dataset represents a comprehensive indoor RGB-D collection encompassing 1,613 distinct sequences that cumulatively offer 2.5 million unique views. Each view within this dataset is meticulously annotated with a corresponding ground truth camera pose and depth map. We follow the same training and testing split used by [36].

Results. We compare our approach with SuperPoint [16], SuperGlue [36], LoFTR [20], ASpanFormer [19] and Quadtree [43]. The results are demonstrated in Table 3. We find that our results are comparable with LoFTR and are superior to SuperPoint+search and SuperPoint+SuperGlue, while much faster than any other methods.

4.4 Ablation Studies

To evaluate the impact of each design component on the overall structure, we perform an ablation study using the MegaDepth dataset. We systematically add each design element one at a time. The quantitative results are presented in Table 4.

Base32 w/o Homography We match the units on M 1 at H/32 * W/32 resolution and permit the target of unit centroid to be continuous, and we can compute it as Section. 3.2, while other parameters for the unit are still fixed. We output four virtual correspondences as matches. While it offers rapid processing, it does not achieve a high level of accuracy.

Base8 w/o Homography We set this ablation experiment as the coarse matching of LoFTR [20]. Here we match every possible 8*8 units. We output the center of corresponding units as matches. It is more accurate but too slow.

Base32 w/ Homography. Following Section. 3.2, we estimate the whole homography matrix and output four virtual correspondences as matches. It performs better than the coarse matching of LoFTR [20] while providing higher efficiency at the same time,

Basic Refinement w/ Segmentation. Following [20], we set a layer of transformer between 25 tokens on these two images and try to refine our results while the transformer is trained for 12 hours, which is the same as the training time of our uni-directional attention for the refinement stage. While full attention execution speed is considerably slower than that of uni-directional attention, its accuracy is merely comparable with the latter.

Uni-directional w/o Segmentation. Here we directly choose the homography hypotheses H 5 for each unit j which is in the center. Then, we conduct the refinement as the same. The results show that the segmentation stage significantly improves the accuracy.

5 Conclusion and Limitations.

In this paper, we propose Efficient Transformer-based Local Feature Matching by Organizing Multiple Homography hypotheses (ETO). ETO tries to approximate a continuous corresponding function with multiple homography hypotheses with fewer tokens fed to the transformer. Multiple datasets demonstrate that ETO delivers nearly comparable performance in relative pose estimation and homography estimation with other transformer-based methods, while its speed surpasses all of them by a large margin. However, there remains significant space for improvement in ETO's matching accuracy. Next, we could explore an end-to-end training mode, which would allow for further enhancement of the feature extractor at a fine-grained level. Moreover, we believe that intermediate-level features can provide not only segmentation information but also data conducive to more precise matching. Finally, the form of parametric scheme we present here may not be optimal and complete for homography transformation, so we will continue to explore better parametric schemes. These strategies are expected to enable our method to compete with approaches like PATS [18] and DKM [45] in terms of matching precision, without considerably compromising speed. Acknowledgements. This work was partially supported by NSF of China (No. 61932003).

References

Ethan Rublee, Vincent Rabaud, Kurt Konolige, and Gary R. Bradski. ORB: an efficient alternative to SIFT or SURF. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 2564-2571. IEEE, 2011.

Herbert Bay, Andreas Ess, Tinne Tuytelaars, and Luc Van Gool. Speeded-up robust features (SURF). Comput. Vis. Image Underst. , 110(3):346-359, 2008.

Raul Mur-Artal, J. M. M. Montiel, and Juan D. Tardós. ORB-SLAM: A versatile and accurate monocular SLAM system. IEEE Trans. Robotics , 31(5):1147-1163, 2015.

Xingrui Yang, Hai Li, Hongjia Zhai, Yuhang Ming, Yuqian Liu, and Guofeng Zhang. Voxfusion: Dense tracking and mapping with voxel-based neural implicit representation. In IEEE International Symposium on Mixed and Augmented Reality , pages 499-507. IEEE, 2022.

Xinyang Liu, Yijin Li, Yanbin Teng, Hujun Bao, Guofeng Zhang, Yinda Zhang, and Zhaopeng Cui. Multi-modal neural radiance field for monocular dense slam with a light-weight tof sensor. In Proceedings of the ieee/cvf international conference on computer vision , pages 1-11, 2023.

Jiarui Hu, Xianhao Chen, Boyin Feng, Guanglin Li, Liangjing Yang, Hujun Bao, Guofeng Zhang, and Zhaopeng Cui. Cg-slam: Efficient dense rgb-d slam in a consistent uncertainty-aware 3d gaussian field. arXiv preprint arXiv:2403.16095 , 2024.

Jiarui Hu, Mao Mao, Hujun Bao, Guofeng Zhang, and Zhaopeng Cui. Cp-slam: Collaborative neural point-based slam system. Advances in Neural Information Processing Systems , 36, 2024.

Hongjia Zhai, Gan Huang, Qirui Hu, Guanglin Li, Hujun Bao, and Guofeng Zhang. Nis-slam: Neural implicit semantic rgb-d slam for 3d consistent scene understanding. IEEE Transactions on Visualization and Computer Graphics , pages 1-11, 2024.

Shuo Chen, Mao Peng, Yijin Li, Bing-Feng Ju, Hujun Bao, Yuan-Liu Chen, and Guofeng Zhang. Multi-view neural 3d reconstruction of micro-and nanostructures with atomic force microscopy. Communications Engineering , 3(1):131, 2024.

Bangbang Yang, Yinda Zhang, Yijin Li, Zhaopeng Cui, Sean Fanello, Hujun Bao, and Guofeng Zhang. Neural rendering in a room: amodal 3d understanding and free-viewpoint rendering for the closed scene composed of pre-captured objects. ACM Transactions on Graphics (TOG) , 41 (4):1-10, 2022.

Paul-Edouard Sarlin, Cesar Cadena, Roland Siegwart, and Marcin Dymczyk. From coarse to fine: Robust hierarchical localization at large scale. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12716-12725, 2019.

Zhaoyang Huang, Han Zhou, Yijin Li, Bangbang Yang, Yan Xu, Xiaowei Zhou, Hujun Bao, Guofeng Zhang, and Hongsheng Li. Vs-net: Voting with segmentation for visual localization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6101-6111, 2021.

Hongjia Zhai, Xiyu Zhang, Zhao Boming, Hai Li, Yijia He, Zhaopeng Cui, Hujun Bao, and Guofeng Zhang. Splatloc: 3d gaussian splatting-based visual localization for augmented reality. arXiv preprint arXiv:2409.14067 , 2024.

Yan Xu, Kwan-Yee Lin, Guofeng Zhang, Xiaogang Wang, and Hongsheng Li. Rnnpose: Recurrent 6-dof object pose refinement with robust correspondence field estimation and pose optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14880-14890, 2022.

Guanglin Li, Yifeng Li, Zhichao Ye, Qihang Zhang, Tao Kong, Zhaopeng Cui, and Guofeng Zhang. Generative category-level shape and pose estimation with semantic primitives. In Conference on Robot Learning , pages 1390-1400. PMLR, 2022.

Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. Superpoint: Self-supervised interest point detection and description. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 224-236, 2018.

Mihai Dusmanu, Ignacio Rocco, Tomás Pajdla, Marc Pollefeys, Josef Sivic, Akihiko Torii, and Torsten Sattler. D2-net: A trainable CNN for joint description and detection of local features. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8092-8101, 2019.

Junjie Ni, Yijin Li, Zhaoyang Huang, Hongsheng Li, Hujun Bao, Zhaopeng Cui, and Guofeng Zhang. Pats: Patch area transportation with subdivision for local feature matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 17776-17786, 2023.

Hongkai Chen, Zixin Luo, Lei Zhou, Yurun Tian, Mingmin Zhen, Tian Fang, David Mckinnon, Yanghai Tsin, and Long Quan. Aspanformer: Detector-free image matching with adaptive span transformer. In European Conference on Computer Vision , 2022.

Jiaming Sun, Zehong Shen, Yuang Wang, Hujun Bao, and Xiaowei Zhou. Loftr: Detector-free local feature matching with transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8922-8931, 2021.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems , 30, 2017.

Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. In International conference on machine learning , pages 5156-5165. PMLR, 2020.

Adam W Harley, Zhaoyuan Fang, and Katerina Fragkiadaki. Particle video revisited: Tracking through occlusions using point trajectories. In European Conference on Computer Vision , pages 59-75. Springer, 2022.

Yijin Li, Yichen Shen, Zhaoyang Huang, Shuo Chen, Weikang Bian, Xiaoyu Shi, Fu-Yun Wang, Keqiang Sun, Hujun Bao, Zhaopeng Cui, Guofeng Zhang, and Hongsheng Li. Blinkvision: A benchmark for optical flow, scene flow and point tracking estimation using rgb frames and events. In European conference on computer vision . Springer, 2024.

Weikang Bian, Zhaoyang Huang, Xiaoyu Shi, Yitong Dong, Yijin Li, and Hongsheng Li. Context-pips: Persistent independent particles demands context features. Advances in Neural Information Processing Systems , 36, 2024.

Paul-Edouard Sarlin, Cesar Cadena, Roland Siegwart, and Marcin Dymczyk. From coarse to fine: Robust hierarchical localization at large scale. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12716-12725, 2019.

Connelly Barnes, Eli Shechtman, Adam Finkelstein, and Dan B Goldman. Patchmatch: A randomized correspondence algorithm for structural image editing. ACM Trans. Graph. , 28(3): 24, 2009.

Joseph Redmon and Ali Farhadi. Yolov3: An incremental improvement. arXiv preprint arXiv:1804.02767 , 2018.

Philipp Lindenberger, Paul-Edouard Sarlin, and Marc Pollefeys. LightGlue: Local Feature Matching at Light Speed. In ICCV , 2023.

Zhengqi Li and Noah Snavely. Megadepth: Learning single-view depth prediction from internet photos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2018.

Bart Thomee, David A. Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. YFCC100M: the new data in multimedia research. Commun. ACM , 59(2):64-73, 2016.

Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas A. Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5828-5839, 2017.

Vassileios Balntas, Karel Lenc, Andrea Vedaldi, and Krystian Mikolajczyk. Hpatches: A benchmark and evaluation of handcrafted and learned local descriptors. In CVPR , 2017.

David G. Lowe. Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis. , 60(2):91-110, 2004.

Jérôme Revaud, César Roberto de Souza, Martin Humenberger, and Philippe Weinzaepfel. R2D2: reliable and repeatable detector and descriptor. Advances in Neural Information Processing Systems , 32, 2019.

Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. Superglue: Learning feature matching with graph neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4938-4947, 2020.

Kwang Moo Yi, Eduard Trulls, Yuki Ono, Vincent Lepetit, Mathieu Salzmann, and Pascal Fua. Learning to find good correspondences. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2666-2674, 2018.

Wei Jiang, Eduard Trulls, Jan Hosang, Andrea Tagliasacchi, and Kwang Moo Yi. COTR: correspondence transformer for matching across images. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 171-180. Springer, 2021.

Dongli Tan, Jiang-Jiang Liu, Xingyu Chen, Chao Chen, Ruixin Zhang, Yunhang Shen, Shouhong Ding, and Rongrong Ji. Eco-tr: Efficient correspondences finding via coarse-to-fine refinement. In European Conference on Computer Vision , pages 317-334. Springer, 2022.

Yijin Li, Xinyang Liu, Wenqi Dong, Han Zhou, Hujun Bao, Guofeng Zhang, Yinda Zhang, and Zhaopeng Cui. DELTAR: depth estimation from a light-weight tof sensor and RGB image. In European Conference on Computer Vision , pages 619-636. Springer, 2022.

Zhaoyang Huang, Xiaoyu Shi, Chao Zhang, Qiang Wang, Yijin Li, Hongwei Qin, Jifeng Dai, Xiaogang Wang, and Hongsheng Li. Flowformer: A transformer architecture and its masked cost volume autoencoding for optical flow. arXiv preprint arXiv:2306.05442 , 2023.

Yijin Li, Zhaoyang Huang, Shuo Chen, Xiaoyu Shi, Hongsheng Li, Hujun Bao, Zhaopeng Cui, and Guofeng Zhang. Blinkflow: A dataset to push the limits of event-based optical flow estimation. In 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 3881-3888. IEEE, 2023.

Shitao Tang, Jiahui Zhang, Siyu Zhu, and Ping Tan. Quadtree attention for vision transformers. In The International Conference on Learning Representations . OpenReview.net, 2021.

Prune Truong, Martin Danelljan, Radu Timofte, and Luc Van Gool. Pdc-net+: Enhanced probabilistic dense correspondence network. arXiv preprint arXiv:2109.13912 , 2021.

Johan Edstedt, Ioannis Athanasiadis, Mårten Wadenbäck, and Michael Felsberg. Dkm: Dense kernelized feature matching for geometry estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 17765-17775, 2023.

Songyan Zhang, Xinyu Sun, Hao Chen, Bo Li, and Chunhua Shen. Rgm: A robust generalist matching model. arXiv preprint arXiv:2310.11755 , 2023.

Zachary Teed and Jia Deng. RAFT: recurrent all-pairs field transforms for optical flow. In European Conference on Computer Vision , pages 402-419. Springer, 2020.

Zixin Luo, Lei Zhou, Xuyang Bai, Hongkai Chen, Jiahui Zhang, Yao Yao, Shiwei Li, Tian Fang, and Long Quan. Aslfeat: Learning local features of accurate shape and localization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6589-6598, 2020.

Tony Lindeberg. Feature detection with automatic scale selection. Int. J. Comput. Vis. , 30(2): 79-116, 1998.

Axel Barroso-Laguna, Yurun Tian, and Krystian Mikolajczyk. Scalenet: A shallow architecture for scale estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12808-12818, 2022.

Miao Fan, Mingrui Chen, Chen Hu, and Shuchang Zhou. Occˆ 2net: Robust image matching based on 3d occupancy estimation for occluded regions. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 9652-9662, 2023.

Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Łukasz Kaiser. Universal transformers. arXiv preprint arXiv:1807.03819 , 2018.

Maha Elbayad, Jiatao Gu, Edouard Grave, and Michael Auli. Depth-adaptive transformer. arXiv preprint arXiv:1910.10073 , 2019.

Yifan Wang, Xingyi He, Sida Peng, Dongli Tan, and Xiaowei Zhou. Efficient loftr: Semi-dense local feature matching with sparse-like speed. arXiv preprint arXiv:2403.04765 , 2024.

Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention-MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18 , pages 234-241. Springer, 2015.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in Neural Information Processing Systems , 26, 2013.

Xiaoyang Guo, Kai Yang, Wukui Yang, Xiaogang Wang, and Hongsheng Li. Group-wise correlation stereo network. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 3273-3282, 2019.

Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision , pages 2980-2988, 2017.

Irving John Good. Rational decisions. Journal of the Royal Statistical Society: Series B (Methodological) , 14(1):107-114, 1952.

Qunjie Zhou, Torsten Sattler, and Laura Leal-Taixé. Patch2pix: Epipolar-guided pixel-level correspondences. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4669-4678, 2021.

Johan Edstedt, Qiyu Sun, Georg Bökman, Mårten Wadenbäck, and Michael Felsberg. Roma: Robust dense feature matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 19790-19800, 2024.

Checklist

Claims: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Claims in the paper should match theoretical and experimental results in terms of how much the results can be expected to generalize. The paper's contributions should be clearly stated in the abstract and introduction, along with any important assumptions and limitations. It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. Enter yes, no, or an explanation if appropriate. Answers are visible to reviewers. Claims in the paper should match theoretical and experimental results in terms of how much the results can be expected to generalize. [Yes] The main claims are clearly stated in the abstract and introduction.

Limitations: The authors are encouraged to create a separate "Limitations" section in their paper. The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). [Yes] Limitations are discussed in Sec. 5.

Theory, Assumptions and Proofs: If you are including theoretical results, did you state the full set of assumptions of all theoretical results, and did you include complete proofs of all theoretical results? All assumptions should be clearly stated or referenced in the statement of any theorems. The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, authors are encouraged to provide a short proof sketch to provide intuition. Enter yes, no, n/a, or an explanation if appropriate. Answers are visible to reviewers. [Yes] We provide a simple proof for using homography hypotheses in supplementary materials.

Experimental Result Reproducibility: If the contribution is a dataset or model, what steps did you take to make your results reproducible or verifiable? Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), release of a model checkpoint, or other means that are appropriate to your research. Enter yes, no, n/a, or an explanation if appropriate. Answers are visible to reviewers. [Yes]

Open Access to Data and Code: If you ran experiments, did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)? Please see the NeurIPS code and data submission guidelines for more details. While we encourage release of code and data, we understand that this might not be possible, so no is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark). At submission time, to preserve anonymity, remember to release anonymized versions. Enter yes, no, n/a, or an explanation if appropriate. Answers are visible to reviewers. [No]

Experimental Setting/ Details: If you ran experiments, did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? The full details can be provided with the code, but the important details should be in the main paper, and information about how hyperparameters were selected should appear either in the paper or supplementary materials. Enter yes, no, n/a, or an explanation if appropriate. Answers are visible to reviewers. [Yes]

Experiment Statistical Significance: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? [No]

Experiments Compute Resource: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments? [Yes]

Code Of Ethics: Have you read the NeurIPS Code of Ethics and ensured that your research conforms to it? Enter yes, no, or an explanation if appropriate. Answers are visible to reviewers. [Yes]

Broader Impacts: If appropriate for the scope and focus of your paper, did you discuss potential negative societal impacts of your work? Please see the Paper Checklist Guidelines for detailed instructions and examples of points that you may choose to discuss. Enter yes, no, n/a, or an explanation if appropriate. Answers are visible to reviewers. [No]

Safeguards: Do you have safeguards in place for responsible release of models with a high risk for misuse (e.g., pretrained language models)? Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model. Enter yes, no, n/a, or an explanation if appropriate. Answers are visible to reviewers. [Yes]

Licenses: If you are using existing assets (e.g., code, data, models), did you cite the creators and respect the license and terms of use? Cite the original paper that produced the code package or dataset. If possible, include a URL. Be sure to check the original license and respect its conditions. Enter yes, no, n/a, or an explanation if appropriate. Answers are visible to reviewers. [Yes]

Assets: If you are releasing new assets, did you document them and provide these details alongside the assets? Researchers should communicate the details of the dataset or the model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. Enter yes, no, n/a, or an explanation if appropriate. Answers are visible to reviewers. [NA]

Crowdsourcing and Research with Human Subjects: If you used crowdsourcing or conducted research with human subjects, did you include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? Including this information in the supplemental material is fine, but if the main contribution of your paper involves human subjects, then we strongly encourage you to include as much detail as possible in the main paper. According to the NeurIPS Code of Ethics, you must pay workers involved in data collection, curation, or other labor at least the minimum wage in your country. Enter yes, no, n/a, or an explanation if appropriate. Answers are visible to reviewers. [NA]

IRB Approvals: Did you describe any potential participant risks and obtain Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your institution), if applicable? Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. For initial submissions, do not include any information that would break anonymity, such as the institution conducting the review. Enter yes, no, n/a, or an explanation if appropriate. Answers are visible to reviewers. [NA]

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an efficient transformer-based local feature matching approach named ETO. ETO consists of three steps: hypothesis estimation, segmentation and refinement. The first and second steps can obtain coarse matching results whose resolution is similar to LoFTR’s coarse results. The third step can provide fine-level matching results. The hypothesis estimation performs the Transformer computation on the 1/32 resolution (rather than the 1/8 resolution in LoFTR), which is the core design that reduces the inference time. Experimental results demonstrate that ETO provides a good balance between accuracy and speed.\n\nSTRENGTHS:\n1. The motivation of this paper is clear and valuable. Transformer-based approaches are prevalent in the area of feature matching. Nonetheless, it is still an open problem how to design an efficient and accurate transformer-based model.\n2. The technical design of ETO is novel. The manner of predicting the local homography parameters on a very coarse resolution (and then refining them) differs from most of the existing coarse-to-fine transformer-based approaches.\n3. The accuracy of ETO is acceptable, considering its superiority in inference speed.\n\nWEAKNESSES:\n**1. This paper lacks some important technical details.**\n\n1.1 What is the computation manner of the Transformer before the hypothesis estimation step? In line 237, the authors state that \"then we perform transformer five times at M_1\". Does the transformer contain some cross-attention processes? Or does it only involve the self-attention?\n\n1.2 The computation process of the hypothesis estimation step is confusing. The authors seem to use the notations \"M_1, M_2, M_3, f_i^1\" to represent the features of both the source and target images. Such definitions make many statements in Sec. 3.2 hard to understand. For example, the meaning of the variable M_1 seems inconsistent in the statements \"For each unit i on M_1\" (line 144) and \"within the neighborhood of the target units on M_1\" (line 155).\n\n1.3 The computation architecture of the segmentation step is unclear. The authors should provide the computation details on how to predict the classification confidence from the intermediate features. The statement in lines 177-180 is too brief to understand.\n\n1.4 Why is the classification label in the segmentation step termed as the \"pseudo\" ground truth (line 181)? To my understanding, the classification label of every unit should be obtained from the ground truth camera pose and depth. It should be the \"normal ground truth\" rather than the \"pseudo ground truth\" if the above understanding is correct. The authors should clarify this detail.\n\n**2. Some details should be further clarified.**\n\n2.1 The statement \"while we only need to feed 300\" (line 50-51) was not clarified in the subsequent text. Is it the unit number on the 1/32 resolution for a 640x480 input image?\n\n2.2 In the segmentation step, what is the size of output segmentation results for a local input window whose size is 3x3? Is it 4x4? Or 12x12?\n\n2.3 The statement in lines 304-305 should be further discussed. The authors just state that ETO is better on the more difficult YFCC100M without discussing the probable reason.\n\n2.4 The title \"Hypotheses Estimation\" (line 137) should be \"Hypothesis Estimation\".\n\n**3. Some intermediate results should be visualized.**\n\nSome real intermediate results after the hypothesis estimation/segmentation steps should be visualized to show how these steps provide appropriate predictions. The virtual intermediate \"results\" in Figure 3 are helpful to understand, but the actual results are still necessary.\n\nQUESTIONS:\nPlease provide more discussions and experimental results to address the above weaknesses.\n\nLIMITATIONS:\nThe authors have discussed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a local feature matching method that leverages homography to accelerate the transformer-based feature matching pipeline. Additionally, they employ unidirectional cross-attention in the refinement stage to further reduce computational overhead. Experimental results demonstrate the efficiency and effectiveness of this approach.\n\nSTRENGTHS:\n1. The paper is well-written and easy to understand.\n2. Integrating homography as theoretical guidance into the transformer pipeline is a commendable approach, enhancing the transformer-based pipeline with theoretical support.\n3. Experimental results demonstrate that the proposed method achieves much smaller time usage, validating its efficiency in practice.\n\nWEAKNESSES:\n1. Though the time usage is decrease, the accuracy is also decrease from Tab.1&2&3.\n2. The paper lacks significant citations in feature matching methods, such as Efficient LoFTR[1], RoMa[2]. Some of these works also focus on improving efficiency in feature matching and should be referenced to provide a comprehensive background. Including these methods in the experiments would offer a more thorough comparison of the proposed approach's performance.\n3. Though the authors acknowledge that some other methods (e.g., [14, 38]) are better in certain aspects (line356-357), it would be beneficial to include a comparative analysis with these methods. This would provide a clearer understanding of the strengths and weaknesses of the proposed approach.\n\n[1]. Wang, Yifan, et al. \"Efficient LoFTR: Semi-dense local feature matching with sparse-like speed.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n[2] Edstedt, Johan, et al. \"RoMa: Robust dense feature matching.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\nQUESTIONS:\nNo question to the authors.\n\nLIMITATIONS:\nThe authors have addressed the limitations in their paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel framework for efficient transformer-based local feature matching. Transformer-based local feature matching usually contains two stages: a coarse matching stage which applies self-attention and cross-attention on coarse-level features (usually H/8 x W/8) to obtain coarse matches, and a refinement stage that refines the coarse matches in a local window based on fine-grained features. This paper proposed two methods to improve the efficiency of this pipeline. First, the coarse matching is performed on a even coarser level (H/32 x W/32) to reduce the computational cost of the costly attention operations. Additionally, homography hypothesis for each patch is estimated at this level. Leveraging the piece-wise smooth prior, the matching at H/8 x W/8 resolution is directly approximated by selecting the most probable homography hypothesis in local windows using a proposed segmentation technique. Second, in the refinement stage, the bi-directional cross attention is reduced to uni-direction one. Experiments show that the proposed framework achieves comparable performance with less inference time compared with state-of-the-art transformer-based and local feature matching algorithms.\n\nSTRENGTHS:\n1. The idea of using piece-wise smooth prior to accelerate transformer-based feature matching is novel and promising, and the supervision and the homography hypothesis re-selection algorithms are carefully designed.\n2. Comprehensive ablation studies show that the key design features, including homography hypothesis proposal, homography hypothesis re-selection and uni-directional cross attention have positive effects on the performance and are necessary.\n\nWEAKNESSES:\n1. The symbols used in the method part are too complicated, making it hard to read.\n2. The symbol $\\mathscr{H}$ used in figure 4 is undefined in the text.\n3. The framework adopts two-stage training, possibly making it hard to train and limiting the accuracy.\n\nQUESTIONS:\n1. Do you have any plan to reorganize the symbols?\n2. If the framework is trained end-to-end, will it yield acceptable performance?\n\nLIMITATIONS:\nThe authors have adequately addressed the limitation of this work in the conclusion and limitations section of the paper. Enabling the network to be trained end-to-end, allowing for dense matching and further improving accuracy are my suggestions for directions of improvement.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a local feature matching method based on the multiple homography hypotheses. The paper explicitly introduces the homography hypotheses for coarse matching, combined with homography segmentation, cross-attention, and sub-pixel refinement to obtain fine matching results. The introduction of the homography hypotheses significantly reduces the number of input tokens for the attention mechanism, and with further model modification, the algorithm proposed in this paper has achieved a substantial increase in inference speed while maintaining performance as much as possible. Experiments on multiple datasets have proven the effectiveness and efficiency of the proposed method.\n\nSTRENGTHS:\nClear motivation, reasonable structural design, convincing experiments\n\nWEAKNESSES:\n-The pre-compiled transformer model may have some impact on inference speed, can additional tests be conducted on the efficiency of a normal transformer module?\n\n-Lack of the latest state-of-the-art comparison: Some of the latest methods, such as RoMA[1] and EfficientLoFTR[2], have been made public at an early stage, and thus it is necessary to compare and analyze with these methods.\n\n[1] Edstedt, Johan, et al. \"RoMa: Robust dense feature matching.\" CVPR 2024.\n[2] Wang, Yifan, et al. \"Efficient LoFTR: Semi-dense local feature matching with sparse-like speed.\"  CVPR 2024.\n\nQUESTIONS:\n-In Section 3, it is 2-stage training adopted, but in Implementation Details, the training phase contains 3 stages. Why there is a joint training stage at first? Moreover, the joint training is usually conducted for the final finetuning, why it is the first stage in your strategy?\n\n-For Basic Refinement w/ Segmentation in Section 4.4, why it is the same training hours instead of the same training samples? Early stopping may lead to performance drop.\n\n-In the introduction, the authors claim that the multiple self- and cross-attention in the fine-level stage are redundant, is there any numerical results? What if you stack several uni-directional cross-attention in your method?\n\nLIMITATIONS:\nsee Weaknesses", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper propose an efficient framework to reduce the computational load of transformer-based matching approaches.  It is a coarse-to-fine two-stage solution. During the coarse matching phase, multiple homography hypotheses are estimated to approximate continuous matches. Each hypothesis encompasses few features to be matched, reducing the number of features that require enhancement via transformers. In the refinement stage, the unidirectional cross-attention is proposed to replace the classical bidirectional self-attention and cross-attention mechanism, which further decreases the cost of computation. Comprehensive evaluations on other open datasets such as Megadepth, YFCC100M, ScanNet, and HPatches are presented to demonstrate its efficacy.\n\nSTRENGTHS:\nThe whole idea makes sense to me. The experimental studies are quite comprehensive.\n\nWEAKNESSES:\nSome parts of the description are quite vague to me, which may hinder its reproducibility. Some of the experimental results may need further clarification.\n\nQUESTIONS:\n1. I think some descriptions in Sec 3.4 need further clarification, e.g., Line 192-193 $\\hat{f^3_j}$ is computed by querying the $f3_k$ with …, is the query done by nearest neighboring searching?\n2. How is $\\triangle P_j^t$ in Line 188 obtained or is it predicted?\n3. In Figure 3, is there an MLP after cross-attention block? If yes, what is the input/output.  Also, the presence of “Module 3” is confusing, should it contain the cross attention?\n4. In Table 4, it seems the usage of bidirectional cross-attention performs worse than that of uni-directional cross-attention at 5. Is that reasonable, please give some analysis. \n5. From Table 2, the performance of lightGlue is much worse than ASpanFormer, LoFTR. Interestingly, ASpanFormer is better than LoFTR with a small margin in the Megadepth dataset, similar to the results reported in LightGlue. The performance gap between LightGlue and LoFTR is much larger than that in LightGlue, could you please explain? Moreover, why LightGlue is not reported in the dataset of Scannet?\n\nLIMITATIONS:\n1. Some of the network architectures are not depicted in figures, making it difficult to know the overall diagram of the method.\n2. The two-stage training process may not be easily trained.\n3. It would be better to present some failure case studies.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-14"}