Semantic-aware Representation Learning for Homography Estimation

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-05 08:54:35

Abstract: Homography estimation is the task of determining the transformation from an image pair. Our approach focuses on employing detector-free feature matching methods to address this issue.  Previous work has underscored the importance of incorporating semantic information, however there still lacks an efficient way to utilize semantic information. Previous methods suffer from treating the semantics as a pre-processing, causing the utilization of semantics overly coarse-grained and lack adaptability when dealing with different tasks. In our work, we seek another way to use the semantic information, that is semantic-aware feature representation learning framework. Based on this, we propose SRMatcher, a new detector-free feature matching method, which encourages the network to learn integrated semantic feature representation. Specifically, to capture precise and rich semantics, we leverage the capabilities of recently popularized vision foundation models (VFMs) trained on extensive datasets.  Then, a cross-image Semantic-aware Fusion Block (SFB) is proposed to integrate its fine-grained semantic features into the feature representation space. In this way, by reducing errors stemming from semantic inconsistencies in matching pairs, our proposed SRMatcher is able to deliver more accurate and realistic outcomes. Extensive experiments show that SRMatcher surpasses solid baselines and attains SOTA results on multiple real-world datasets. Compared to the previous SOTA approach GeoFormer, SRMatcher increases the area under the cumulative curve (AUC) by about 11% on HPatches. Additionally, the SRMatcher could serve as a plug-and-play framework for other matching methods like LoFTR, yielding substantial precision improvement.

Main paper: 
 introduction: 
 Homography estimation, often referred to as perspective transformation or planar projection from source image to target image as Figure 1 shown, is a core issue in the fields of computer vision and robotics [13, 41] . This process entails calculating a matrix to align points between two images captured from distinct viewpoints. This estimation is pivotal for numerous applications, including including image/video stitching [11, 40] , camera calibration [44] , HDR imaging [9] , and SLAM [23, 45] . Feature matching techniques are widely used to establish feature correspondences in tasks related to homography estimation. The traditional matching methods [22, 29] rely on the quality of keypoints, and can sometimes falter in challenging scenarios such as low texture and blurred [31] . A novel detector-free matching methods have surfaced as more dependable alternatives. These techniques generate dense feature maps without the need for feature detection and execute dense matching on a pixel-wise basis [3, 4, 15, 20, 32] . Existing methods typically extract local features related to position and context but overlook the semantic information of matching pairs, which is crucial for feature matching.Previous work [42, 43] such as MESA do consider semantics, these methods view the matching issue as a search problem, searching corresponding matching points for specific source points in the target image. The introduction of semantics aims to narrow the search space, by establishing area matches the initial search space for point matching is confined to the matched area. This method resembles the human cognitive process, humans first identify matched areas using semantic information and subsequently search for corresponding points within those areas. However, their work faces two notable problems: 1) The first problem is these methods restrict the application of semantic priors to coarse-grained scenarios, failing to fully leverage the knowledge that semantic extraction networks can offer. The goal of incorporating semantic priors into the matching network is to reduce erroneous matches by constraining the semantic consistency of the matching results. However these area matching method merely narrows down the matching area, and semantics can not guide the matching result in subsequent steps, failing to achieve the initial goal. 2) For the second problem, the interaction between semantic information and matching is designed for particular approaches. When dealing with different tasks or data types, processing methods such as area size and quantity need optimization for particular applications. This means these patch-wise method suffer from a lack of adaptability and generalize issues, limiting the ability of the algorithm to understand the overall scene.We identify the key driver of these two problems as the ill-use of the semantic information, specifically, treating semantics as merely a pre-processing step. These methods initially identify semantic area matches based on semantic segmentation and then acquire precise correspondences within these regions using a standard point matcher. This process is intuitive and closely resembles human thought processes [33, 43] . However, for the human cognitive system, an implicit prerequisite in the task of feature matching is that the features being matched should share the same semantics [10, 35, 36] . Based on this, we propose another way to use the semantic information that encourages the network to learn semantic-aware representation.In this paper, we propose SRMatcher, the first semantic-aware representation learning framework which consists of two essential parts: (a) Semantic Extractor. (b) Semantic-aware Fusion Block(SFB).The key challenge to achieving the semantic-aware representation is how to generate similar representations for semantically similar points in image pairs. For (a), we propose to use the DINOv2 [24] as the semantic extractor to acquire richer and more diverse semantic information. This considers the outstanding performance of vision foundation models (VFMs) in tasks related to semantics. For (b), we develop a Semantic-aware Fusion Block (SFB) to utilize fine-grained semantic features to contribute to matching quality. Different from previous works [17, 37, 38] , SFB executes cross-images feature fusion which means image features should not only integrate their own semantic information but also consider the semantics across images. This difference comes from considering the nature of the task, that is the image feature should be enhanced with semantic information of other images to find semantically identical points. In more detail, we employ a semantic-guide interactions block (SGIB) to establish the interactions between image features and semantic features within the SFB.In summary, this work makes the following contributions:• We explore a new way to integrate semantics into the feature matching model for homography estimation. For the first time, we propose SRMatcher, a novel detector-free feature matching method by integrating semantics from vision foundation models (VFMs) into the network.• To enhance the semantic guidance for the matching model, we introduce the Semantic-aware Fusion Block (SFB) to conduct cross-images semantic feature fusion, whereby features are prompted to concurrently consider the semantics from multiple images.• Extensive experiments demonstrate that SRMatcher achieves the SOTA performance on homography estimation tasks and other downstream vision tasks. Additionally, our SRMatcher can be seamlessly incorporated into various benchmarks for detector-free feature matching methods in a plug-and-play fashion. 
related work 2.1 local feature matching: 
 Local feature matching can be categorized into detector-based and detector-free methods. Detector-based methods operate in four stages: detecting keypoints, extracting local features for each keypoint, matching features based on content, and finally fitting a homography using the identified matches. SIFT [22] features exhibit invariance to rotation and scale. With the advent of deep learning, many learning-based methods have been proposed, to further enhance the robustness of descriptors under varying illumination and viewpoint changes, such as Superpoint [5] , D2-Net [28] . SuperGlue [30] utilizes Transformer [34] to establish correspondences between two sets of local features generated by SuperPoint. The drawback of detector-based methods lies in their excessive reliance on the quality of keypoint detection, which is particularly unsatisfactory in low-texture and repetitive texture regions. Detector-free methods establish dense matching relationships directly between pixels, eliminating the need for keypoint detection. LoFTR [31] utilizes self-and cross-attention mechanisms on feature maps derived from CNN, producing matches progressively from coarse to fine detail. GeoFormer [20] using the RANSAC algorithm for attention region search, the computation scope of attention is confined to a specific region, enabling the use of a standard transformer for processing. While these approaches are methodologically robust, they fall short in capturing high-level contexts such as semantic information. They fail to explicitly represent the semantic correspondences within image pairs and lack clarity in their interpretability. Conversely, our method, through a Semantic-aware Fusion Block(SFB), can utilize fine-grained semantic-aware features, making our matching outcomes significantly more interpretable and accurate.Lately, methods guided by semantics have demonstrated the dependability of semantic priors. SGAM [43] incorporating semantics segmentation maps as an additional input, region matching based on identical semantics is conducted within the image before point matching. However, SGAM utilizing the direct semantic label results in the area matching's performance being highly sensitive to the precision of semantic labeling. Conversely, MESA [42] employs pure image segmentation for area matching, offering a more pragmatic approach and overcoming the limitations associated with direct semantics labels. Nevertheless, the semantic representation of this approach is coarse-grained, and semantic inconsistency still exists in the matched regions, when it comes to detailed point matching. In comparison, we explore the adaptive latest network frameworks that use a novel interplay between semantic priors and the original tasks, allowing the model to utilize rich semantic priors from a pre-trained encoder. Our approach integrates image appearance features with high-dimensional semantics, integrate fine-grained semantic into feature representation space. This allows our method to be a fully handcrafted, plug-and-play framework.C 0 , C 1 𝐹 0 , 𝐹 1 C 0 , C 1 ෪ C 0 , ෪ C 1 𝑆 0 , 𝑆 1 𝐹 0 ′ , 𝐹 1 ′ 𝐹 0 ′ , 𝐹 1 ′ 𝐼 0 , 𝐼Leveraging extensive pre-training, foundational models in vision have garnered significant achievements in the field of computer vision, shows excellent performance in various downstream tasks [8, 17, 21, 42] . Inspired by the masked language modeling [6] from natural language processing, Masked Autoencoder (MAE) [12] utilizes an asymmetric encoder-decoder architecture for masked image modeling, facilitating the effective and efficient training of scalable vision transformer models, with MAE displaying outstanding finetuning capabilities in a variety of downstream tasks. CLIP [27] acquires image representations from the ground up using 400 million image-text pairs, showcasing remarkable zero-shot image classification skills. Through discriminative self-supervised learning at both the image and patch levels, DINOv2 [24] acquires versatile visual features applicable to a broad spectrum of downstream tasks, also exhibiting notable patch-matching proficiency that identifies semantic components executing similar functions across diverse objects or species. Motivated by these developments, our objective is to utilize more fine-grained semantics from DINOv2 to direct our SRMatcher. 
approach 3.1 overall framework: 
 As our approach is built upon LoFTR, we initially provide a concise description of this method. Given a pair of input images 𝐼 0 and 𝐼 1 with dimensions 𝑤 × ℎ in gray-scale, a 2D-CNN (ResNetFPN [19] ) extracts two feature maps per image: a coarse-level feature map at 1/8 of the original size and a fine-level feature map at 1/2 of the original size. The coarse features 𝐶 0 and 𝐶 1 are input into a attention mechanisms to produce Ĉ0 and Ĉ1 . Then the coarse matching module generates a pixel-to-pixel confidence matrix 𝑃 𝑐 .Coarse-level matches 𝑀 𝑐 = {( ĩ, j)} are established by applying a threshold to 𝑃 𝑐 and conducting a mutual nearest neighbor (MNN) search. Given 𝑀 𝑐 and the fine-level features 𝐹 0 and 𝐹 1 , the finematching module computes sub-pixel matches in the following way. For each pair ( ĩ, j) ∈ 𝑀 𝑐 , a square region of dimensions 𝑤 × 𝑤 centered around î = ĩ × 4 is extracted from 𝐹 0 , and a similar region centered around ĵ = j × 4 is extracted from 𝐹 1 . The optimal correspondence for pixel î within the 𝐹 1 region determines the set of sub-pixel matches 𝑀 𝑓 .Inherited from LoFTR, we propose SRMatcher the semanticaware feature representation learning framework. The comprehensive structure of our SRMatcher is shown in Figure 2 . Starting with a pair of images, we initially process them through a CNN backbone to extract coarse and fine image features. And a frozen pre-trained semantic extractor from a large vision model to extract the semantics. Subsequently, the enhanced image features by self-and cross-attention undergo processing in our SFB module to integrate with fine-grained semantic features. Following this, coarse matching is utilized to establish patch-level matches. Ultimately, the overlap based fine-matching is employed to predict fine-level matches. With the semantically informed features, the network can perform semantic-aware feature matching with our proposed framework. The problem definition of the semantic-aware representation learning framework is as follows:Given a pair of image 𝐼 0 and 𝐼 1 ∈ R 𝐻 ×𝑊 ×3 with width 𝑊 and height 𝐻 . Combining with fine-grained semantic features, the matching process can be modeled as follow, first:EQUATIONwhere 𝑆 𝑖 represents the fine-grained semantic features. 𝐹 𝑠𝑒𝑚𝑎𝑛𝑡𝑖𝑐 denotes the pre-trained semantic extraction network, with 𝜃 𝑠 being frozen during the training stage. 𝑆 𝑖 is then utilized as input:EQUATIONwhere 𝑀 ℎ is the matched result and 𝐹 𝑚𝑎𝑡𝑐ℎ represents the matching network. Throughout the training stage, 𝜃 𝑚 is updated by minimizing the objective function with the collaboration of 𝑆 𝑖 , while 𝜃 𝑠 is frozen:EQUATIONwhere Mℎ is the ground truth, 𝐿( Mℎ , 𝑀 ℎ , 𝑆 𝑖 ) is the objective function of semantic-aware matching model. The details of our semanticaware feature representation learning framework will be elaborated in the following section.When enhancing image features using semantic features, it's crucial to address the discrepancies between the two sources. To mitigate this, we introduce the SFB module to refine the image feature maps, as depicted in Figure 3 (a) . Serving as a bridges between the semantic net and the matching net, the SFB modules create connections between these two distinct tasks. The efficacy of semantic extraction is crucial for the semanticaware feature representation learning framework. The motivation is that fine-grained semantics will enable the network to generate more precise matching results. The advanced vision foundation models (VFMs), renowned for their robust representational prowess, are ideal for this task. Particularly, the DINOv2 model stands out for its effectiveness and versatility across diverse applications [21, 39] , making it a preferred choice for semantic extraction. For our purposes, we select the pretrained DINOv2 model as our semantic extractor to obtain pixel-level semantic information.Upon acquiring the semantics 𝑆 𝑖 , we utilize the proposed Semantic-aware Fusion Block (SFB) to integrate semantic features into the matching model. With the enhanced image features of coarselevel Ĉ0 and Ĉ1 , our goal is to warp the semantic features from the images to the matching model. Within the SFB, we have designed a cross-images feature fusion strategy, as image features are to be augmented not only based on their own semantic information but also taking into account the semantic information from across different images. For the input features 𝑆 𝑖 and Ĉ𝑖 , we propose a semantic-guide interactions block (SGIB) within SFB to more effectively integrate fine-grained semantics features with image features. Semantic-guide Interactions Block (SGIB). In the first stage of SFB, the image feature and semantic features from the same image are fused. The SGIB module conducts a pixel-level interaction between image features Ĉ𝑖 and semantic features 𝑆 𝑖 and output the refined feature map C𝑖 ′ . As shown in Figure 3 (b), given the semantic features 𝑆 𝑖 , first go through a self-attention(SA) module and then supplied to the cross-attention module as query 𝑄 𝑖 , then the image features Ĉ𝑖 are identified as the keys 𝐾 𝑖 and values 𝑉 𝑖 :EQUATIONwhere C denote the channel and the Concat represent the concatenate operation. And then at the second stage of SFB, the image feature and semantic features from the different image are fused. The calculation process is similar to the first stage, the keys 𝐾 𝑖 ′ and values 𝑉 𝑖 ′ are produced by C𝑖 ′ :EQUATIONCoarse Matching. Given the fusion features C𝑖 , we adopt the LoFTR approach to generate coarse matches 𝑀 𝑐 . We apply a dualsoftmax function to process a confidence matrix 𝑃 𝑐 , where 𝑃 𝑐 (𝑖, 𝑗) signifies the probability of pixel 𝑖 and pixel 𝑗 is a correct match. This probability takes into account both low-level image information and high-level semantic information. By applying a threshold-based filter and a mutual nearest neighbor (MNN) search on 𝑃 𝑐 , we derive 𝑀 𝑐 .As mentioned in Section 3.1, we also integrate the patch-wise refinement module from LoFTR for sub-pixel matching accuracy enhancement. Retrospect that the fine matching process utilized by LoFTR is oversimplified: For every coarse matching ( ĩ, j), LoFTR finds its position ( î, ĵ) on fine-level feature map 𝐹 𝑖 and then crop two local windows based the location. After being transformed by the LoFTR module, for each local window only the center î of the enhanced feature is used to find its matched pixel ĵ′ . When î is not within the overlap of two windows, this method may result in matching errors. As a supplement, we propose an overlap based fine-matching method. After the windows cropping, we calculate pixel-to-pixel similarities between the two windows. After applying threshold-based masking, a mutual nearest neighbor(MNN) search is conducted to refine the matches in the overlap area of windows, we represent the predictions for fine-level matches 𝑀 𝑓 as:EQUATION)where W is the warping operation, ∧ is the and operation.Inspired by LoFTR, SRMatcher is trained using a self-supervised approach, eliminating the need for manual annotations. We create a geometrically transformed image 𝐼 1 from an original image 𝐼 0 by applying a predefined homography to 𝐼 0 . Utilizing this homography, the true correspondence 𝑗 in 𝐼 1 for each element 𝑖 in 𝑀 𝑓 can be precisely determined. Consequently, a set of groundtruth matches 𝐺 𝑓 corresponding to 𝑀 𝑓 is dynamically produced. Similarly, we generate ground-truth matches 𝐺 𝑐 for 𝑀 𝑐 . Data augmentation includes random sampling of homographies. We also introduce various random photometric distortions to the image pairs, such as adjustments in brightness and contrast, motion blur, and Gaussian noise. As shown in Figure 2 , the coarse matching module, and the fine matching module each generate three pixel-to-pixel similarity matrices: 𝑃 𝑐 , and 𝑃 𝑓 , respectively. Ideally, these matrices should closely align with their respective ground truths. Following, the Focal binary cross-entropy Loss (FL) is employed to optimize multiscale matching, with the loss defined as:EQUATIONThe final loss is composed of the losses for the coarse-level and the fine-level, the cumulative loss function is expressed as:𝐿 𝑡𝑜𝑎𝑡𝑙 = 𝐿 𝑐 + 𝐿 𝑓 . 
experiments 4.1 implementation details: 
 Experimental data. SRMatcher is trained on the outdoor Oxford-Paris dataset, adhering to the protocols established following [20] . This dataset is merged by the Oxford5K [25] and Paris6K [26] datasets, collectively referred to as Oxford-Paris, and encompasses a variety of outdoor and urban scenes. For evaluation, SRMatcher is tested on diverse image datasets, including HPatches [2] , ISC-HE [20] and Megadepth [18] . These test datasets feature a range of image categories, from natural scenes with pronounced viewpoint and illumination shifts to extensively edited photographs. The training dataset's registered image pairs are generated automatically, ensuring the entire network training through a completely self-supervised approach.Implementation. SRMatcher is developed in PyTorch. Given our computational resources (8 NVIDIA GeForce RTX 3090 GPUs), we set the larger image dimension to 640 for natural images during training. We use the Adam optimizer, with 𝛽 values of (0.9, 0.999) and a starting learning rate of 0.001. Each mini-batch consists of a pair of images. Training is capped at 15 epochs. SRMatcher is adapted for LoFTR [31] and its variations, including ASpan-Former [4] and GeoFormer [20] , resulting in the creation of SR-Matcher_LoFTR, SRMatcher_ASpan, and SRMatcher_GeoFormer, respectively. The distinguishing feature among these three versions lies in the attention mechanism employed within the feature interaction module, specifically linear attention [16] , span attention [4] and geometrized attention [20] . For the inference phase, with the obtained fine matches 𝑀 𝑓 , we employ cv2.findHomography with RANSAC for reliable homography determination.Test Data. The extensively utilized HPatches [2] dataset is employed, comprising 57 sequences with substantial illumination shifts and 59 sequences exhibiting significant viewpoint changes, making it a particularly demanding benchmark for homography estimation.Metrics. Following [5, 31] , we employ corner correctness to assess the accuracy of the estimated homography. The four corners from the first reference image are mapped to the second image using the estimated homography. The area under the cumulative curve (AUC) of the corner error is reported for threshold values of 1, 3, 5, and 10 pixels, respectively. For evaluation, all test images are resized to have their shorter dimension equal to 480.Comparative methods. Matching approaches are categorized into two kinds of methods, i.e. detector-based feature matching and detector-free feature matching. We compiling a selection of 11 baseline methods as detailed below: 1) Detector-based feature matching methods including SIFT [22] +RooTSIFT [1] , Superpoint [5] , SuperGlue [30] and R2D2 [28] ; 2) Detector-free feature matching methods including TopicFM [10] , AdaMatcher [14] , CasMTR [3] , MESA [42] , LoFTR [31] , ASpanFormer [4] , GeoFormer [20] . For LoFTR, ASpanFormer, GeoFormer, we test them with their original version and their upgraded version. For fair comparison we retrained all these detector-free models using the same Oxford-Paris dataset. Due to the limitation of computing resources, we choose the 4c version of CasMTR with NMS.Result. demonstrates a notable lead over alternative methodologies. SR-Matcher outperforms the previous SOTA GeoFormer in accuracy across various thresholds. SRMatcher demonstrates the most significant enhancement in AUC@1px, scoring 44.3 compared to 49.2. This superiority stems from the introduction of semantic constraints alongside geometric constraints. This integration helps eliminate unsound matches within the same set while strengthening matches based on semantic information. Qualitative results of homography estimation and matching are shown in Figure 4 and Figure 5 . From the matching result of ISC-HE, we can see the all the matched point shared same semantic.The models from Section 4.2 are utilized as is, without any further training.Test Data. We follow [20] and use the ISC-HE datasets for the evaluation of manipulated images. Images were sourced from the Facebook AI Image Similarity Challenge (ISC) [7] , where an original image undergoes various edits, like rotations and merges with other images, resulting in highly manipulated images. Given that ISC image pairs are unregistered, they were manually and collectively annotated, yielding 186 registered pairs. Each pair contains at least 8 correspondences.Results. Table 2 illustrates that SRMatcher outperforms all baseline methods, even though the performance margin is smaller compared to the HPatches experiment. This is attributed to the highly challenging nature of ISC-HE. Unlike HPatches, ISC-HE displays forgery characteristics such as watermarks, cutouts, and image stitching, which pose challenges for semantic extraction in accurately capturing the image's semantics.One thought-provoking finding from table 2 is the detectorbased methods notably SuperGlue and SIFT outperform the advanced detector-baseline, i.e. ASpan. This result can be interpreted as follows: ISC-HE images underwent significant modifications, including watermark insertion and background replacement. Unlike detector-free methods that rely on dense feature matching, detectorbased approaches are less affected by these alterations. The superior performance of SRMatcher compared to both detector-based and detector-free methods represents the robustness and accuracy of semantic-aware feature matching methods.To demonstrate the applicability of our SRMatcher to other tasks, we test our model on the MegaDepth dataset for relative pose estimation. The models described in Section 4.2 are used directly, without undergoing additional training.Test Data. MegaDepth includes 196 scene reconstructions from 1 million Internet images and 1500 pairs from two distinct scenes chosen for the test set, SRMatcher is tested in 1152 × 1152 following CasMTR [3] .Metrics. Following [42] , we present the AUC of the pose error for thresholds (5°, 10°, 20°), where the pose error is determined by the maximal angular error of relative rotation and translation. In our evaluation approach, relative poses are deduced from the essential matrix, which is calculated from feature matching using RANSAC.Results. As table 3 show, SRMatcher achieves the highest in AUC@5 • and AUC@10 • shows SRMatcher's strong task adaptability. It's important to note that MESA is designed specifically for relative pose estimation. Furthermore, SRMatcher offers significantly enhancement for GeoFormer, elevating the precision to the state-of-the-art level. This demonstrates that the semantic-aware matching provided by SRMatcher significantly augments the performance of feature matching. To assess the efficacy of our design, we perform an extensive ablation study on the components of SRMatcher on HPatches. Semantic-aware fusion block. In order to demonstrated the significance of incorporating semantic information in the context of homography estimation, as row 1 of table 4 shows, without using semantic information (i.e., w/o SFB) we can see that the performance of AUC@1 decreases from 49.2 to 45.3. This outcome highlights the superior performance of the proposed semantic-aware feature representation learning framework. To show the impact of various fusion techniques utilized in SGIB, we compare our proposed SGIB against spatial attention (i.e., w/o SGIB). Comparing rows 2 and 7, it is evident that the feature produced by our SGIB outperforms the other, achieving the best performance. It indicates that our proposed SGIB by means of executes cross-attention between semantic and image features, fully leveraging the rich semantic information present in VFMs while preserving spatial details. More details are discussed in the supplementary.Semantic Extractors. To further explore whether the finegrained semantic features improve the effectiveness of matching results, we choose two separate semantic extractors. As Table 4 shows, we compare our approach with the ResNet-50, which is pretrained on ImageNet (i.e., w/ ResNet_50). The results are reported in row 3. Compared with row 7, it's evident that the semantic features produced by DINOv2 outperform the others in terms of effectiveness. This excellent performance is due to DINOv2's self-supervised training approach, which forces the model to learn image features that remain stable under different transformations, which tend to have a high semantic level. This result highlights the potential of integrating the more refined understanding of semantics, as features enriched with semantics provide robust representational ability.Overlap based fine matching. To validate the usefulness of our overlap based fine matching, we use fine-matching2 in GeoFormer [20] as an alternative (i.e., w/o overlap based fine matching). As reported in row 6, we can see that the performance undergoes a significant drop. This result indicates the effects of our overlap based fine matching method. Fusion object. In our matching network, the semantic features 𝑆 0 and 𝑆 1 are fused with the features Ĉ0 and Ĉ1 , which are updated through self-and cross-attention. In this experiment, we try to verify the optimal fusion object. An alternative fusion type uses the coarse-level features 𝐶 0 and 𝐶 1 . Before feeding into the selfattention and cross-attention mechanisms, we use the SFB to fuse the semantic features with the coarse-level features (i.e., w/ FPN). From row 4 of table 4, it is noted that fusion with coarse-level features 𝐶 0 and 𝐶 1 results in 47.8 of AUC@1, markedly lower than the 49.2 obtained when fusion with Ĉ0 and Ĉ1 . These results indicate that Ĉ0 and Ĉ1 updated through the attention mechanism are better suited for fusion due to their enhanced position and context-related local features.Cross-images feature fusion. To explore the effects of crossimages feature fusion strategy, we only utilize the operation of stage1 within SFB as mentioned in Section 3.2 (i.e., w/o cross-images fusion). This means that each image will only be fused with its own semantic features, regardless of semantic information from other images. The results are reported in row 5. Compared with row 7, it clearly show the usefulness of our cross-images feature fusion strategy. It's the same as human intuition, when humans perform matching operations, the area with the same semantics between two images should be paid attention to, because most matching points will be generated in this area. Therefore, it is necessary to pay attention to the area with the same semantics by referring to the semantics of other images. 
limitation and discussion: 
 Firstly, the inclusion of an additional semantic extractor and Semantic-aware Fusion Block (SFB) increases the computational complexity of SRMatcher compared to previous methods. Consequently, when processing large input images, the inference time of the model is prolonged. However, computational efficiency can be enhanced by optimizing the model's structure.The generalization ability of the model is another critical aspect. When confronted with scientific data sets, such as medical images, DINOv2 may be unable to accurately identify image semantics. This is due to the limited training data set of DINOv2, which prevents generalization to more scenarios. In the future, constructing more extensive datasets for training could significantly improve its semantic extraction capabilities.GeoFormer MESA Ours , GeoFormer [20] , MESA [42] , and our SRMatcher.Megadepth 
conclusion: 
 This work has proposed a novel detector-free feature matching method for homography estimation, named SRMatcher. We have discovered that previous matching networks only utilized coarsegrained semantic features, which prevents the full utilization of the knowledge that semantic extraction networks can offer and lacks adaptability with other tasks. Different from previous works, we explore a new way to integrate semantic information with the matching network. Specifically, we incorporate fine-grained image semantics derived from vision foundation models (VFMs), using our proposed Semantic-aware Fusion Block (SFB) to conduct crossimages feature interaction. This encourages our SRMatcher to to learn integrated semantic feature representation acquiring more accurate and realistic outcomes. Additionally, our SRMatcher can be seamlessly integrated into the majority of LoFTR-based feature matching methods, where it consistently delivers outstanding results. Through comprehensive experimentation, we have explored the impact of our discoveries and highlighted the advantages of our proposed SRMatcher. We are confident that SRMatcher will provide new insights to the homography estimation community. 
 
 Appendix:  


Decision: This submission to ACM MM underwent meticulous evaluation by three expert reviewers who have all recommended acceptance. The consensus summary of their assessments follows:

Innovativeness and Contributions: The reviewers commend the novelty and efficacy of the proposed method, which makes significant strides in addressing  homography estimation. Its robust theoretical underpinnings and empirical results that surpass current methodologies demonstrate a meaningful addition to the field.
Technical Depth and Experimental Validation: The paper presents thorough technical details, rigorous experimental designs utilizing appropriate datasets, and comprehensive comparative analyses. The reviewers appreciate the strength of the experimental outcomes in supporting the authors' claims and applaud the provision of code and data, enhancing reproducibility and future research potiential.

Writing Quality and Clarity: Structurally sound and logically coherent, the manuscript is well-argued with effective use of figures and supplementary materials to augment explanations. While minor suggestions for refinement or additional contextual information were mentioned, these do not detract from the overall readability or comprehension.

Limitations and Recommendations for Improvement:
Despite minor critiques and suggestions for enhancements, such as questions of Reviewer BrsR, these are viewed as avenues for future work rather than barriers to acceptance. The authors have pledged in their response to consider these recommendations for refinement.

Conclusion and Recommendation:

Given the paper's innovativeness, technical depth, and potential impact on the fields of computer vision and pattern recognition, coupled with the unanimous positive feedback from all three reviewers, this meta-review recommends the acceptance of the paper for presentation at ACM MM, either orally or as a poster. We anticipate that the findings will spark keen interest and foster productive discussions among conference attendees, thereby contributing to advancements in the domain.

Reviews: review: paper_summary: paper_summary: The paper introduces SRMatcher that integrates semantic information into feature matching for improved homography estimation. Using DINOv2 for semantic extraction and a Semantic-aware Fusion Block (SFB) to merge features across images, SRMatcher enhances matching accuracy. It achieves state-of-the-art performance on various datasets and improves other vision tasks, though it increases computational complexity and may have generalization limitations.
strength_weakness: strength_weakness: The structure of the paper is very clear, and the authors clearly articulate the features of the method. The experimental setup is quite reasonable, and the integration experiments are also sufficiently comprehensive. Several recent top detectors. e.g., LoFTR, ASpan, and GeoFormer, all achieve better results.
limitations: limitations: The paper lacks sufficient novelty. I have a question: Is the improvement of the model mostly achieved from the DINOv2 backbone? What if I change the model to VGG16 or ViT? What would be the performance of the model then? Additionally, what is the parameter count of the model compared to LoFTR and ASpan? Is there a significant increase?
paper_summary: paper_summary: This paper introduces SRMatcher as a detector-free feature matching method for homography estimation task. Unlike previous approaches that use coarse-grained semantic pre-processing, SRMatcher integrates semantic-aware feature representation using vision foundation models and a Semantic-aware Fusion Block. This reduces errors from semantic inconsistencies and improves accuracy. Experiments show SRMatcher outperforms previous methods and can other existing matching methods.
strength_weakness: strength_weakness: 1. The structure of the proposed framework is concise and rational, especially for the proposed SFB and SGIB modules that are simple but suitable for homography estimation.
2. The performance improvement when using the presented SRMatcher are quite sizable and can be observed on three different test datasets for homography estimation.
3. The qualitative results show the remarkable advantages of SRMatcher compared to other existing approaches.
4. The paper is understandable, Figures and captions make sense and supplement the paper well.
limitations: limitations: 1. The SFB illustration shows two stages designed to refine image feature maps. However, stage 2 can be repeated multiple times to further refine features with minimal additional time cost, suggesting the possibility of more than two stages. The choice of limiting to two stages as the optimal configuration needs validation.
2. What would happen if the pre-trained semantic extractor from a large vision model were not frozen or updated by the other encoder using exponential moving average? The authors should provide explanations regarding this aspect.
3. While ablation studies on different components of SRMatcher are presented, additional studies on the impact of different scales of the coarse-level feature map and the threshold in the masking filter within the sub-pixel refinement module are necessary to demonstrate the effectiveness of these settings.
paper_summary: paper_summary: This paper mainly discusses a novel detector-free feature matching method called SRMatcher, which is used to address the task of determining the transformation (homography estimation) between a pair of images. In contrast to previous methods that relied on treating semantic information as a preprocessing step, leading to coarse-grained semantic application and lack of adaptability to different tasks, SRMatcher focuses on utilizing a semantic-aware feature representation learning framework. The aim of this method is to improve feature matching by learning integrated feature representations that incorporate semantic information.
strength_weakness: strength_weakness: A new method has been proposed for integrating semantics into feature matching models for homography estimation.
limitations: limitations: 1.Homography is typically assumed to be a rigid transformation, but in reality, some scenes may involve non-rigid transformations. How does the method in the paper address and achieve good results in such cases?

2.While semantic information plays a crucial role in improving the accuracy of feature matching, in some cases such as occluded scenes, extreme lighting variations, or low contrast images, relying solely on semantic information may not be sufficient to obtain accurate matching results. How does the method overcome these challenges?