Title: Improving Graph Matching with Positional Reconstruction Encoder-Decoder Network
Venue: NeurIPS 2023
Accepted: yes
Keywords: Graph Matching

Positional Encoding
Abstract: Deriving from image matching and understanding, semantic keypoint matching aims at establishing correspondence between keypoint sets in images. As graphs are powerful tools to represent points and their complex relationships, graph matching provides an effective way to find desired semantic keypoint correspondences. Recent deep graph matching methods have shown excellent performance, but there is still a lack of exploration and utilization of spatial information of keypoints as nodes in graphs. More specifically, existing methods are insufficient to capture the relative spatial relations through current graph construction approaches from the locations of semantic keypoints. To address these issues, we introduce a positional reconstruction encoder-decoder (PR-EnDec) to model intrinsic graph spatial structure, and present an end-to-end graph matching network PREGM based on PR-EnDec. Our PR-EnDec consists of a positional encoder that learns effective node spatial embedding with the affine transformation invariance, and a spatial relation decoder that further utilizes the high-order spatial information by reconstructing the locational structure of graphs contained in the node coordinates. Extensive experimental results on three public keypoint matching datasets demonstrate the effectiveness of our proposed PREGM.

Improving Graph Matching with Positional Reconstruction Encoder-Decoder Network

Abstract

Deriving from image matching and understanding, semantic keypoint matching aims at establishing correspondence between keypoint sets in images. As graphs are powerful tools to represent points and their complex relationships, graph matching provides an effective way to find desired semantic keypoint correspondences. Recent deep graph matching methods have shown excellent performance, but there is still a lack of exploration and utilization of spatial information of keypoints as nodes in graphs. More specifically, existing methods are insufficient to capture the relative spatial relations through current graph construction approaches from the locations of semantic keypoints. To address these issues, we introduce a positional reconstruction encoder-decoder (PR-EnDec) to model intrinsic graph spatial structure, and present an end-to-end graph matching network PREGM based on PREnDec. Our PR-EnDec consists of a positional encoder that learns effective node spatial embedding with the affine transformation invariance, and a spatial relation decoder that further utilizes the high-order spatial information by reconstructing the locational structure of graphs contained in the node coordinates. Extensive experimental results on four public keypoint matching datasets demonstrate the effectiveness of our proposed PREGM.

1 Introduction

Image matching is a common and basic matching problem in multimedia and visual applications. The goal of image matching is to find a dense semantic content correspondence between two images, while feature-based matching is a typical strategy to solve the above problems, and in feature matching,

the semantic keypoints in images are the most commonly used matching targets. Semantic keypoint matching aims at establishing correspondence between keypoint sets in images, and it has been applied to a wide range of applications, such as object tracking [35, 24], image retrieval [40, 11], and pose estimation [39, 3], etc.

The semantic keypoint data has the properties of non-Euclidean and unstructured for its disorder and discreteness, and graphs are powerful tools to represent some complex objects and their interactions. Specifically, we take each semantic keypoint as a node, and build edges by heuristic methods according to the positions of nodes to construct a corresponding graph. Graph matching is an effective method to solve the correspondence between the semantic keypoints in two images.

As deep learning models are scalable and time-efficient, Graph Neural Networks (GNNs) are increasingly utilized in graph matching tasks, such as GMN [44], PCA [32], and BBGM [29], etc.

However, most graph matching approaches rely on the matching of basic items of graphs, i.e. computing affinities of nodes and edges, lack the mining and utilization of spatial context information hidden in the locations of keypoints as nodes of graphs. Recently, in task scenarios such as point cloud and knowledge graph, which the nodes in graphs have locational information, there are methods such as [21, 18] utilizing positional encoder to add the spatial information as additional features. However, current methods in graph matching domain lack such spatial positional information supplementation. As the positional information is closer to the intrinsic characteristic of the graph, it is important to integrate spatial descriptions into graph feature embeddings in graph matching.

But utilizing underlying spatial information faces the following challenges: 1) Through taking key points as graph nodes and edges are associated with pairwise distance, such as delaunay triangulation, it is not sufficient to mine hidden relative positional relationship. 2) The description of spatial information from coordinates of keypoints is relatively simple compared with conventional visual features, so it needs more refined network structure to learn location information, in order to facilitate subsequent feature fusion.

To address the two challenges mentioned above, we propose a positional reconstruction encoderdecoder network (PR-EnDec) in our PREGM by integrating more spatial information into semantic features and learning more distinguishable node features. The positional encoder in PR-EnDec aims to learn the encoding of node positional information with spatial invariance, specifically, the affine transformation invariance of node coordinates. The spatial relation decoder in PR-EnDec is designed to recover the locational structure from the learned features of the encoder, and it reconstructs the relative distances and areas among the sampled nodes. To implement the joint learning of PR-EnDec about the intrinsic positional features in graphs, we, consequently, propose a corresponding contrastive loss function and a reconstruction loss function in the encoder and the decoder respectively.

Our main contributions are summarized as follows:

We propose an end-to-end graph matching network, PREGM, which learns more distinguishable node features, not only captures visual information in images but also models intrinsic spatial structure of keypoints by our designed PR-EnDec.

Our PR-EnDec consists of a positional encoder that learns effective graph positional encoding with affine transformation invariance, and a spatial relation decoder used to enhance such learning by reconstructing the graph locational information such as relative distances and areas from learned positional encoding. We also design a corresponding contrastive loss function and a reconstruction loss function in the encoder and decoder respectively to help learn effective node positional characteristics.

We evaluate our PREGM on four public keypoint matching datasets. Our experimental results demonstrate that PREGM outperforms state-of-the-art methods. We also present an ablation study, which shows the effectiveness of each component of PREGM.

2 Related Work

2.1 Graph Matching

Graph matching is always an effective strategy to find correspondence between semantic keypoints as graphs have sufficient representation ability for points and their complex relationships, and it can then be applied to downstream tasks such as image matching and understanding. As it is famous for being one of the practically most difficult NP-complete problems, researchers often use optimization algorithms to find an acceptable suboptimal solution for the graph matching problem. Traditional graph matching methods [17, 5, 7, 20, 36] generally use heuristic search algorithms to find suboptimal solutions.

Recent graph matching methods begin to combine combinatorial optimization with deep learning models. GMN [44] proposes an end-to-end graph matching framework, calculates the unary and pairwise affinities of nodes through CNN features, and also uses spectral matching as a differentiable solver to combine deep learning with graph matching. PCA [32] uses graph neural network (GNN) to learn and aggregate graph structure information into node feature representation, and introduces Sinkhorn network [1] as the combinatorial solver for finding node correspondence. BBGM [29] presents an end-to-end trainable architecture that incorporates a state-of-the-art differentiable combinatorial graph matching solver, and introduces a global attention mechanism to improve the matching performance.

Although the research on graph matching has made the above progress, the existing methods lack the distinguishing and informative graph spatial feature descriptions, and adaptive adjustment of node correspondence on graph matching. In this paper, we overcome these limitations by proposing a positional reconstruction encoder-decoder (PR-EnDec) and designing a contrastive learning and positional feature reconstruction procedure to guide the learning of graph matching.

2.2 Positional Encoding on Graphs

Positional Encoding was first proposed in the Transformer [31] model. In general, positional encoding refers to the process of encoding spatial relationships or topological information of elements as supplementary information in order to augment the representational capacity of a model. The categorization of positional encoding can be further elaborated as follows: spatial positional encoding utilizes point location as input, while topological positional encoding employs the graph structure as input.

For spatial positional encoding, the point location is used as input to obtain a high-dimensional embedding that contains effective locational information. Space2Vec [21] introduces theory and grid as a 2D multi-scale spatial positional encoder by using sinusoidal functions with different frequencies. PointCNN [18] designs a point set spatial positional encoder with Point Conv layers consisting of sampling layer, grouping layer, and aggregation layer. DGCNN [38] considers both global shape structure and local neighborhood information, and proposes spatial positional encoder layers containing dynamic graph neighborhood and edge convolution module. This paper proposes a spatial position encoding on graphs by leveraging the characteristic of node representation containing coordinate information in image keypoint matching.

Topological positional encoding is specific to graph matching, where the graph structure is used as input to obtain the topological embedding of the nodes. Maskey et al. [22] generalizes Laplacianbased positional encoding by defining the Laplace embedding to more general dissimilarity functions such as p-norm rather than the 2-norm used in the original formulation. INFMCS [16] designs a permutation-invariant node ordering based on closeness centrality, and effectively enhances the representation of graph structural features.

In recent years, there has also been research on spatial positional encoding in the field of graph matching. GCAN [13] utilizes SplineCNN [10] to encode positional information in 2D spatial space, and successfully capture structural information of the graph.

However, there remains the open question of how to further mine intrinsic spatial node information in graphs. In this paper, we propose a positional reconstruction encoder-decoder architecture to obtain effective node spatial positional encoding using 2D point location.

𝔾1

(

𝑥

𝑖

𝐹 𝑣𝑖𝑠

1

VGG16_bn

Feature Fusion

𝐹 𝑝𝑜𝑠

1

𝔾2

(

𝐹 𝑣𝑖𝑠

2

𝐹 𝑝𝑜𝑠

2

Node

Affinity

Computation

𝐹 𝑓𝑢𝑠𝑒𝑑

1

X

Self-Attention

+

Normalization

2-layer MLP

Pretrained Positional Encoder

𝐹 𝑓𝑢𝑠𝑒𝑑

2

ℒ𝐺𝑀

(x,y)

SplineCNN

𝑖

𝑥

3 Problem Formulation of Graph Matching

Given the keypoints in the images, each node is generally associated with a keypoint, and edges are built according to the spatial position relationship between nodes when building an undirected graph. The graph is represented by G = ( V , E , V , E ) :

V = { v 1 , ..., v n } denotes the node set.

E ⊆ V × V denotes the edge.

V = { v i | v i ∈ R d v , i = 1 , 2 , ..., | V |} denotes the node feature set.

E = { e i | e i ∈ R d e , i = 1 , 2 , ..., | E |} denotes the edge feature set.

We use the adjacency matrix A to represent the connections of nodes in an undirected graph G , that is, A ij = 1 iff there is an edge e ij = ( v i , v j ) ∈ E .

Given two graphs G 1 = ( V 1 , E 1 , V 1 , E 1 ) and G 2 = ( V 2 , E 2 , V 2 , E 2 ) , where | V 1 | = | V 2 | = n , the goal of graph matching is to find the correspondence matrix X ∈ { 0 , 1 } n × n between nodes V 1 and V 2 of the two graphs G 1 and G 2 , wherein X ij = 1 iff v i ∈ V 1 corresponds to v j ∈ V 2 .

The ultimate goal of the graph matching problem is to find the optimal node correspondence between two graphs G 1 and G 1 , so the graph matching problem is often considered as a quadratic assignment programming problem (QAP problem):

where x = vec ( X ) ∈ { 0 , 1 } n 2 , X1 n = 1 n , and X T 1 n = 1 n . x ∗ denotes the desired node correspondence, and 1 n denotes a vector of n ones. K ∈ R n 2 × n 2 is the corresponding affinity matrix:

where s v ia represents the affinity between node features v i ∈ V 1 and v a ∈ V 2 , s e ia,jb represents the affinity between edge features e ij ∈ E 1 and e ab ∈ E 2 .

4 Methodology

The framework of our PREGM Network is illustrated in Figure 1. The positional encoder module of pretrained PR-EnDec takes the coordinates of keypoints as input. It extracts a positional encoding vector for each keypoint in the images. Next, the positional encoding and the visual features extracted by vgg16_bn are further fused. Then, the fused features are fed into the message-passing module to obtain the refined node feature. Finally, the node affinity is computed and processed by the LPMP solver, generating the node correspondence matrix X introduced in the problem formulation.

2D Cartesian

coordinates

ℒ𝑐𝑜𝑛

𝐹 𝑝𝑜𝑠

Triangle

Sampling

𝜌𝑑

𝜌𝑎

𝑑𝑖𝑠 1

𝑑𝑖𝑠 3

𝑑𝑖𝑠 2

𝑎𝑟𝑒𝑎

Reconstruction

ℒ𝑟𝑒𝑐

Positional Encoder

Spatial Relation Decoder

𝜌 𝑣

𝐴𝑡𝑡𝑛

𝑁𝑜𝑟𝑚

𝑅𝐶 = [𝑑𝑖𝑠1 , 𝑑𝑖𝑠

2

, 𝑑𝑖𝑠

3

, 𝑎𝑟𝑒𝑎]

MLP

MLP

𝐴𝑡𝑡𝑛

Self-Attention

Self-Attention

𝐹 𝑝𝑜𝑠

𝐹𝑠𝑢𝑏

As an independent stage, we pretrain the abovementioned positional encoder along with the spatial relation decoder module of PR-EnDec (as shown in Figure 2). The pretraining stage enables the positional encoder to learn the high-order spatial information. The PR-EnDec consisting two modules,

Positional Encoder. This module takes the coordinates of keypoints as input and learns positional encoding. We first obtain the high-dimensional coordinate embedding vectors by an MLP. The vectors are then fed into self-attention blocks and a normalization layer to learn positional encoding. The attention mechanism provides sequence independence and relative information of nodes. Besides, we propose a contrastive loss for the encoder module. The loss function ensures the affine transformation invariance and learns the relative position information.

Spatial Relation Decoder. To reconstruct the spatial structure of graphs, this module generates triangle assignments by distance-based random sampling. In each triangle assignment, the module takes positional encoding corresponding to the three keypoints as the input of self-attention blocks. Next, the processed features are fed into MLPs, and the relative spatial relations of the three points are reconstructed. We adopt a Mean Squared Error (MSE) loss to compare the ground truth relative spatial relations and the decoder's predictions.

4.1 Positional Encoder

The Positional Encoder only takes geometric features of keypoints as inputs,

V g = { v g i | v g i ∈ R d g v , i = 1 , ..., n } denotes the node geometric feature set.

Specifically, we utilize the 2D Cartesian coordinates of each node v i as its geometric feature v g i = [ x i , y i ] . The coordinate embedding sub-module embeds the geometric feature V g into latent representations by a two-layer MLP, denoted as ρ v . The updated geometric graph G g = ( V , ρ v ( V g )) is then fed into the following attention sub-module.

The attention sub-module consists of l m attention blocks (denoted as Attn e i ) and a normalization layer (denoted as Norm ), which updates latent representations of G g . An attention block is composed of a multi-head self-attention layer and a feed-forward layer, which captures the spatial feature of G g by aggregating the latent representations of each node.

The self-attention mechanism is sequence-independent, which is well suited for extracting high-order relative position information between nodes. There is a normalization layer after attention blocks to allow the positional encoding of nodes to be better fused with visual features nodes on the graph matching task. The final positional encoding obtained by the encoder module is denoted as F pos .

Encoder Loss. For two given graphs G 1 = ( V 1 , E 1 , V 1 , E 1 ) and G 1 = ( V 1 , E 1 , V 1 , E 1 ) , the positional encoder generates the positional encoding denoted as F 1 pos = { f 1 i | i = 1 , 2 , ..., | V |} and F 2 pos = { f 2 i | i = 1 , 2 , ..., | V |} respectively. We expect that for all matched node i in graph G 1 and node j in graph G 2 , the corresponding node positional encoding f 1 i and f 2 j should be similar. To let the encoder learn the relative position information of the nodes, we perform contrastive learning that

perseves the affine invariance of the positional encoding. Specifically, we adopt negative samples to conduct contrastive learning to avoid the encoder module outputs trivial node features that are all similar.

Therefore, we classify ( G 1 , G 2 ) as a positive graph pair if V g 2 is affine transformed by V g 1 or the keypoints in V g 1 and V g 2 are one-by-one matched. On the contrary, ( G 1 , G 2 ) is a negative graph pair if the keypoints are not matched.

For each training batch, the positive graph pair set GS p and negative graph pair set GS n are generated by node permuting or affine transformation. Let FS p and FS n denote the corresponding positional encoding pair set of GS p and GS n respectively, we propose a contrastive loss L con ,

where S p and S n denote the sum of the exponential of positive/negative pairs' similarity, respectively. Additionally, τ denotes the temperature constant, and sim () denotes the cosine similarity.

4.2 Spatial Relation Decoder

To learn the high-order spatial information, we propose k -th-order geometric reconstruction assignments. Sampling k points from the node set as an assignment, the corresponding geometric feature set is denoted as V g s and positional encoding set as F sub ⊆ F pos . If a decoder Dec takes F sub as input, we claim the decoder provides k -th-order relative position information if there exists a non-degenerate geometric function F , F ( V g s ) ≈ Dec ( F sub ) . Specifically, we choose k = 3 and function F with Euclidean distance and triangle area to learn the third-order relative position information.

The decoder module first samples three keypoints and obtains the corresponding geometric feature set V g s = { v g a , v g b , v g c } and positional encoding set F sub = { f a , f b , f c } . The random sampling method is related to the Euclidean distance from the previous sampled node to learn the relative information better, for the sampling method guarantees that the distances between the sampled nodes are not too far. Next, the sampled feature set F sub is fed into l n attention blocks. The intermediate feature is denoted as ˆ F sub = { ˆ f a , ˆ f b , ˆ f c } . We utilize 2 MLPs, denoted as ρ d and ρ a , to approximate Euclidean distance and triangle area function, respectively. The reconstructed relative position information of the decoder is represented as:

Let Dis denotes Euclidean distance and Area denotes triangle area function, RC 's corresponding geometric function F is obviously

where dis 1 = Dis ( v g a , v g b ) , dis 2 = Dis ( v g b , v g c ) , dis 3 = Dis ( v g a , v g c ) , area = Area ( v g a , v g b , v g c ) denote ground-truth third-order relative position information for the decoder to reconstruct.

Decoder Loss. Since we calculate the approximated relative position information RC and the ground-truth RC gt = F ( V g s ) , we propose the reconstruction loss L rec :

PR-EnDec Loss. Finally, we combine the two losses to guide the training of our PR-EnDec jointly,

where λ controls the relative importance of L rec .

Through pretraining of PR-EnDec, we obtain an effective encoder for extracting spatial positional encoding for utilization in subsequent PREGM.

4.3 Graph Matching with PR-EnDec

Our training procedure is divided into two stages: In the first stage, we pretrain our PR-EnDec, and in the second stage of graph matching, our positional encoder module of PR-EnDec serves as a sub-module of the base graph matching model, providing learned positional encoding in the first stage.

After the training of PR-EnDec, we freeze all parameters of the encoder module and put it in the training of graph matching tasks. The positional encoding F pos generated by the positional encoder module is fused with visual features F vis extracted by a standard CNN:

where Linear denotes a linear layer.

Next, we feed the fused features into a message-passing module to produce the refined node features, and the message-passing module is implemented by a two-layer SplineCNN [10]. After the messagepassing module, node and edge affinities are computed and passed to the differentiable graph matching solver LPMP in [25]. The resulting correspondence matrix X is compared to the ground truth X gt and the loss function is their Hamming distance:

So far, the graph matching problem formulated previously is solved by our PREGM with the two sequential training phases: the PR-EnDec pretrain and the graph matching task.

5 Experiments

Weconduct experiments on four public keypoint matching datasets and verify the effectiveness of each component of our PREGM by ablation study. We introduce the datasets, baselines, implementation details, and then report the results. The results demonstrate that our PREGM consistently outperforms all other approaches.

5.1 Datasets

We evaluate our PREGM on four public keypoint matching datasets: PascalVOC [8], Willow ObjectClass [4], SPair-71k [23], and IMC-PT-SparseGM [14].

The PascalVOC dataset includes 20 classes of keypoints with Berkeley annotations [2] and images with bounding boxes. The PascalVOC dataset is relatively challenging, since the scale, pose and illumination of instances in images are rich and diverse, and the number of annotated keypoints in each image also varies from 6 to 23. When conducting experiments on the PascalVOC dataset, we follow the standard protocol [32, 37]: First, each object is cropped according to its corresponding bounding box and scaled to 256 × 256 px. Second, we use 7,020 images for training and 1,682 for testing.

The Willow ObjectClass dataset contains images of five categories: face, duck, winebottle, car, and motorbike, the first three categories are from the Caltech-256 dataset [12], and the last two categories are from the PascalVOC 2007 dataset [8]. Each category contains at least 40 different images, and each image is labeled with 10 distinctive keypoints on the target object to be matched. Following the default setting in [4, 32], we crop the images to the bounding boxes of the objects and rescale to 256 × 256 px, 20 images of each class are selected during training, and the rest are for testing.

The SPair-71k dataset is a relatively novel dataset, which was recently published in the paper [23] about dense image matching. Spair-71k contains 70958 image pairs of 18 categories from PascalVOC 2012 dataset [8] and Pascal 3D+ dataset [41]. Compared with the other two datasets, SPair71-k has the advantages of higher image quality and richer annotations which includes detailed semantic keypoints, object bounding boxes, viewpoints, scales, truncation, and occlusion differences of image pairs. In addition, compared with PascalVOC dataset, SPair-71k removes two classes with ambiguous and poor annotations: sofa and dining table. Following [29], we use 53,340 image pairs for training, 5,384 for validation, and 12,234 for testing, and we also scale each image to 256× 256 px.

The IMC-PT-SparseGM dataset contains 16 object categories and 25061 images [14], which gather from 16 tourist attractions around the world. The IMC-PT-SparseGM benchmark involves matching

larger scenes and take a step closer to the real-world downstream tasks. We take 13 classes as the training set and the other 3 classes as the test set. Experiments are conducted on the benchmark with 50 anchors.

5.2 Baselines and Metrics

We compare our PREGM with the state-of-the-art methods, including two types of baselines: (1) 2 non-learning methods GNCCP [20], ABPF [36]; (2) 11 learning-based methods GMN [44], PCA [32], CIE [43], NGM [33], DGMC [9], EAGM [27], BBGM [29], DIP-GM [42], IA-NGM-v2 [26], ASAR [28], COMMON [19]. For a fair comparison, we follow previous work [32] and extract node visual initial features from relu4_2 and relu5_1 of vgg16_bn [30] via feature alignment. For the input image pairs, we construct the edges of graphs by Delaunay triangulation [6]. We adopt a common metric matching accuracy to evaluate the experimental results, which is computed as the number of correctly matched keypoint pairs averaged by the total number of all true matched keypoint pairs.

5.3 Experimental Settings

In our implementation, we build up our PREGM based on the state-of-the-art method [29], which presents an end-to-end trainable architecture that incorporates a state-of-the-art differentiable combinatorial graph matching solver. Our codes will be available in the future. In all experiments, we use the same set of hyperparameters. We employ Adam [15] optimizer with an initial learning rate of 1 × 10 -4 for PR-EnDec, and 9 × 10 -4 for other models, and the learning rate is halved every three epochs. We empirically set l m = 3 and l n = 2 in the encoder and decoder, and choose temperature constant τ = 2 in L con , and balance factor λ = 1 / 32 in L PR -EnDec . We also set batch size = 8 for PascalVOC, Willow ObjectClass, and Spair-71k datasets. Our learning procedure is divided into two stages: The pre-training of the PR-EnDec is uniformly conducted on the PascalVOC dataset. Then the graph matching framework is trained on each dataset, while the parameters of the positional encoder are fixed. All experiments are run on a single GTX-1080Ti GPU.

5.4 Performance Evaluation

We conduct experiments on four public datasets: PascalVOC, WillowObject Class, Spair-71k and IMC-PT-SparseGM for the keypoint matching problem, and follow the most common experimental setting, where intersection filtering is applied to generate graphs with the equal size.

Firstly, we report the matching accuracy of the 20 classes and average accuracy on PascalVOC dataset in Table 1, where the best results are shown in bold. The results demonstrate that our model PREGM performs much better than all traditional graph matching methods, and also achieves better performance against state-of-the-art deep graph matching models with matching accuracy 84.1%. And in the 20 classes of PasalVOC dataset, our PREGM has achieved the best results in the other 17 classes except for the boat, bottle and train class. As there are large differences in the scale, pose and illumination of matching objects, PascalVOC is a complex matching dataset, thus the matching accuracy of traditional graph matching methods is not more than 50%, and the performance of deep graph matching methods is relatively better. Our PREGM learns effective node spatial characteristics by PR-EnDec, so as to further improve the graph matching performance.

For the relatively simple WillowObject Class dataset, as shown in Table 2, there are two different training strategies: PT and WT, which PT means matching frameworks are pre-trained on PascalVOC, and WT means learning models are then fine-tuned on WillowObject Class Dataset. In this paper, we adopt two strategies: PT only, both PT and WT, and in both situations, our model achieves the best performance among other state-of-the-art methods with matching accuracy 98.2% and 99.3%. The results again demonstrate the effectiveness of learning positional node features hidden in graphs in our PREGM.

We also conduct experiments on Spair-71k dataset, as shown in Table 3, we compare our PREGM with deep learning methods GMN, PCA, DGMC, BBGM, and DIP-GM. Our PREGM performs best with matching accuracy 81.9%, and shows superiority in total 18 classes, which demonstrates the generalization ability of our model on different objects. SPair-71k dataset, as a relatively new dataset, has the advantages of high image quality, rich annotation, and fixed image pairs for matching, which is a more convincing dataset. Thus our PREGM achieves the best results on the SPair-71k dataset, which further proves the effectiveness of our PREGM.

Additionally, we evaluate our model on the IMC-PT-SparseGM dataset, as shown in Table 4, our model demonstrates outstanding performance on this demanding benchmark. The results outshine the performance of the other methods, GANN-GM [34] and BBGM, by a significant margin. In terms of the average accuracy across these landmarks, our model excels with an impressive mean accuracy of 90.7%. The IMC-PT-SparseGM dataset is characterized by its substantial number of images, nodes, and high partial rate, making it one of the most comprehensive benchmarks for visual graph matching. Our approach, showcased in these results, demonstrates its ability to handle larger scenes and move closer to real-world applications, such as structure from motion.

5.5 Ablation Study and Parameter Analysis

To evaluate the effect of each component in our framework, we conduct comprehensive ablation studies with/without positional encoder, spatial relation decoder, and we also consider the two separate cases of no reconstruction of areas and relative distances in the loss function of spatial relation decoder. Furthermore, we conducted additional ablation experiments by removing visual features to evaluate the effectiveness of the spatial features extracted by our model. The experiments are performed on PascalVOC dataset. As shown in Table 5, compared with the baseline, the results demonstrate that all modules bring substantial performance gains, the reconstruction of relative distance is more important in the decoder, and the spatial relation decoder contributes most to the overall performance. Moreover, with the fact that only taking spatial features as node features can achieve relatively good performance, it further proves the effectiveness of the spatial features learned from our PR-EnDec.

Wealso conduct parameter analysis to select hyperparameters. As shown in Table 6, PREGM achieves the best performance when learning rate = 9 × 10 -4 , which is our default setting, and it also shows that adjusting the learning rate causes an accuracy fluctuation of about 1%. For the balance factor λ in the loss function, when λ = 1 / 128 , 1 / 32 , and 1 / 8 , the matching accuracy is 83.7%, 84.1%, and 83.3% respectively. Thus we select λ = 1 / 32 as our default setting, and the results show that our method is rather sensitive to the choice of λ , and our designed loss function in positional encoder and spatial relation decoder indeed improves the performance.

6 Conclusion

In this paper, we present an end-to-end novel deep graph matching network PREGM that learns more distinguishable node features by modeling spatial positional information in graphs. Our PREGM designs a common encoder-decoder architecture consisting of a positional encoder that learns effective node positional encoding and a spatial relation decoder that reconstructs the positional structure of graphs. Our experiments and ablation studies on four public keypoint matching datasets demonstrate the state-of-the-art performance of our method. The exploration direction of future work includes optimizing the loss function of the positional encoder to extract purer spatial structural information and improve the feature fusion method to obtain better fused feature representation.

References

Ryan Prescott Adams and Richard S Zemel. Ranking via sinkhorn propagation. arXiv preprint arXiv:1106.1925 , 2011.

Lubomir Bourdev and Jitendra Malik. Poselets: Body part detectors trained using 3d human pose annotations. In 2009 IEEE 12th International Conference on Computer Vision , pages 1365-1372. IEEE, 2009.

Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. Realtime multi-person 2d pose estimation using part affinity fields. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 7291-7299, 2017.

Minsu Cho, Karteek Alahari, and Jean Ponce. Learning graphs to match. In Proceedings of the IEEE International Conference on Computer Vision , pages 25-32, 2013.

Minsu Cho, Jungmin Lee, and Kyoung Mu Lee. Reweighted random walks for graph matching. In European conference on Computer vision , pages 492-505. Springer, 2010.

Paolo Cignoni, Claudio Montani, and Roberto Scopigno. Dewall: A fast divide and conquer delaunay triangulation algorithm in ed. Computer-Aided Design , 30(5):333-341, 1998.

Amir Egozi, Yosi Keller, and Hugo Guterman. A probabilistic approach to spectral graph matching. IEEE transactions on pattern analysis and machine intelligence , 35(1):18-27, 2012.

Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision , 88(2):303-338, 2010.

M. Fey, J. E. Lenssen, C. Morris, J. Masci, and N. M. Kriege. Deep graph matching consensus. In International Conference on Learning Representations (ICLR) , 2020.

Matthias Fey, Jan Eric Lenssen, Frank Weichert, and Heinrich Müller. Splinecnn: Fast geometric deep learning with continuous b-spline kernels. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 869-877, 2018.

Ivan Gonzalez-Diaz, Murat Birinci, Fernando Diaz-de Maria, and Edward J Delp. Neighborhood matching for image retrieval. IEEE Transactions on Multimedia , 19(3):544-558, 2016.

Gregory Griffin, Alex Holub, and Pietro Perona. Caltech-256 object category dataset. 2007.

Zheheng Jiang, Hossein Rahmani, Plamen Angelov, Sue Black, and Bryan M Williams. Graphcontext attention networks for size-varied deep graph matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2343-2352, 2022.

Yuhe Jin, Dmytro Mishkin, Anastasiia Mishchuk, Jiri Matas, Pascal Fua, Kwang Moo Yi, and Eduard Trulls. Image matching across wide baselines: From paper to practice. International Journal of Computer Vision , pages 517-547, 2021.

Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR) , 2015.

Zixun Lan, Binjie Hong, Ye Ma, and Fei Ma. More interpretable graph similarity computation via maximum common subgraph inference. arXiv preprint arXiv:2208.04580 , 2022.

Marius Leordeanu, Martial Hebert, and Rahul Sukthankar. An integer projected fixed point method for graph matching and map inference. Advances in neural information processing systems , 22, 2009.

Yangyan Li, Rui Bu, Mingchao Sun, Wei Wu, Xinhan Di, and Baoquan Chen. Pointcnn: Convolution on x-transformed points. Advances in neural information processing systems , 31, 2018.

Yijie Lin, Mouxing Yang, Jun Yu, Peng Hu, Changqing Zhang, and Xi Peng. Graph matching with bi-level noisy correspondence, 2022.

Zhi-Yong Liu and Hong Qiao. Gnccp-graduated nonconvexityand concavity procedure. IEEE transactions on pattern analysis and machine intelligence , 36(6):1258-1267, 2013.

Gengchen Mai, Krzysztof Janowicz, Bo Yan, Rui Zhu, Ling Cai, and Ni Lao. Multiscale representation learning for spatial feature distributions using grid cells. arXiv preprint arXiv:2003.00824 , 2020.

Sohir Maskey, Ali Parviz, Maximilian Thiessen, Hannes Stärk, Ylli Sadikaj, and Haggai Maron. Generalized laplacian positional encoding for graph representation learning. In NeurIPS 2022 Workshop on Symmetry and Geometry in Neural Representations .

Juhong Min, Jongmin Lee, Jean Ponce, and Minsu Cho. Hyperpixel flow: Semantic correspondence with multi-layer neural features. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 3395-3404, 2019.

Hyeonseob Nam and Bohyung Han. Learning multi-domain convolutional neural networks for visual tracking. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 4293-4302, 2016.

Marin Vlastelica Poganˇ ci´ c, Anselm Paulus, Vit Musil, Georg Martius, and Michal Rolinek. Differentiation of blackbox combinatorial solvers. In International Conference on Learning Representations , 2019.

Tianxiang Qin, Shikui Tu, and Lei Xu. Ia-ngm: A bidirectional learning method for neural graph matching with feature fusion. Machine Learning , pages 1-27, 2022.

Jingwei Qu, Haibin Ling, Chenrui Zhang, Xiaoqing Lyu, and Zhi Tang. Adaptive edge attention for graph matching with outliers. In IJCAI , pages 966-972, 2021.

Qibing Ren, Qingquan Bao, Runzhong Wang, and Junchi Yan. Appearance and structure aware robust deep visual graph matching: Attack, defense and beyond. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 15263-15272, 2022.

Michal Rolínek, Paul Swoboda, Dominik Zietlow, Anselm Paulus, Vít Musil, and Georg Martius. Deep graph matching via blackbox differentiation of combinatorial solvers. In European Conference on Computer Vision , pages 407-424. Springer, 2020.

Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 , 2014.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Runzhong Wang, Junchi Yan, and Xiaokang Yang. Learning combinatorial embedding networks for deep graph matching. In Proceedings of the IEEE/CVF international conference on computer vision , pages 3056-3065, 2019.

Runzhong Wang, Junchi Yan, and Xiaokang Yang. Neural graph matching network: Learning lawler's quadratic assignment problem with extension to hypergraph and multiple-graph matching. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2021.

Runzhong Wang, Junchi Yan, and Xiaokang Yang. Unsupervised learning of graph matching with mixture of modes via discrepancy minimization. IEEE Transactions of Pattern Analysis and Machine Intelligence , 2023.

Tao Wang and Haibin Ling. Gracker: A graph-based planar object tracker. IEEE transactions on pattern analysis and machine intelligence , 40(6):1494-1501, 2017.

Tao Wang, Haibin Ling, Congyan Lang, and Songhe Feng. Graph matching with adaptive and branching path following. IEEE Transactions on Pattern Analysis and Machine Intelligence , 40(12):2853-2867, 2018.

Tao Wang, He Liu, Yidong Li, Yi Jin, Xiaohui Hou, and Haibin Ling. Learning combinatorial solver for graph matching. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 7568-7577, 2020.

Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E Sarma, Michael M Bronstein, and Justin M Solomon. Dynamic graph cnn for learning on point clouds. Acm Transactions On Graphics (tog) , 38(5):1-12, 2019.

Paul Wohlhart and Vincent Lepetit. Learning descriptors for object recognition and 3d pose estimation. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3109-3118, 2015.

Jun Wu, Hong Shen, Yi-Dong Li, Zhi-Bo Xiao, Ming-Yu Lu, and Chun-Li Wang. Learning a hybrid similarity measure for image retrieval. Pattern Recognition , 46(11):2927-2939, 2013.

Yu Xiang, Roozbeh Mottaghi, and Silvio Savarese. Beyond pascal: A benchmark for 3d object detection in the wild. In IEEE winter conference on applications of computer vision , pages 75-82. IEEE, 2014.

Zhoubo Xu, Puqing Chen, Romain Raveaux, Xin Yang, and Huadong Liu. Deep graph matching meets mixed-integer linear programming: Relax at your own risk? arXiv preprint arXiv:2108.00394 , 2021.

Tianshu Yu, Runzhong Wang, Junchi Yan, and Baoxin Li. Learning deep graph matching with channel-independent embedding and hungarian attention. In International conference on learning representations , 2019.

Andrei Zanfir and Cristian Sminchisescu. Deep learning of graph matching. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 2684-2693, 2018.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors propose to improve the existing deep graph matching paradigm via positional reconstruction. The authors claim that existing deep GM works mainly focus on the visual feature to compute the affinities while neglecting the location and position of the key points in the graphs. To this end, they propose to capture the spatial features as well as the visual features in their model PREGM. The proposed encoder decoder PR-EnDec can learn effective graph positional encoding/coding with well-designed loss functions. The experiments on multiple datasets show the performance of the proposed method.\n\nSTRENGTHS:\n1. The consideration of utilizing spatial information is interesting to me since existing deep GM works mainly focus on how to improve the extraction of visual features. The positional encoding is further fused with the visual features, and updated by the well-defined loss functions.\n2. The experiments on multiple datasets show that the proposed method PREGM can outperform current SOTA methods.  \n3. The paper is well-written and easy to follow.\n\nWEAKNESSES:\n1. In fact, in the framework of existing works such as BBGM, the geometry information is already the input of SplineCNN. So, I think the authors may over-claim their novelty since existing works also consider spatial information to some extent.\n2. In the experiments on Willow and Spair71k datasets, not all baselines are reported in the table. I wonder is there a reason for that? I check the best baseline (COMMON) in the Pascal VOC dataset and find that they have reported their performance on Spair71k (84.5\\%), which is higher than the performance of the proposed method PREGM.\n\nQUESTIONS:\n1. I think the authors should further show the difference in spatial information between their work and BBGM since BBGM does consider spatial information. \n2. The experiments table may need to be completed. I think it is OK when your method does not outperform all the baselines in one of the datasets, but you need to show the results, right?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-28"}

comments: SUMMARY:
This paper introduces a positional reconstruction encoder-decoder (PR-EnDec) to model intrinsic graph spatial structure for image key-points matching. By using graph to represent the image structure, the proposed model can utilize the high-order spatial information by reconstructing  the locational structure of graphs contained in the node coordinates.

STRENGTHS:
1.The structure of the whole manuscript is good. The introduced positional encoder that learns effective graph positional encoding with affine transformation invariance.

2. The figures and tables are in good illustration.

WEAKNESSES:
1.For experimental comparison, the compared methods are old. Besides, the evaluation datasets, such as PascalVOC [8], Willow ObjectClass [4], and SPair-71k  are small and little old.  Some more recent large-scale datasets, such as PhotoTourism (IMC-PT) 2020 should be evaluated. 

2.Given the Problem Formulation of Graph Matching in sec.3, the compared two graphs have the same number of nodes. However, for the real application, the keypoints number constraint is not reasonable. 

3.Figure 2 shows the framework of the proposed PR-EnDec network that consists of the detailed structure of its positional encoder and spatial relation decoder.  However, in figure1, the Spatial Relation Decoder is missing. 

4.Compared to the previous works, the two stage training pipeline is complex.

QUESTIONS:
The proposed method takes the reference [28] as the baseline model (line 278), thus, this method should be compared on all three datasets.

LIMITATIONS:
See weakness and questions
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces an improved method for graph matching in semantic keypoint matching - the Positional Reconstruction Encoder-Decoder Network (PR-EnDec) and an end-to-end graph matching network PREGM. PR-EnDec efficiently learns node spatial embedding and reconstructs the locational structure of graphs from node coordinates. PREGM models the intrinsic spatial structure of keypoints and captures visual information, enhancing node positional features. Tests on three keypoint matching datasets showed improved performance over existing methods, with an ablation study demonstrating the effectiveness of each PREGM component.\n\nSTRENGTHS:\nThe paper introduces an innovative PR-EnDec model to effectively capture and utilize the spatial context information hidden in the locations of keypoints, which has not been adequately addressed in existing methods. The PR-EnDec incorporates a positional encoder and a spatial relation decoder, which not only capture the relative spatial relations but also learn the affine transformation invariance, enabling the network to learn more refined location information. Given the widespread use of image matching in various fields, such as object tracking, image retrieval, and pose estimation, the proposed improvements could have significant impacts on a variety of applications.\n\nWEAKNESSES:\n1. Lack of Visualizations\n\nWhile the paper presents an innovative approach and comprehensively assesses the performance of the proposed method, I would like to express a concern regarding the absence of sufficient visualizations. Visualizations can offer empirical evidence of model performance and further support the reported quantitative results. For instance, visual representations of keypoint matching results can provide intuitive insights into how the proposed method works in practice and highlight its ability to handle complex spatial relationships. In light of the above, the lack of sufficient visualizations in this paper might impede comprehensive understanding and thorough evaluation of the presented method. Therefore, I would suggest incorporating necessary visual aids into the paper to provide a more effective and comprehensive presentation of the methodology, the performance, and the practical implications of the proposed PREGM model.\n\n2. The Formatting of Table 4 and Table 5\n\nIn the current presentation, multiple experimental results appear to be consolidated into single rows within these tables. This presentation may potentially lead to confusion and misinterpretation of the results. Placing more than one set of results on a single line may obscure important details, making it harder for readers to draw meaningful conclusions from the data. As a suggestion to enhance clarity and readability, it would be beneficial to dedicate each row to one specific experimental result. This layout would allow for more detailed descriptions of the corresponding experimental settings and the related results, thus improving understanding. Moreover, it will facilitate the direct comparison of different experimental conditions and results, which is particularly critical in identifying trends, nuances, and potential implications. Therefore, I would recommend revising the formatting of Table 4 and Table 5 to present one set of experimental results per row, thereby ensuring that the wealth of information is conveyed as comprehensibly as possible.\n\n3. Lack of discussion about limitations and potential negative societal impact\n\nWhile the paper exhibits a commendable effort in proposing and testing a new methodology in graph matching, I would like to raise a concern regarding the lack of discussion on the potential limitations of the proposed method and its possible negative social implications. I suggest that the authors include a discussion of the possible limitations of the PREGM model and potential negative societal impacts in the paper, thereby presenting a more comprehensive and nuanced understanding of their work.\n\nQUESTIONS:\nPlease see the weaknesses part in the above section.\n\nLIMITATIONS:\nLimitations are not described in this paper, and potential negative societal impact is missing.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a new method to improve graph matching by supplementing visual features with positional encodings. Specifically, an encoder-decoder model is pre-trained to reconstruct a graph’s relative spatial relation based on node coordinates only. The encoder is additionally trained under a contrastive loss to classify positive or negative graph pairs. The pre-trained encoder provides the positional encodings, which is used to compute node and edge affinities with CNN visual features jointly. The proposed method is validated on three graph matching datasets: PascalVOC, WillowObject Class, and Spair-71k, with the best performance among all compared methods. Ablation study shows the contribution of different components in the proposed method.\n\nSTRENGTHS:\n1. The proposed positional reconstruction encoder-decoder network is a simple and effective method to extract positional features. Reconstructing the spatial relation between nodes in a graph is a good target for the encoder-decoder network.\n2. This paper proposes to use positional encodings to improve graph matching, which proves to be effective on three datasets. This is also in line with conclusions of some existing research, such as the vision transformer, in which positional encodings are shown to be very important. This work suggests all future work on graph matching can benefit by adding positional features.\n3. The ablation study has shown the importance of each component in the proposed method, including both encoder and decoder, the reconstruct targets, and visual features. Parameter analysis also provides the reason for the choice of hyper-parameters.\n4. This paper is well-written and easy to follow.\n\nWEAKNESSES:\nSome technical details of the proposed method are not clarified. What are the detailed configurations of the multi-head self-attention layers? How are positive and negative pairs for the encoder generated? Are the visual features sampled for each node of the graph? How are the graph edges defined? How are the edges used in the encoder? How are the node and edge affinities computed?\n\nQUESTIONS:\n1. For the visual features, why is VGG-16 used instead of a deeper model like ResNet-50?\n2. For the decoder reconstruction target, is the original coordinate, which is a natural choice, a good choice? If not, why?\n\nLIMITATIONS:\nThis paper has not discussed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

comments: SUMMARY:
The authors introduce a positional reconstruction encoder-decoder (PR-EnDec) to model intrinsic graph spatial structure, and present an end-to-end graph matching network PREGM based on PR12 EnDec. The PR-EnDec consists of a positional encoder that learns effective node spatial embedding with the affine transformation invariance, and a spatial relation decoder that further utilizes the high-order spatial information by reconstructing the locational structure of graphs contained in the node coordinates.

STRENGTHS:
I haven not personally conducted any research on graph matching, but the idea look interesting to me.
I suggest AC rely on other more experienced reviewer in this area.

WEAKNESSES:
My knowledge of graph matching is limited and therefore cannot find any weakness.

QUESTIONS:
What is the purpose of graph matching? Any use case related to NeRFs or generative models?

LIMITATIONS:
I have no idea about the limitations of graph matching.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-26