SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Next, we feed the fused features into a message-passing module to produce the refined node features, and the message-passing module is implemented by a two-layer SplineCNN [10]. After the message-passing module, node and edge affinities are computed and passed to the differentiable graph matching solver LPMP in [25]. The resulting correspondence matrix  $\mathrm{X}$  is compared to the ground truth  $\mathrm{X}^{gt}$  and the loss function is their Hamming distance:
$$
\mathcal {L} _ {G M} = \mathrm {X} \cdot (1 - \mathrm {X} ^ {g t}) + \mathrm {X} ^ {g t} \cdot (1 - \mathrm {X}). \tag {14}
$$
So far, the graph matching problem formulated previously is solved by our PREGM with the two sequential training phases: the PR-EnDec pretrain and the graph matching task.
We conduct experiments on four public keypoint matching datasets and verify the effectiveness of each component of our PREGM by ablation study. We introduce the datasets, baselines, implementation details, and then report the results. The results demonstrate that our PREGM consistently outperforms all other approaches.
We evaluate our PREGM on four public keypoint matching datasets: PascalVOC [8], Willow ObjectClass [4], SPair-71k [23], and IMC-PT-SparseGM [14].
The PascalVOC dataset includes 20 classes of keypoints with Berkeley annotations [2] and images with bounding boxes. The PascalVOC dataset is relatively challenging, since the scale, pose and illumination of instances in images are rich and diverse, and the number of annotated keypoints in each image also varies from 6 to 23. When conducting experiments on the PascalVOC dataset, we follow the standard protocol [32, 37]: First, each object is cropped according to its corresponding bounding box and scaled to  $256 \times 256$  px. Second, we use 7,020 images for training and 1,682 for testing.
The Willow ObjectClass dataset contains images of five categories: face, duck, winebottle, car, and motorbike, the first three categories are from the Caltech-256 dataset [12], and the last two categories are from the PascalVOC 2007 dataset [8]. Each category contains at least 40 different images, and each image is labeled with 10 distinctive keypoints on the target object to be matched. Following the default setting in [4, 32], we crop the images to the bounding boxes of the objects and rescale to  $256 \times 256$  px, 20 images of each class are selected during training, and the rest are for testing.
The SPair-71k dataset is a relatively novel dataset, which was recently published in the paper [23] about dense image matching. Spair-71k contains 70958 image pairs of 18 categories from PascalVOC 2012 dataset [8] and Pascal 3D+ dataset [41]. Compared with the other two datasets, SPair71-k has the advantages of higher image quality and richer annotations which includes detailed semantic keypoints, object bounding boxes, viewpoints, scales, truncation, and occlusion differences of image pairs. In addition, compared with PascalVOC dataset, SPair-71k removes two classes with ambiguous and poor annotations: sofa and dining table. Following [29], we use 53,340 image pairs for training, 5,384 for validation, and 12,234 for testing, and we also scale each image to  $256 \times 256$  px.
The IMC-PT-SparseGM dataset contains 16 object categories and 25061 images [14], which gather from 16 tourist attractions around the world. The IMC-PT-SparseGM benchmark involves matching
Table 1: Matching accuracy (%) on PascalVOC dataset.
Method	aero	bike	bird	boat	bottle	bus	car	cat	chair	cow	dtable	dog	horse	mbike	person	plant	sheep	sofa	train	tv	Avg
GNCCP	28.9	37.1	46.2	53.1	48.0	36.3	45.5	34.7	36.3	34.2	25.2	35.3	39.8	39.6	40.7	61.9	37.4	50.5	67.0	34.8	41.6
ABPF	30.9	40.4	47.3	54.5	50.8	35.1	46.7	36.3	40.9	38.9	16.3	34.8	39.8	39.6	39.3	63.2	37.9	50.2	70.5	41.3	42.7
GMN	31.9	47.2	51.9	40.8	68.7	72.2	53.6	52.8	34.6	48.6	72.3	47.7	54.8	51.0	38.6	75.1	49.5	45.0	83.0	86.3	55.3
PCA	40.9	55.0	65.8	47.9	76.9	77.9	63.5	67.4	33.7	65.5	63.6	61.3	68.9	62.8	44.9	77.5	67.4	57.5	86.7	90.9	63.8
CIE	51.2	69.2	70.1	55.0	82.8	72.8	69.0	74.2	39.6	68.8	71.8	70.0	71.8	66.8	44.8	85.2	69.9	65.4	85.2	92.4	68.9
NGM	50.8	64.5	59.5	57.6	79.4	76.9	74.4	69.9	41.5	62.3	68.5	62.2	62.4	64.7	47.8	78.7	66.0	63.3	81.4	89.6	66.1
DGMC	50.4	67.6	70.7	70.5	87.2	85.2	82.5	74.3	46.2	69.4	69.9	73.9	73.8	65.4	51.6	98.0	73.2	69.6	94.3	89.6	73.2
EAGM	49.4	62.1	64.6	75.3	90.9	80.9	71.1	61.3	48.7	65.9	87.5	58.4	66.3	60.1	56.3	97.1	64.7	60.6	96.0	93.0	70.5
BBGM	61.5	75.0	78.1	80.0	87.4	93.0	89.1	80.2	58.1	77.6	76.5	79.3	78.6	78.8	66.7	97.4	76.4	77.5	97.7	94.4	80.1
DIP-GM	58.5	74.9	76.2	76.0	87.1	94.7	89.8	79.8	60.4	77.5	79.8	78.0	76.2	78.2	64.3	97.1	76.4	76.4	96.5	93.2	79.6
IA-NGM-v2	61.5	73.8	74.0	79.4	89.1	94.6	89.7	77.5	67.1	77.3	92.4	76.8	77.1	77.4	65.8	98.5	77.5	79.5	96.6	92.3	80.9
ASAR	62.9	74.3	79.5	80.1	89.2	94.0	88.9	78.9	58.8	79.8	88.2	78.9	79.5	77.9	64.9	98.2	77.5	77.1	98.6	93.7	81.1
COMMON	65.6	75.2	80.8	79.5	89.3	92.3	90.1	81.8	61.6	80.7	95.0	82.0	81.6	79.5	66.6	98.9	78.9	80.9	99.3	93.8	82.7
Ours	62.6	76.4	79.3	78.7	90.3	94.1	92.3	83.4	71.9	82.3	92.4	82.4	82.0	85.0	69.2	98.7	78.4	90.7	96.7	95.1	84.1
Table 2: Matching accuracy (%) on Willow dataset.
Method	PT	WT	car	duck	face	mbike	wbottle	Avg
GNCCP	×	×	86.4	77.4	100.0	95.6	95.7	91.0
ABPF	×	×	88.4	80.1	100.0	96.2	96.7	92.3
GMN	×	×	74.3	82.8	99.3	71.4	76.7	80.9
PCA	×	×	84.0	93.5	100.0	76.7	96.9	90.2
CIE	×	×	82.2	81.2	100.0	90.0	97.6	90.2
NGM	×	✓	84.1	77.4	99.2	82.1	93.5	87.3
DGMC	×	✓	98.3	90.2	100.0	98.5	98.1	97.0
✓	✓	96.5	93.2	100.0	98.8	99.9	97.7
EAGM	×	✓	94.4	89.7	100.0	99.3	99.2	96.5
BBGM	×	✓	96.9	89.0	100.0	99.2	98.8	96.8
✓	✓	95.7	93.1	100.0	98.9	99.1	97.4
Ours	×	✓	97.4	96.8	100.0	99.8	96.8	98.2
✓	✓	97.8	99.0	100.0	99.6	100.0	99.3
larger scenes and take a step closer to the real-world downstream tasks. We take 13 classes as the training set and the other 3 classes as the test set. Experiments are conducted on the benchmark with 50 anchors.
5.2 Baselines and Metrics
We compare our PREGM with the state-of-the-art methods, including two types of baselines: (1) 2 non-learning methods GNCCP [20], ABPF [36]; (2) 11 learning-based methods GMN [44], PCA [32], CIE [43], NGM [33], DGMC [9], EAGM [27], BBGM [29], DIP-GM [42], IA-NGM-v2 [26], ASAR [28], COMMON [19]. For a fair comparison, we follow previous work [32] and extract node visual initial features from relu4_2 and relu5_1 of vgg16_bn [30] via feature alignment. For the input image pairs, we construct the edges of graphs by Delaunay triangulation [6]. We adopt a common metric matching accuracy to evaluate the experimental results, which is computed as the number of correctly matched keypoint pairs averaged by the total number of all true matched keypoint pairs.
5.3 Experimental Settings
In our implementation, we build up our PREGM based on the state-of-the-art method [29], which presents an end-to-end trainable architecture that incorporates a state-of-the-art differentiable combinatorial graph matching solver. Our codes will be available in the future. In all experiments, we use the same set of hyperparameters. We employ Adam [15] optimizer with an initial learning rate of  $1 \times 10^{-4}$  for PR-EnDec, and  $9 \times 10^{-4}$  for other models, and the learning rate is halved every three epochs. We empirically set  $l_{m} = 3$  and  $l_{n} = 2$  in the encoder and decoder, and choose temperature constant  $\tau = 2$  in  $L_{con}$ , and balance factor  $\lambda = 1/32$  in  $L_{PR-EnDec}$ . We also set batch size  $= 8$  for PascalVOC, Willow ObjectClass, and Spair-71k datasets. Our learning procedure is divided into two stages: The pre-training of the PR-EnDec is uniformly conducted on the PascalVOC dataset. Then the graph matching framework is trained on each dataset, while the parameters of the positional encoder are fixed. All experiments are run on a single GTX-1080Ti GPU.
Table 3: Matching accuracy (%) on Spair-71k dataset.
Method	aero	bike	bird	boat	bottle	bus	car	cat	chair	cow	dog	horse	mbike	person	plant	sheep	train	tv	Avg
GMN	50.1	40.4	62.7	46.8	60.8	66.8	57.2	62.0	40.5	61.5	49.5	46.5	56.5	43.9	76.1	44.7	61.2	75.7	55.7
PCA	58.9	42.3	72.1	54.1	61.2	77.3	66.1	65.2	50.4	64.9	56.8	55.5	64.3	53.4	86.2	49.1	75.5	91.4	63.6
DGMC	54.8	44.8	80.3	70.9	65.5	90.1	78.5	66.7	66.4	73.2	66.2	66.5	65.7	59.1	98.7	68.5	84.9	98.0	72.2
BBGM	66.9	57.7	85.8	78.5	66.9	95.4	86.1	74.6	68.3	78.9	73.0	67.5	79.3	73.0	99.1	74.8	95.0	98.6	78.9
DIP-GM	63.7	54.5	89.0	80.9	64.2	95.0	87.3	73.5	71.0	79.7	73.4	68.1	75.1	71.2	98.8	76.9	96.0	99.2	78.7
Ours	71.3	61.0	89.6	82.0	68.4	98.4	91.5	75.1	77.6	84.1	77.3	74.9	83.4	74.8	99.5	77.6	97.5	99.8	82.4
Table 4: Matching accuracy (%) on the IMC-PT-SparseGM.
Method	Reichstag	Sacre_coeur	St_peters_square	Avg
GANN-GM	76.0	44.2	50.5	56.9
BBGM	99.1	79.5	86.8	88.4
Ours	99.8	84.8	87.6	90.7
5.4 Performance Evaluation
We conduct experiments on four public datasets: PascalVOC, WillowObject Class, Spair-71k and IMC-PT-SparseGM for the keypoint matching problem, and follow the most common experimental setting, where intersection filtering is applied to generate graphs with the equal size.
Firstly, we report the matching accuracy of the 20 classes and average accuracy on PascalVOC dataset in Table 1, where the best results are shown in bold. The results demonstrate that our model PREGM performs much better than all traditional graph matching methods, and also achieves better performance against state-of-the-art deep graph matching models with matching accuracy  $84.1\%$ . And in the 20 classes of PasalVOC dataset, our PREGM has achieved the best results in the other 17 classes except for the boat, bottle and train class. As there are large differences in the scale, pose and illumination of matching objects, PascalVOC is a complex matching dataset, thus the matching accuracy of traditional graph matching methods is not more than  $50\%$ , and the performance of deep graph matching methods is relatively better. Our PREGM learns effective node spatial characteristics by PR-EnDec, so as to further improve the graph matching performance.
For the relatively simple WillowObject Class dataset, as shown in Table 2, there are two different training strategies: PT and WT, which PT means matching frameworks are pre-trained on PascalVOC, and WT means learning models are then fine-tuned on WillowObject Class Dataset. In this paper, we adopt two strategies: PT only, both PT and WT, and in both situations, our model achieves the best performance among other state-of-the-art methods with matching accuracy  $98.2\%$  and  $99.3\%$ . The results again demonstrate the effectiveness of learning positional node features hidden in graphs in our PREGM.
We also conduct experiments on Spair-71k dataset, as shown in Table 3, we compare our PREGM with deep learning methods GMN, PCA, DGMC, BBGM, and DIP-GM. Our PREGM performs best with matching accuracy  $81.9\%$ , and shows superiority in total 18 classes, which demonstrates the generalization ability of our model on different objects. SPair-71k dataset, as a relatively new dataset, has the advantages of high image quality, rich annotation, and fixed image pairs for matching, which is a more convincing dataset. Thus our PREGM achieves the best results on the SPair-71k dataset, which further proves the effectiveness of our PREGM.
Additionally, we evaluate our model on the IMC-PT-SparseGM dataset, as shown in Table 4, our model demonstrates outstanding performance on this demanding benchmark. The results outshine the performance of the other methods, GANN-GM [34] and BBGM, by a significant margin. In terms of the average accuracy across these landmarks, our model excels with an impressive mean accuracy of  $90.7\%$ . The IMC-PT-SparseGM dataset is characterized by its substantial number of images, nodes, and high partial rate, making it one of the most comprehensive benchmarks for visual graph matching. Our approach, showcased in these results, demonstrates its ability to handle larger scenes and move closer to real-world applications, such as structure from motion.
Table 5: Ablation Study on PascalVOC dataset.
Method	baseline	w/o positional encoder	w/o spatial relation decoder	w/o recon_area	w/o recon_dis	w/o visual features
Accuracy	84.1%	83.4%	81.7%	83.8%	82.7%	76.8%
Table 6: Parameter analysis of learning rate on PascalVOC.
Ir (×10-4)	6	7	8	9	10	11	12
Accuracy	83.2%	83.7%	83.8%	84.1%	83.9%	83.7%	83.7%
5.5 Ablation Study and Parameter Analysis
To evaluate the effect of each component in our framework, we conduct comprehensive ablation studies with/without positional encoder, spatial relation decoder, and we also consider the two separate cases of no reconstruction of areas and relative distances in the loss function of spatial relation decoder. Furthermore, we conducted additional ablation experiments by removing visual features to evaluate the effectiveness of the spatial features extracted by our model. The experiments are performed on PascalVOC dataset. As shown in Table 5, compared with the baseline, the results demonstrate that all modules bring substantial performance gains, the reconstruction of relative distance is more important in the decoder, and the spatial relation decoder contributes most to the overall performance. Moreover, with the fact that only taking spatial features as node features can achieve relatively good performance, it further proves the effectiveness of the spatial features learned from our PR-EnDec.
We also conduct parameter analysis to select hyperparameters. As shown in Table 6, PREGM achieves the best performance when learning rate  $= 9 \times 10^{-4}$ , which is our default setting, and it also shows that adjusting the learning rate causes an accuracy fluctuation of about  $1\%$ . For the balance factor  $\lambda$  in the loss function, when  $\lambda = 1/128, 1/32$ , and  $1/8$ , the matching accuracy is  $83.7\%$ ,  $84.1\%$ , and  $83.3\%$  respectively. Thus we select  $\lambda = 1/32$  as our default setting, and the results show that our method is rather sensitive to the choice of  $\lambda$ , and our designed loss function in positional encoder and spatial relation decoder indeed improves the performance.
In this paper, we present an end-to-end novel deep graph matching network PREGM that learns more distinguishable node features by modeling spatial positional information in graphs. Our PREGM designs a common encoder-decoder architecture consisting of a positional encoder that learns effective node positional encoding and a spatial relation decoder that reconstructs the positional structure of graphs. Our experiments and ablation studies on four public keypoint matching datasets demonstrate the state-of-the-art performance of our method. The exploration direction of future work includes optimizing the loss function of the positional encoder to extract purer spatial structural information and improve the feature fusion method to obtain better fused feature representation.
We acknowledge support from National Key R&D Program of China (2021ZD01133001).
[1] Ryan Prescott Adams and Richard S Zemel. Ranking via sinkhorn propagation. arXiv preprint arXiv:1106.1925, 2011.
[2] Lubomir Bourdev and Jitendra Malik. Poselets: Body part detectors trained using 3d human pose annotations. In 2009 IEEE 12th International Conference on Computer Vision, pages 1365-1372. IEEE, 2009.
[3] Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. Realtime multi-person 2d pose estimation using part affinity fields. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7291-7299, 2017.
[4] Minsu Cho, Karteek Alahari, and Jean Ponce. Learning graphs to match. In Proceedings of the IEEE International Conference on Computer Vision, pages 25-32, 2013.
[5] Minsu Cho, Jungmin Lee, and Kyoung Mu Lee. Reweighted random walks for graph matching. In European conference on Computer vision, pages 492-505. Springer, 2010.
[42] Zhoubo Xu, Puqing Chen, Romain Raveaux, Xin Yang, and Huadong Liu. Deep graph matching meets mixed-integer linear programming: Relax at your own risk? arXiv preprint arXiv:2108.00394, 2021.
[43] Tianshu Yu, Runzhong Wang, Junchi Yan, and Baoxin Li. Learning deep graph matching with channel-independent embedding and hungarian attention. In International conference on learning representations, 2019.
[44] Andrei Zanfir and Cristian Sminchisescu. Deep learning of graph matching. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2684-2693, 2018.