SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yixiao Zhou
Wangxuan Institute of Computer Technology Peking University
Ruiqi Jia
Hongxiang Lin
Hefeng Quan
School of computer science and engineering Nanjing University of Technology
Yumeng Zhao
Xiaoqing Lyu*
Wangxuan Institute of Computer Technology, Beijing Institute of Big Data Research Peking University
Deriving from image matching and understanding, semantic keypoint matching aims at establishing correspondence between keypoint sets in images. As graphs are powerful tools to represent points and their complex relationships, graph matching provides an effective way to find desired semantic keypoint correspondences. Recent deep graph matching methods have shown excellent performance, but there is still a lack of exploration and utilization of spatial information of keypoints as nodes in graphs. More specifically, existing methods are insufficient to capture the relative spatial relations through current graph construction approaches from the locations of semantic keypoints. To address these issues, we introduce a positional reconstruction encoder-decoder (PR-EnDec) to model intrinsic graph spatial structure, and present an end-to-end graph matching network PREGM based on PR-EnDec. Our PR-EnDec consists of a positional encoder that learns effective node spatial embedding with the affine transformation invariance, and a spatial relation decoder that further utilizes the high-order spatial information by reconstructing the locational structure of graphs contained in the node coordinates. Extensive experimental results on four public keypoint matching datasets demonstrate the effectiveness of our proposed PREGM.
Image matching is a common and basic matching problem in multimedia and visual applications. The goal of image matching is to find a dense semantic content correspondence between two images, while feature-based matching is a typical strategy to solve the above problems, and in feature matching,
*Correspondence: lvxiaqing@pku.edu.cn. Author emails: {chnzyx, jiaruiqi}@pku.edu.cn, linhongxiang@stu.pku.edu.cn, 920106840131@njust.edu.cn, zhaoyumeng_311@bupt.edu.cn, lvxiaqing@pku.edu.cn
the semantic keypoints in images are the most commonly used matching targets. Semantic keypoint matching aims at establishing correspondence between keypoint sets in images, and it has been applied to a wide range of applications, such as object tracking [35, 24], image retrieval [40, 11], and pose estimation [39, 3], etc.
The semantic keypoint data has the properties of non-Euclidean and unstructured for its disorder and discreteness, and graphs are powerful tools to represent some complex objects and their interactions. Specifically, we take each semantic keypoint as a node, and build edges by heuristic methods according to the positions of nodes to construct a corresponding graph. Graph matching is an effective method to solve the correspondence between the semantic keypoints in two images.
As deep learning models are scalable and time-efficient, Graph Neural Networks (GNNs) are increasingly utilized in graph matching tasks, such as GMN [44], PCA [32], and BBGM [29], etc.
However, most graph matching approaches rely on the matching of basic items of graphs, i.e. computing affinities of nodes and edges, lack the mining and utilization of spatial context information hidden in the locations of keypoints as nodes of graphs. Recently, in task scenarios such as point cloud and knowledge graph, which the nodes in graphs have locational information, there are methods such as [21, 18] utilizing positional encoder to add the spatial information as additional features. However, current methods in graph matching domain lack such spatial positional information supplementation. As the positional information is closer to the intrinsic characteristic of the graph, it is important to integrate spatial descriptions into graph feature embeddings in graph matching.
But utilizing underlying spatial information faces the following challenges: 1) Through taking key points as graph nodes and edges are associated with pairwise distance, such as delaunay triangulation, it is not sufficient to mine hidden relative positional relationship. 2) The description of spatial information from coordinates of keypoints is relatively simple compared with conventional visual features, so it needs more refined network structure to learn location information, in order to facilitate subsequent feature fusion.
To address the two challenges mentioned above, we propose a positional reconstruction encoder-decoder network (PR-EnDec) in our PREGM by integrating more spatial information into semantic features and learning more distinguishable node features. The positional encoder in PR-EnDec aims to learn the encoding of node positional information with spatial invariance, specifically, the affine transformation invariance of node coordinates. The spatial relation decoder in PR-EnDec is designed to recover the locational structure from the learned features of the encoder, and it reconstructs the relative distances and areas among the sampled nodes. To implement the joint learning of PR-EnDec about the intrinsic positional features in graphs, we, consequently, propose a corresponding contrastive loss function and a reconstruction loss function in the encoder and the decoder respectively.
- We propose an end-to-end graph matching network, PREGM, which learns more distinguishable node features, not only captures visual information in images but also models intrinsic spatial structure of keypoints by our designed PR-EnDec.
- Our PR-EnDec consists of a positional encoder that learns effective graph positional encoding with affine transformation invariance, and a spatial relation decoder used to enhance such learning by reconstructing the graph locational information such as relative distances and areas from learned positional encoding. We also design a corresponding contrastive loss function and a reconstruction loss function in the encoder and decoder respectively to help learn effective node positional characteristics.
- We evaluate our PREGM on four public keypoint matching datasets. Our experimental results demonstrate that PREGM outperforms state-of-the-art methods. We also present an ablation study, which shows the effectiveness of each component of PREGM.
2.1 Graph Matching
Graph matching is always an effective strategy to find correspondence between semantic keypoints as graphs have sufficient representation ability for points and their complex relationships, and it can then be applied to downstream tasks such as image matching and understanding. As it is famous for being one of the practically most difficult NP-complete problems, researchers often use optimization algorithms to find an acceptable suboptimal solution for the graph matching problem. Traditional graph matching methods [17, 5, 7, 20, 36] generally use heuristic search algorithms to find suboptimal solutions.
Recent graph matching methods begin to combine combinatorial optimization with deep learning models. GMN [44] proposes an end-to-end graph matching framework, calculates the unary and pairwise affinities of nodes through CNN features, and also uses spectral matching as a differentiable solver to combine deep learning with graph matching. PCA [32] uses graph neural network (GNN) to learn and aggregate graph structure information into node feature representation, and introduces Sinkhorn network [1] as the combinatorial solver for finding node correspondence. BBGM [29] presents an end-to-end trainable architecture that incorporates a state-of-the-art differentiable combinatorial graph matching solver, and introduces a global attention mechanism to improve the matching performance.
Although the research on graph matching has made the above progress, the existing methods lack the distinguishing and informative graph spatial feature descriptions, and adaptive adjustment of node correspondence on graph matching. In this paper, we overcome these limitations by proposing a positional reconstruction encoder-decoder (PR-EnDec) and designing a contrastive learning and positional feature reconstruction procedure to guide the learning of graph matching.
2.2 Positional Encoding on Graphs
Positional Encoding was first proposed in the Transformer [31] model. In general, positional encoding refers to the process of encoding spatial relationships or topological information of elements as supplementary information in order to augment the representational capacity of a model. The categorization of positional encoding can be further elaborated as follows: spatial positional encoding utilizes point location as input, while topological positional encoding employs the graph structure as input.
For spatial positional encoding, the point location is used as input to obtain a high-dimensional embedding that contains effective locational information. Space2Vec [21] introduces theory and grid as a 2D multi-scale spatial positional encoder by using sinusoidal functions with different frequencies. PointCNN [18] designs a point set spatial positional encoder with Point Conv layers consisting of sampling layer, grouping layer, and aggregation layer. DGCNN [38] considers both global shape structure and local neighborhood information, and proposes spatial positional encoder layers containing dynamic graph neighborhood and edge convolution module. This paper proposes a spatial position encoding on graphs by leveraging the characteristic of node representation containing coordinate information in image keypoint matching.
Topological positional encoding is specific to graph matching, where the graph structure is used as input to obtain the topological embedding of the nodes. Maskey et al. [22] generalizes Laplacian-based positional encoding by defining the Laplace embedding to more general dissimilarity functions such as p-norm rather than the 2-norm used in the original formulation. INFMCS [16] designs a permutation-invariant node ordering based on closeness centrality, and effectively enhances the representation of graph structural features.
In recent years, there has also been research on spatial positional encoding in the field of graph matching. GCAN [13] utilizes SplineCNN [10] to encode positional information in 2D spatial space, and successfully capture structural information of the graph.
However, there remains the open question of how to further mine intrinsic spatial node information in graphs. In this paper, we propose a positional reconstruction encoder-decoder architecture to obtain effective node spatial positional encoding using 2D point location.
Figure 1: The framework of our PREGM. The visual features are extracted by vgg16_bn from images, and the positional encoding is obtained by our pretrained positional encoder from graphs. The visual features and positional encoding of nodes are further fused to be fed into the following modules. Next, the fused features are updated and refined by the message passing module SplineCNN. Finally, we compute node affinity from updated features and adopt a differentiable graph matching solver LPMP to find the correspondence matrix X.
3 Problem Formulation of Graph Matching
Given the keypoints in the images, each node is generally associated with a keypoint, and edges are built according to the spatial position relationship between nodes when building an undirected graph. The graph is represented by  $\mathbb{G} = (\mathbb{V},\mathbb{E},\mathcal{V},\mathcal{E})$ :
-  $\mathbb{V} = \{v_1, \dots, v_n\}$  denotes the node set.
-  $\mathbb{E} \subseteq \mathbb{V} \times \mathbb{V}$  denotes the edge.
-  $\mathcal{V} = \{\mathbf{v_i}|\mathbf{v_i}\in \mathbb{R}^{d_v},i = 1,2,\dots,|\mathbb{V}|\}$  denotes the node feature set.
-  $\mathcal{E} = \{\mathbf{e_i}|\mathbf{e_i}\in \mathbb{R}^{d_e},i = 1,2,\dots,|\mathbb{E}|\}$  denotes the edge feature set.
We use the adjacency matrix  $\mathbb{A}$  to represent the connections of nodes in an undirected graph  $\mathbb{G}$ , that is,  $\mathbb{A}_{ij} = 1$  iff there is an edge  $e_{ij} = (v_i,v_j)\in \mathbb{E}$ .
Given two graphs  $\mathbb{G}_1 = (\mathbb{V}_1, \mathbb{E}_1, \mathcal{V}_1, \mathcal{E}_1)$  and  $\mathbb{G}_2 = (\mathbb{V}_2, \mathbb{E}_2, \mathcal{V}_2, \mathcal{E}_2)$ , where  $|\mathbb{V}_1| = |\mathbb{V}_2| = n$ , the goal of graph matching is to find the correspondence matrix  $\mathbf{X} \in \{0, 1\}^{n \times n}$  between nodes  $\mathbb{V}_1$  and  $\mathbb{V}_2$  of the two graphs  $\mathbb{G}_1$  and  $\mathbb{G}_2$ , wherein  $\mathbf{X}_{ij} = 1$  iff  $v_i \in \mathbb{V}_1$  corresponds to  $v_j \in \mathbb{V}_2$ .
The ultimate goal of the graph matching problem is to find the optimal node correspondence between two graphs  $\mathbb{G}_1$  and  $\mathbb{G}_1$ , so the graph matching problem is often considered as a quadratic assignment programming problem (QAP problem):
$$
\mathbf {x} ^ {*} = \operatorname {a r g m a x} _ {\mathbf {x}} \mathbf {x} ^ {T} \mathbf {K} \mathbf {x}, \tag {1}
$$
where  $\mathbf{x} = \text{vec}(\mathbf{X}) \in \{0,1\}^{n^2}$ ,  $\mathbf{X}\mathbf{1}_n = \mathbf{1}_n$ , and  $\mathbf{X}^T\mathbf{1}_n = \mathbf{1}_n$ .  $\mathbf{x}^*$  denotes the desired node correspondence, and  $\mathbf{1}_n$  denotes a vector of  $n$  ones.  $\mathbf{K} \in \mathbb{R}^{n^2 \times n^2}$  is the corresponding affinity matrix:
$$
\mathbf {K} _ {i a, j b} = \left\{ \begin{array}{l l} s _ {i a} ^ {v}, & \text {i f} i = j \& a = b, \\ s _ {i a, j b} ^ {e}, & \text {e l s e i f} \mathbb {A} _ {i j} ^ {1} \mathbb {A} _ {a b} ^ {2} > 0, \\ 0, & \text {o t h e r w i s e .} \end{array} \right. \tag {2}
$$
where  $s_{ia}^{v}$  represents the affinity between node features  $\mathbf{v}_i \in \mathcal{V}_1$  and  $\mathbf{v}_a \in \mathcal{V}_2$ ,  $s_{ia,jb}^{e}$  represents the affinity between edge features  $\mathbf{e}_{ij} \in \mathcal{E}_1$  and  $\mathbf{e}_{ab} \in \mathcal{E}_2$ .
The framework of our PREGM Network is illustrated in Figure 1. The positional encoder module of pretrained PR-EnDec takes the coordinates of keypoints as input. It extracts a positional encoding vector for each keypoint in the images. Next, the positional encoding and the visual features extracted by vgg16_bn are further fused. Then, the fused features are fed into the message-passing module to obtain the refined node feature. Finally, the node affinity is computed and processed by the LPMP solver, generating the node correspondence matrix X introduced in the problem formulation.
Figure 2: The framework of our PR-EnDec network and the detailed structure of its positional encoder and spatial relation decoder.
As an independent stage, we pretrain the abovementioned positional encoder along with the spatial relation decoder module of PR-EnDec (as shown in Figure 2). The pretraining stage enables the positional encoder to learn the high-order spatial information. The PR-EnDec consisting two modules,
Positional Encoder. This module takes the coordinates of keypoints as input and learns positional encoding. We first obtain the high-dimensional coordinate embedding vectors by an MLP. The vectors are then fed into self-attention blocks and a normalization layer to learn positional encoding. The attention mechanism provides sequence independence and relative information of nodes. Besides, we propose a contrastive loss for the encoder module. The loss function ensures the affine transformation invariance and learns the relative position information.
Spatial Relation Decoder. To reconstruct the spatial structure of graphs, this module generates triangle assignments by distance-based random sampling. In each triangle assignment, the module takes positional encoding corresponding to the three keypoints as the input of self-attention blocks. Next, the processed features are fed into MLPs, and the relative spatial relations of the three points are reconstructed. We adopt a Mean Squared Error (MSE) loss to compare the ground truth relative spatial relations and the decoder's predictions.
4.1 Positional Encoder
The Positional Encoder only takes geometric features of keypoints as inputs,
$\mathcal{V}^g = \{v_i^g | v_i^g \in \mathbb{R}^{d_v^g}, i = 1, \dots, n\}$  denotes the node geometric feature set.
Specifically, we utilize the 2D Cartesian coordinates of each node  $v_{i}$  as its geometric feature  $v_{i}^{g} = [x_{i},y_{i}]$ . The coordinate embedding sub-module embeds the geometric feature  $\mathcal{V}^g$  into latent representations by a two-layer MLP, denoted as  $\rho^v$ . The updated geometric graph  $\mathbb{G}^g = (\mathbb{V},\rho^v (\mathcal{V}^g))$  is then fed into the following attention sub-module.
The attention sub-module consists of  $l_{m}$  attention blocks (denoted as  $Attn_{i}^{e}$ ) and a normalization layer (denoted as Norm), which updates latent representations of  $\mathbb{G}^g$ . An attention block is composed of a multi-head self-attention layer and a feed-forward layer, which captures the spatial feature of  $\mathbb{G}^g$  by aggregating the latent representations of each node.
$$
\mathcal {V} _ {0} ^ {g} = \rho^ {v} (\mathcal {V} ^ {g}), \tag {3}
$$
$$
\mathcal {V} _ {i} ^ {g} = \operatorname {A t t n} _ {i} ^ {e} \left(\mathcal {V} _ {i - 1} ^ {g}\right), i = 1, 2, \dots , l _ {m}. \tag {4}
$$
The self-attention mechanism is sequence-independent, which is well suited for extracting high-order relative position information between nodes. There is a normalization layer after attention blocks to allow the positional encoding of nodes to be better fused with visual features nodes on the graph matching task. The final positional encoding obtained by the encoder module is denoted as  $F_{pos}$ .
$$
F _ {p o s} = \operatorname {N o r m} \left(\mathcal {V} _ {l _ {m}} ^ {g}\right). \tag {5}
$$
Encoder Loss. For two given graphs  $\mathbb{G}_1 = (\mathbb{V}_1, \mathbb{E}_1, \mathcal{V}_1, \mathcal{E}_1)$  and  $\mathbb{G}_1 = (\mathbb{V}_1, \mathbb{E}_1, \mathcal{V}_1, \mathcal{E}_1)$ , the positional encoder generates the positional encoding denoted as  $F_{pos}^{1} = \{f_{i}^{1} | i = 1, 2, \ldots, |\mathbb{V}|\}$  and  $F_{pos}^{2} = \{f_{i}^{2} | i = 1, 2, \ldots, |\mathbb{V}|\}$  respectively. We expect that for all matched node  $i$  in graph  $\mathbb{G}_1$  and node  $j$  in graph  $\mathbb{G}_2$ , the corresponding node positional encoding  $f_{i}^{1}$  and  $f_{j}^{2}$  should be similar. To let the encoder learn the relative position information of the nodes, we perform contrastive learning that
perseves the affine invariance of the positional encoding. Specifically, we adopt negative samples to conduct contrastive learning to avoid the encoder module outputs trivial node features that are all similar.
Therefore, we classify  $(\mathbb{G}_1,\mathbb{G}_2)$  as a positive graph pair if  $\nu_{2}^{g}$  is affine transformed by  $\nu_{1}^{g}$  or the keypoints in  $\nu_{1}^{g}$  and  $\nu_{2}^{g}$  are one-by-one matched. On the contrary,  $(\mathbb{G}_1,\mathbb{G}_2)$  is a negative graph pair if the keypoints are not matched.
For each training batch, the positive graph pair set  $\mathbb{G}\mathbb{S}_p$  and negative graph pair set  $\mathbb{G}\mathbb{S}_n$  are generated by node permuting or affine transformation. Let  $\mathbb{F}\mathbb{S}_p$  and  $\mathbb{F}\mathbb{S}_n$  denote the corresponding positional encoding pair set of  $\mathbb{G}\mathbb{S}_p$  and  $\mathbb{G}\mathbb{S}_n$  respectively, we propose a contrastive loss  $\mathcal{L}_{con}$ ,
$$
S _ {p} = \sum_ {\left(F _ {p o s} ^ {1}, F _ {p o s} ^ {2}\right) \in \mathbb {F S} _ {p}} \exp \left(\tau * \operatorname {s i m} \left(F _ {p o s} ^ {1}, F _ {p o s} ^ {2}\right)\right), \tag {6}
$$
$$
S _ {n} = \sum_ {\left(F _ {p o s} ^ {1}, F _ {p o s} ^ {2}\right) \in \mathbb {F S} _ {n}} \exp \left(\tau * \operatorname {s i m} \left(F _ {p o s} ^ {1}, F _ {p o s} ^ {2}\right)\right), \tag {7}
$$
$$
\mathcal {L} _ {\text {c o n}} = - \operatorname {L o g} \left(\frac {S _ {p}}{S _ {p} + S _ {n}}\right), \tag {8}
$$
where  $S_{p}$  and  $S_{n}$  denote the sum of the exponential of positive/negative pairs' similarity, respectively. Additionally,  $\tau$  denotes the temperature constant, and  $sim()$  denotes the cosine similarity.
4.2 Spatial Relation Decoder
To learn the high-order spatial information, we propose  $k$ -th-order geometric reconstruction assignments. Sampling  $k$  points from the node set as an assignment, the corresponding geometric feature set is denoted as  $\mathcal{V}_s^g$  and positional encoding set as  $F_{sub} \subseteq F_{pos}$ . If a decoder  $Dec$  takes  $F_{sub}$  as input, we claim the decoder provides  $k$ -th-order relative position information if there exists a non-degenerate geometric function  $\mathcal{F}$ ,  $\mathcal{F}(\mathcal{V}_s^g) \approx Dec(F_{sub})$ . Specifically, we choose  $k = 3$  and function  $\mathcal{F}$  with Euclidean distance and triangle area to learn the third-order relative position information.
The decoder module first samples three keypoints and obtains the corresponding geometric feature set  $\mathcal{V}_s^g = \{v_a^g, v_b^g, v_c^g\}$  and positional encoding set  $F_{sub} = \{f_a, f_b, f_c\}$ . The random sampling method is related to the Euclidean distance from the previous sampled node to learn the relative information better, for the sampling method guarantees that the distances between the sampled nodes are not too far. Next, the sampled feature set  $F_{sub}$  is fed into  $l_n$  attention blocks. The intermediate feature is denoted as  $\hat{F}_{sub} = \{\hat{f}_a, \hat{f}_b, \hat{f}_c\}$ . We utilize 2 MLPs, denoted as  $\rho_d$  and  $\rho_a$ , to approximate Euclidean distance and triangle area function, respectively. The reconstructed relative position information of the decoder is represented as:
$$
R C = \left[ \rho_ {d} \left(\hat {f} _ {a}, \hat {f} _ {b}\right), \rho_ {d} \left(\hat {f} _ {b}, \hat {f} _ {c}\right), \rho_ {d} \left(\hat {f} _ {a}, \hat {f} _ {c}\right), \rho_ {a} \left(\hat {f} _ {a}, \hat {f} _ {b}, \hat {f} _ {c}\right) \right]. \tag {9}
$$
Let  $Dis$  denotes Euclidean distance and  $Area$  denotes triangle area function,  $RC$ 's corresponding geometric function  $\mathcal{F}$  is obviously
$$
\mathcal {F} \left(\mathcal {V} _ {s} ^ {g}\right) = [ d i s 1, d i s 2, d i s 3, a r e a ], \tag {10}
$$
where  $dis1 = Dis(v_a^g, v_b^g)$ ,  $dis2 = Dis(v_b^g, v_c^g)$ ,  $dis3 = Dis(v_a^g, v_c^g)$ ,  $area = Area(v_a^g, v_b^g, v_c^g)$  denote ground-truth third-order relative position information for the decoder to reconstruct.
Decoder Loss. Since we calculate the approximated relative position information  $RC$  and the ground-truth  $RC^{gt} = \mathcal{F}(\mathcal{V}_s^g)$ , we propose the reconstruction loss  $\mathcal{L}_{rec}$ :
$$
\mathcal {L} _ {r e c} = \operatorname {M S E} \left(R C, R C ^ {g t}\right). \tag {11}
$$
PR-EnDec Loss. Finally, we combine the two losses to guide the training of our PR-EnDec jointly,
$$
\mathcal {L} _ {P R - E n D e c} = \mathcal {L} _ {c o n} + \lambda \cdot \mathcal {L} _ {r e c}, \tag {12}
$$
where  $\lambda$  controls the relative importance of  $\mathcal{L}_{rec}$ .
Through pretraining of PR-EnDec, we obtain an effective encoder for extracting spatial positional encoding for utilization in subsequent PREGM.
4.3 Graph Matching with PR-EnDec
Our training procedure is divided into two stages: In the first stage, we pretrain our PR-EnDec, and in the second stage of graph matching, our positional encoder module of PR-EnDec serves as a sub-module of the base graph matching model, providing learned positional encoding in the first stage.
After the training of PR-EnDec, we freeze all parameters of the encoder module and put it in the training of graph matching tasks. The positional encoding  $F_{pos}$  generated by the positional encoder module is fused with visual features  $F_{vis}$  extracted by a standard CNN:
$$
F _ {f u s e d} = \operatorname {L i n e a r} \left(F _ {p o s}\right) + F _ {v i s}. \tag {13}
$$
where Linear denotes a linear layer.