# Semantic Matching Based on Semantic Segmentation and Neighborhood Consensus
## 3 Approach
### Objective Function  
Semantic matching lacks dense ground-truth correspondences, and manual annotation is quite difficult. To train the semantic matching network without dense ground truth, one approach is to use auxiliary labels. For example, the image can be rendered by a known  
Figure 4: Four-dimensional convolution. The feature maps \(F_{A}\) and \(F_{B}\) are extracted by a fully convolutional feature extractor. All pairs of individual matches \(p\) and \(q\) are represented in the 4D space of matches \((i,j,k,l)\), and the matching score is stored in the correlation \(C_{pq}\). The 4D convolution can be regarded as the weighted sum of correlations within the 4D neighborhood.  
3D model [21], the matching between images can be converted to the matching between 3D models, and the latter has known matching relationships. However, the final semantic matching accuracy of this method depends on the correctness of auxiliary labels and the accuracy of the conversion process. Another way is to construct training image pairs using a pre-defined geometric transformation (affine/homography) model [40]. As a result, one image can be transformed to another and their correspondences can be calculated since the transformation model are known. However, such synthesized images are still different from real images, that is, the difference between the synthesized image and the original real image is rigid, while there are lots of non-rigid differences between two real images. In contrast, our method directly uses specific labels provided by the semantic matching dataset as the strong supervision signal to train the network, which avoids the potential matching inaccuracy caused by rendering or transforming the image.  
Instead of using the foreground-mask correspondences as supervision signals [34], our method uses keypoint labels since they have pixel-level ground-truth matches. This stronger supervision signal can guide the network to estimate the matching field between images. Specifically, a landmark loss is defined, which is the average Euclidean distance between ground-truth keypoint \(p\) in the source image and the estimated one \(p^{\prime}\) by translating its corresponding target keypoint \(q\) to the source with the predicted correspondence:  
\[L_{ landmark}=\frac{1}{N}\sum_{i}^{N}\lVert p_{i}-p^{\prime}_{i}\rVert_{2}^{2}, \tag{6}\]  
where \(N\) is the number of keypoints. During training, the network can gradually estimate the semantic matching field, where all the estimated keypoints are as close as possible to the real keypoints in space.  
In addition, an unsupervised loss named consistency loss is used to assist network training, which works on all points in the image, but its constraint ability is not as strong as the landmark loss. Specifically, consistency loss is defined as the average Euclidean distance between initial point \(p\) in the source image and estimated point \(p^{\prime\prime}\) calculated by source-to-target and then target-to-source mappings:  
\[L_{consistency}=\frac{1}{N^{\prime}}\sum_{i}^{N^{\prime}}\lVert p_{i}-p^{ \prime\prime}_{i}\rVert_{2}^{2}, \tag{7}\]  
where \(N^{\prime}\) is the number of all pixels in the image. We define the overall objective function as  
\[L=\lambda_{1}L_{ landmark}+\lambda_{2}L_{consistency}, \tag{8}\]  
where \(\lambda_{1}\) and \(\lambda_{2}\) are the coefficients for landmark loss and consistency loss, respectively.