# Semantic Matching Based on Semantic Segmentation and Neighborhood Consensus
## 3 Approach
### Network Architecture  
Given a point in the image, our aim is to search for the semantically corresponding point in another image to consist of a matching pair, in which two points have similar representations. In semantic matching, the representation should reflect high-level semantics, while being insensitive to photometric and geometric variations. Here, we choose the high-dimensional feature \(F_{I}\) extracted by the neural network as the semantic representation, since it satisfies the above two requirements simultaneously:  
\[F_{I}=\text{Norm}(\Phi_{r}(I,\theta_{r})),I\in\{A,B\}. \tag{1}\]  
Here, the feature map is \(F_{I}\in\mathbb{R}^{c\times h\times w}\), where \((c,h,w)\) represent its channel, height, and width, respectively. \(\Phi_{r}\) is the activating output of the \(r\)th layer of feature extraction network with the learnable parameter \(\theta_{r}\). \(\{A,B\}\) represent the source image and the target image, and \(\text{Norm}(\cdot)\) is L2 normalization. There are many ways to evaluate the correlation between two features, such as the L1 or L2 norm of their difference. We use cosine similarity to compute the correlation, following previous works [31; 40], that is, calculating the dot product of two features:  
\[C_{p,q}=F_{T}^{A}(p)\cdot F_{B}(q), \tag{2}\]  
where \(C_{p,q}\) represents the correlation between points \(p\) and \(q\). Traversing all points on the source and target feature maps, we obtain the correlation map \(C\in\mathbb{R}^{h\times w\times h\times w}\) with four dimensions. When two features are similar, their correlation is closer to its upper limit 1. In other words, the more similar the features, the greater the correlation value.  
Figure 2 shows the network architecture of our method. Given an image pair \((A,B)\), the feature extractor extracts their semantic features. Since the deeper output of the feature extractor has more semantic information and a larger receptive field, we use the output of the last two activating layers to calculate the correlation map. These correlation maps are combined to fuse correlations by element-wise product. As a result, only two points with similar features in both activating layers will have a greater correlation, in other words, if the features in any layer are not similar, the final correlation value will be suppressed. To explore the consistency of the spatial context, we use 4D convolution to re-estimate the distribution of correlation, which then guides the semantic matching decision by the soft argmax function. Specifically, we compute the semantic mapping \(p\to q\) of point \(p\) in the feature map \(F_{A}\) by calculating an average position of all candidates in the feature map \(F_{B}\) with correlations as weights:  
\[p\to q=\sum_{q\in F_{B}}\text{softmax}(\beta\cdot C_{p,q})\cdot q, \tag{3}\]  
where \(\beta\) is the temperature parameter that controls the sharpness of the softmax function. All components are differentiable so that the network can be trained in an end-to-end manner.  
Figure 2: The pipeline of the proposed method. Two images \((A,B)\) are fed into the feature extractor, and then the features in different layers are used to calculate the correlation maps \((C_{1},C_{2})\), which will be combined and then go through 4D convolution, finally guiding the semantic matching assignment.