# Semantic Matching Based on Semantic Segmentation and Neighborhood Consensus
## 3 Approach
### Feature Extraction  
The pre-trained networks on ImageNet database can achieve image classification. Although these networks can extract the semantic features of the image, the extracted features are relatively rough since the classification is an image-level task and it only needs the semantics of the whole image without acquiring the semantics of each point. This conflicts with the semantic matching task that needs pixel-level semantics for dense matching. Fortunately, the semantic segmentation task meets this requirement, because it classifies each point instead of the entire image. Therefore, we propose the semantic-segmentation-based feature extractor that is pre-trained on the semantic segmentation dataset such as COCO dataset [41] and PASCAL VOC dataset [42], then adapts to the semantic matching task. Specifically, its construction process can be divided into two steps as shown in Figure 3: the first step is to construct a fully convolutional network and train it to achieve semantic segmentation; the second step intercepts part of the features and then employs learnable convolution layers to transform the hidden space of features to fit the semantic matching task. The detailed introduction is given below.  
First, the fully convolutional neural network encodes the image into a series of semantic feature maps, as shown in Figure 3a. The convolution operation explores the neighborhood structure. For example, it can recognize the low-level structure such as edges in the image. With the increase of convolution times, high-level structure, that is, semantic information can be recognized. In Figure 3a, the last output tensor of the network is the heat map. Different from the feature map, it comes from the feature map by convolution and up-sampling, but the number of channels is equal to the category number of semantic segmentation. For example, 21 candidate categories correspond to a heat map with 21 channels.  
Second, although the deeper feature maps of the fully convolutional neural network describe more semantics, the size of these feature maps is too small. To balance the size and semantics, we use the output of the 3 rd and 4 th layers of the network as feature maps, instead of the last layer. However, the features of the semantic segmentation task cannot be directly used for semantic matching, because the latter requires more neighborhood and location information. For example, the points on a car could have the same features in the semantic segmentation task, but in the semantic matching, it would cause severe matching ambiguity. As a result, contextual information should be introduced into the features to make them more distinguishable from each other. To this end, we propose to transform the hidden space of the features to fit the semantic matching task. Specifically, we use multiple convolution kernels to convolve the feature map \(\Psi_{r}\) of the pre-trained network to obtain a new feature map \(\Phi_{r}\) for the semantic matching task:  
\[\Phi_{r}=\text{Conv}(\Psi_{r}), \tag{4}\]  
Figure 3: The construction of the feature extractor. It is divided into two steps. The first step is (**a**): pre-training the semantic segmentation network, which inputs images, calculates a series of feature maps and heat maps, and finally outputs semantic segmentation results. The second step is (**b**): feature adaptation, where part of the semantic segmentation features (the orange block) are fine-tuned to be semantic matching features (\(\Phi_{3},\Phi_{4}\)) by convolution operation.  
where \(r\) is \(r\)th layer of the pre-trained network, and \(\text{Conv}(\cdot)\) is the convolution operation. Here we further concatenate \(\Phi_{3}\) and \(\Phi_{4}\) to enhance the feature representation ability.