# Hierarchical Visual Place Recognition Based on Semantic-Aggregation
## 2 Hierarchical Visual Place Recognition Based on Semantic-Aggregation
### Coarse Matching  
We propose a simple yet efficient way of image representation, a hybrid global image descriptor, which can be obtained by aggregating semantic residuals for each semantic labels and semantic labels filtering. Then, we match the query with reference datasets by calculating cosine distance, to find the images with top-\(n\) similarity in the query. Those images contribute to the _Candidates_. The whole process of coarse matching is illustrated in Figure 2. We use an advanced cross-season semantic segmentation model [35] to obtain semantic labels and their probabilities, image features, and image segmentation. This model is based on the PSP-Net [27] and it greatly improves the robustness to seasonal changes by adding enforcing label consistency across matching. Firstly, mid-level convolutional feature map with the size of \(W\times H\times D\) is extracted from the pre-trained ResNet [36] model with the dilated network strategy [37; 38], where \(W\), \(H\), and \(D\) are the width, height, and depth of the feature map, respectively. In this task, \(W\) and \(H\) are 1/8 of the input image size, and \(D\) is 2048. Then, a pyramid pooling module is applied to gather context information and mine rich semantic information. In the pyramid module, 4-level pyramid are fused as the global prior and are concatenated with the original feature map to generate a final feature map with the size of \(W\times H\times 4096\), where \(W\) and \(H\) are 1/8 of the input image size too. Figure 1: Overview of our proposed method. (1) Coarse matching: we get the top \(n\)_Candidates_ with a global dataset search based on semantic aggregation and semantic filtering. (2) Fine matching: we select the best match through semantic edges and semantic association in the _Candidates_. Coarse matching helps to locate the query quickly, and fine matching helps to match the query accurately. Such a coarse-to-fine hierarchical progress improves the accuracy of place recognition and maintain computational efficiency. Defining the semantic label \(s_{i}\) at position \(i\) within the feature map is as follows:  
\[s_{i}=\operatorname*{argmax}_{c}p_{ic}\ c=\{0,1,\dots,C\} \tag{1}\]  
where \(c\) refers to the semantic classes corresponding to the related dataset, and \(C\) is the total number of semantic classes; \(p_{ic}\) represents the probability of the pixel at the location \(i\) belonging to a semantic class \(c\). Since each pixel's semantic class is determined, the mean descriptor \(m_{c}\) for each semantic class \(c\) can be computed as follow:  
\[m_{c}=\frac{\sum_{i}^{M}\{x_{i}|s_{i}=c\}}{\sum_{i}^{M}\{i|s_{i}=c\}} \tag{2}\]  
where \(x_{i}\) is the \(D\)-dimensional descriptor for the feature map, and \(M\) is the number of the pixels. Then, feature residuals of each semantic class can be computed by \(|x_{i}-m_{c}|\), which preserves the distribution differences between local features and semantic mean value. Then, we aggregate all the feature residuals of each semantic class for all the pixels in the image and weight with the corresponding semantic label probability to get \(H_{c}\):  
\[H_{c}=\sum_{i}^{M}p_{ic}|x_{i}-m_{c}| \tag{3}\]  
where \(H_{c}\) is essentially a hybrid image descriptor based on semantic aggregation for a semantic class \(c\). However, using \(H_{c}\) for the coarse matching directly will reduce computation efficiency. Moreover, some semantic classes will reduce robustness, such as person, car, since they are dynamic. Those semantic classes will increase non-overlap contents between the images, thus leading to a low accuracy. We keep \(L(L<C)\) main semantic classes to construct the final hybrid image descriptor \(H\). \(H\) is the add of L2-normalized \(H_{1},H_{2},\dots,H_{L}\). \[H=\left\langle\dot{H_{1}}+\dot{H_{2}}+\dots+\dot{H_{L}}\right\rangle \tag{4}\]  
Figure 2: The process of coarse matching. Given an image, we obtain its semantic segmentation, semantic labels and probabilities, and feature maps through a network model.