# SSNet: Learning Mid-Level Image Representation Using Salient Superpixel Network
## 3 The Proposed Method for Image Representation
### The Mid-Level Image Representation Based on SSNet
Thus, in our experiments, each extracted CNN local feature is \(9\times 32\) dimension. At the same time, due to the outstanding performance of the Fisher Vector encoding methods, we only adopt it to evaluate our method in the following experiments. With these CNN local features and the Fisher Vector encoding method, image representation based on the SSNet is built. Here, all the preprocess and settings are same as in Section 4.4. Particularly, the parameter height and width of Equation (24) are assigned as the height and width of the feature map for the last convolutional layer, respectively. The regularization parameters in Equation (25) are set to \(\lambda_{1}=10^{2}\) and \(\lambda_{2}=10^{2}\) by the cross-validation method. We compare our approach with two baselines and state-of-the-art methods [8; 32; 41; 59] respectively, and the results are demonstrated in Table 3. Here, the baseline method FC-CNN [36] uses the feature extracted from the last fully connected layer as image representation. The method CL-FV uses the activations extracted from the last convolutional layer as the local features and encodes them by the Fisher Vector method to build image representation. As we can see, the recognition results in Table 3 show that our method outperforms the baseline and state-of-the-art methods. The proposed method improves the performance of the baselines above 4% in all the datasets. Additionally, our method performs better than the mentioned state-of-the-art methods [8; 32; 41; 59]. Taking the results on the VOC 2007 as an example, our method improves the performance by 1.6% compared with Multi-scale pyramid pooling (MPP) method [41] and 3.1% compared with Multi-scale Orderless Pooling (MOP) method [8]. Meanwhile, it also achieves better results than Deep Spatial Pyramid (DSP) [32] and Multi-Scale Deep Spatial Pyramid (MS-DSP) [61]. The reason for the improvement is that our method alleviates the disturbance of the burstiness and introduces the high-order information by the weighted second-order pooling strategy. #### 4.5.3 Performance of the SSNet Mid-Level Image Representation in Practical Applications

In this section, we further evaluate the proposed method on the PFID61 and Food-101 datasets to validate its effectiveness in practical applications. In the experiments, we integrate the SSNet framework with the Dense SIFT and the off-the-shelf CNN features respectively to build image representations and \begin{table}
\begin{tabular}{c c c c c c} \hline \hline  & \multicolumn{2}{c}{**PASCAL VOC 2007**} & **Caltech 101** & **Caltech 256** & **15-Scene** \\ \hline \multirow{2}{*}{Baseline} & FC-CNN [36] & 84.3 & 89.7 & 81.3 & 90.2 \\  & CL-FV & 85.1 & 86.1 & 79.5 & 90.5 \\ \hline \multirow{4}{*}{SOA Methods} & MOP [8] & 85.5 & 91.8 & 85.4 & 92.7 \\  & MPP [41] & 88.0 & 93.7 & 87.3 & 94.1 \\ \cline{1-1}  & DSP [32] & 88.6 & 94.7 & 84.2 & 91.1 \\ \cline{1-1}  & MS-DSP [61] & 89.3 & 95.1 & 85.5 & 91.8 \\ \hline
**Ours** & **FV + SSNet(max)** & **89.6** & **95.3** & **87.6** & **95.7** \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Comparison with the baselines and state-of-the-art methods based on CNN local features (MAP in percentage). compare them with different representation approaches, such as FV [22], FC-CNN [36], and MOP [8]. The results (measured in terms of MAP) are summarized in Table 4.

First, with the same settings as in the previous experiments, we conduct a series of experiments on the two datasets using the Dense SIFT features. In these experiments, we choose the FV [22] encoding method as the benchmark and integrate it with our proposed SSNet framework to build image presentations. It can be clearly seen that the performance of the proposed method FV+SSNet (57.1% on PFID61 and 52.4% on Food-101) outperforms the original FV (40.2% on PFID61 and 38.9% on Food-101) on the two datasets. We ascribe this improvement to the saliency detection and the weighted second-order pooling successfully leveraged in the proposed SSNet.