# SSNet: Learning Mid-Level Image Representation Using Salient Superpixel Network
## 3 The Proposed Method for Image Representation
### The Mid-Level Image Representation Based on SSNet
We observe that the weighted second-order pooling scheme is effective for lots of commonly used encoding methods. Different from traditional feature pooling methods, e.g., the max pooling and average pooling, the weighted second-order pooling provides more discriminative ability. Moreover, we find that WSOMP consistently outperforms WSOAP. So, in the following experiments, we only discuss the WSOMP representation. ### Performance Analysis of Mid-Level Representation Based on SSNet

In this section, we assess the validation of the proposed SSNet representation framework. In the experiments, integrating the SSNet framework with the handcrafted and off-the-shelf CNN features respectively, we test their performances on four benchmark datasets. #### 4.5.1 Comparison with Related BoVW Baselines

We integrate the BoVW baselines, such as ScSPM [3], LLC [4], FV [22], and VLAD [5], with the proposed SSNet framework respectively and compare their classification results. In the experiments, the same settings as in Section 4.4 are adopted. The results in Table 2 show that the representations based on the SSNet outperform the corresponding baseline methods by a large margin on all datasets. Taking the results of the VOC 2007 as an example, when Sparse Coding (SC) [3] vectors are integrated with SSNet, the classification accuracy is over +14% more than the original one. When the SSNet framework is combined with other baseline representations, such as LLC, FV, and VLAD, their performances have also remarkably improved. FV+SSNet achieves the best performance among them and reaches 76.1% recognition accuracy. \begin{table}
\begin{tabular}{c c c c c} \hline \hline  & **PASCAL VOC 2007** & **Caltech 101** & **Caltech 256** & **15-Scene** \\ \hline ScSPM [3] & 51.9 & 73.2 & 37.5 & 80.3 \\ LLC [4] & 58.7 & 75.4 & 46.3 & 82.0 \\ FV [22] & 61.8 & 77.8 & 52.1 & 85.3 \\ VLAD [5] & 55.3 & 77.6 & 48.9 & 83.2 \\ SC + WSOAP & 63.1 & 80.1 & 47.3 & 85.7 \\ SC + WSOMP & 64.9 & 81.9 & 51.8 & 86.1 \\ LLC + WSOAP & 69.8 & 82.7 & 58.2 & 87.4 \\ LLC + WSOMP & 71.3 & 83.1 & 60.3 & 87.9 \\ FV + WSOAP & 72.4 & 86.1 & 66.1 & 89.6 \\
**FV + WSOMP** & **73.5** & **87.5** & **69.9** & **91.0** \\ VLAD + WSOAP & 69.6 & 84.2 & 61.6 & 86.7 \\ VLAD + WSOMP & 72.1 & 85.6 & 63.2 & 88.2 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Classification accuracy (MAP in %) of the weighted second-order pooling scheme integrating with the bag-of-visual-words (BoVW) baseline encoding methods. FV: Fisher Vector, VLAD: Vector of Locally Aggregated Descriptor, WSOAP: weighted second-order average pooling, WSOMP: weighted second-order pooling. \begin{table}
\begin{tabular}{c c c c c} \hline \hline  & **PASCAL VOC 2007** & **Caltech 101** & **Caltech 256** & **15-Scene** \\ \hline ScSPM [3] & 51.9 & 73.2 & 37.5 & 80.3 \\ LLC [4] & 58.7 & 75.4 & 46.3 & 82.0 \\ FV [22] & 61.8 & 77.8 & 52.1 & 85.3 \\ VLAD [5] & 55.3 & 77.6 & 48.9 & 83.2 \\ SC + SSNet(max) & 66.5 & 83.3 & 53.1 & 87.2 \\ LLC + S



#### 4.5.2 Comparison with the Baselines and State-of-the-Art Methods with the CNN Local Features

We further evaluate the performance of the SSNet framework with the off-the-shelf CNN local features. Recent works [9; 38] advocate that the activations of a convolutional layer can be used as the local features, since they are more general and preserve spatial information. In the following experiments, we employ the pretrained VGG-16 model [32] and choose the activations of its last convolutional layer as the CNN local features. Given an image, the patches with the top-\(S\) saliency value are chosen first, and the regions of their circumscribed rectangle are used as the input to extract CNN features. Following the extracting process depicted in [9], i.e., the method introduced in Section 2.2, for a saliency patch, thousands of 9 \(\times\) 512 dimension CNN features are extracted. While considering running efficiency, we reduce the dimension of the activation achieved from 512 to 32, which is determined by cross-validation from the set \(\{128,64,32\}\).