# SSNet: Learning Mid-Level Image Representation Using Salient Superpixel Network
## 3 The Proposed Method for Image Representation
### The Mid-Level Image Representation Based on SSNet  
In the SSNet representation framework, in order to reduce the side effect of the burstiness in the background of an image, the saliency detection method proposed in Section 3.1 is applied, but here, superpixels not pixels are used to generate the saliency map. After that, only the patches with the top-S large salient value are used to represent the image. Here, we set  
\[S=\left[NumSuppixel_{img}*Ratio_{sal}\right] \tag{33}\]  
where \(NumSuppixel_{img}\) is the number of the superpixels included in an image, and \(Ratio_{sal}\) is a predefined constant that represents the ratio between the number of the chosen saliency superpixels and the total number of the superpixels in an image. At the same time, in order to introduce higher-order feature information, the weighted second-order pooling is proposed. For each saliency superpixel \(R_{i},\ i=1,2,\cdots S\), we extract \(M_{i}\) features \(x_{ij},\ j=1,2,\cdots M_{i}\), and encode them by the encoding method \(\varphi(\cdot)\). Assuming the matrix \(\varPhi_{i}=\left[\varphi_{i1},\cdots,\varphi_{iM_{i}}\right]\) includes encoding outcomes of the features inside in the superpixel \(R_{i}\), we can obtain the weight vector \(w_{i}\) for the \(\varPhi_{i}\) based on Equation (23). Further, using Equation (26), the first-order pooling results for patch \(R_{i},\ i=1,2,\cdots S\) can be decided. After that, we can get the weighted second-order pooling result with the proposed methods in Section 4.2, and achieve the final representation using the approaches in Section 4.3. Figure 5 illustrates the pipeline of the SSNet image representation. In this paper, we try to adopt two different features respectively to build the SSNet image representation. (1) Dense SIFT. We extract Dense SIFTs within each superpixel region and encode them by the BoVW way. And the variable _width_ and _height_ in Equation (24) are set as the width and height of an input image respectively. (2) Off-the-shelf CNN local features. For each superpixel, we input the region of its circumscribed rectangle into a pretrained CNN model and extract the CNN local features by the way in [9] as mentioned in Section 2.2, and the variable _width_ and _height_ in Equation (24) are set as the width and height of the feature map, respectively. The proposed image representation method based on the SSNet is summarized in Algorithm 1. Figure 5: Illustration of the pipeline for generating the Salient Superpixel Network (SSNet) mid-level image representation. ```
0: a. Input image \(I\) \(Ratios_{sl}\), the ratio between the number of the chosen saliency superpixels and the total number of the superpixels in an image \(Df\), the reduced dimension of the feature \(f_{i}\) \(Dc\), the reduced dimension of the encoding vector \(Ratios_{sl}\)
```
0: The final mid-level representation \(FinalRep\) for the input image
1. Computing the saliency map of the image \(I\);
2. Computing the \(S\), the number of salient regions used to represent an image, using Equation (33);
3. Choosing the salient regions \(SR=\{sr_{1},sr_{2},\cdots sr_{S}\}\) with the top-\(S\) saliency value;
4. for \(i=1\):\(S\) ```
* 4.1 Extracting \(N_{i}\) Dense SIFTs/CNN local features from \(sr_{i}\) and reducing their dimensionality to \(Df\);
* 4.2 Encoding the features, reducing the dimensionality of the codes to \(Dc\), and achieving the coding matrix \(\Phi_{i}=[\varphi_{1},\cdots,\varphi_{N_{i}}]\);
* 4.3 Computing the weight \(w_{i}\) for the codes using Equation (25);
* 4.4 Computing the first-order pooling vector \(v_{i}\) for the salient region \(sr_{i}\) using Equation (26); end for
5 Computing the second-order pooling \(WSOAP\)/ \(WSOMP\) for the image using Equation (27)/Equation (28);
6 Mapping or Normalizing the \(WSOAP\)/ \(WSOMP\) using Equation (29)/Equation (30);
7 Obtaining the final representation \(FinalRep\) for image \(I\) by vectorizing operation using Equation (31)/Equation (32).