# SSNet: Learning Mid-Level Image Representation Using Salient Superpixel Network
## 2 Related Work
### Research on Mid-Level Image Representation  
Learning a robust and discriminative image representation is a critical issue for image classification. Over the years, researchers have proposed a wide variety of representation methods. Typically, these methods can be categorized into three levels: high level, mid level, and low level. The high-level representation usually offers discriminative power by conveying semantic or prior knowledge about objects. However, the data-driven based high-level information requires a large number of human-labeled tags. Low-level methods often use the basic features obtained from some various hand-crafted descriptors, such as SIFT [6] and histogram of oriented gradients (HOG) [25], to represent the images. Despite many more informative descriptors being put forward, the performance of low-level representation methods still remain limited. Recently, several approaches have been proposed to transform low-level features into the mid-level representation. It is more informative and  
Figure 2: The overall architecture of the proposed representation model and its key modules. The two red bounding boxes denote the proposed modules. The left one denotes the saliency detection method and the right one indicates the weighted second-order pooling module. discriminative than the low-level one and is able to bridge the semantic gap between the low-level and high-level representation. At present, bag-of-visual-words (BoVW) and convolutional neural networks (CNN) models are often applied to learn the mid-level image representation. Over the years, researchers have proposed a wide variety of image representation methods based on the bag-of-visual-words model. This model mainly consists of the following four stages. (1) Feature extraction. Traditionally, local features such as SIFT [6] and Dense SIFT [7] have been widely used for representing images. (2) Building the codebook. Usually, K-means or the Gaussian Mixture Model (GMM) clustering algorithm is used to build the codebook. (3) Feature encoding. In this stage, extracted local features are encoded by the learned codebook. There are some classic encoding methods, such as locality-constrained soft assignment (LCSA) [26], soft assignment (SA) [27], fisher vector (FV) [22], and vector of locally aggregated descriptor (VLAD) [5]. FV has achieved state-of-the-art results in several recognition tasks before the emergence of the Deep Learning. (4) Pooling operation. The pooling operation aggregates all the encoded features to form the final representation vectors for the images, and average pooling and max pooling are often used. In addition, the research [7; 28] obtains the mid-level representation by pooling more discriminative information than the standard BoVW model does. [PERSON] et al. [29] propose a novel descriptor that encapsulates the mid-level information based on superpixel structure. [PERSON] et al. [30] generate visual phrases via combining the visual words depending on the co-occurrence of them. [PERSON] et al. [30] propose a novel method to learn distinctive parts of objects or scenes automatically in a supervised way. With the learned parts, a discriminative mid-level representation is generated. Recently, convolutional neural networks (CNNs) have been successfully applied to various computer vision tasks such as image classification [2; 31; 32] and object detection [33; 34]. Such great success spurs a flurry of research work on further improving CNN architecture [32; 35], as well as on using CNN features as a universal representation for recognition [36; 37; 38]. Additionally, several works extract the local intermediate activations from a pretrained CNN model and aggregate them to generate a generic mid-level representation, i.e., the representation based on an off-the-shelf CNN model. The CNN-based representations achieve state-of-the-art performance in wide visual recognition tasks [8; 39; 40; 41; 42]. In this paper, we mainly discuss the mid-level representation based on BoVW with the handcrafted and off-the-shelf CNN local features.