# Improved CNN Based on Super-Pixel Segmentation

Yadong Yang, Xiaofeng Wang

► To cite this version:

Yadong Yang, Xiaofeng Wang. Improved CNN Based on Super-Pixel Segmentation. 2nd International Conference on Intelligence Science (ICIS), Oct 2017, Shanghai, China. pp.305-310, 10.1007/978-3-319-68121-4_33. hal-01820924

HAL Id: hal-01820924

https://inria.hal.science/hal-01820924

Submitted on 22 Jun 2018

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Improved CNN based on Super-pixel Segmentation

Yang Yadong¹, Wang Xiaofeng¹

¹College of Information Engineering, Shanghai Maritime University, Shanghai, China

yangyadong03@stu.shmtu.edu.cn

xfwang@shmtu.edu.cn

**Abstract.** Convolutional neural network has unique superiority in images pro-
cessing, it can effectively extract features and reduce data dimensions by convo-
lution and pooling. But it takes the "violent segmentation" method in the process
of pooling. This method cannot guarantee the final selected pixel value can be a
good representative of the partial features, neither average pooling nor max pool-
ing. Therefore, three pooling methods based on super-pixel segmentation are pro-
posed, called “super-pixel average pooling”, “super-pixel max pooling” and “su-
per-pixel smooth pooling”. Firstly, the super-pixel segmentation is performed on
the feature images, and then the value of the point which has the smoothest gra-
dient in each super pixel is selected to represent the feature of the local area.
Compared to the violent segmentation pooling operation, this method exhibits
more stable characterization ability, it can retain image features perfectly while
reduce the data dimensions. Experiments show that the improved convolutional
neural network achieved better results than normal algorithm in the standard data
sets.

**Keywords:** convolutional neural network; super-pixel segmentation; super-pixel average pooling; super-pixel max pooling; super-pixel smooth pooling

# 1 Introduction

Convolutional neural network (CNN) is a deep neural network model with convolu-
tion structure. It can effectively reduce the number of weights and lower the complexity
of the network. This model has some invariance for scaling and other forms of defor-
mation. Each neuron only needs to perceive the local image area, The network obtain
global information by combining these neurons that perceive different local regions at
higher levels. In 1989, LeCun Y et al. proposed a CNN model “LeNet-5[1]” for character
recognition. LeNet-5 consists of convolutional layers, sub-sampling layers and fully
connected layers. The system achieves good results in small-scale handwritten numeral
recognition. In the ImageNet contest, “AlexNet[2]”, a new network architecture of CNN,
designed by Krizhevsky et al. won the 2012 championship. AlexNet and OverFeat[3]
are quite powerful CNN models trained on large natural image datasets. By improving
the network performance, Girshick R et al. proposed R-CNN[4] (Regions with CNN)
can complete the target detection task effectively. He K M utilize spatial pyramid pool-
ing (SPP) to deal with different size and aspect ratio of the input images, this new net-
work called SPP-Net[6]. In recent years, more and more popular method to optimize
CNN models is to develop deeper and more complex network structures, and then to
train them with massive training data. VGG[5] (visual geometry group), a 19-layer depth
network, mainly explores the importance of depth for the network. GoogLeNet[6] is the
particular incarnation for ILSVRC14 submit by Szegedy C et al., a 22 layers deep net-
work, the quality of which is evaluated in classification and detection.In order to im-
prove the performance of CNN, the researchers not only discuss the structure of CNN
and its applications, but also improve the design of the network layers, the loss function,
activation function, regular items and many other aspects of the existing network. They
achieved a series of results, for example, Inception network[5], stochastic pooling,
ReLU[2], Leakly ReLU[7], Cross entropy loss, Batch normalization[8], etc.

2 Related work

2.1 Convolutional neural network

In a typical CNN model, the beginning layers are usually alternating between the
convolution layer and the sub-sampling layer. The last few layers of the network near
the output layer are usually fully connection networks. The four basic elements of CNN
are convolutional layers, sub-sampling layers, all connected layers and back propaga-
tion (BP) algorithm.

The convolutional layers receive the input images or the feature maps from the pre-
vious layer, they carry out the convolution operation through the N convolution cores
and activate operation using the activation function, then N new feature maps are gen-
erated and directed into the next sub-sampling layer. The function of convolution is
defined by

$$
x_j^l = f \left( \sum_{i \in M_j} x_i^{l-1} * k_{ij}^l + b_j^l \right) \quad (1)
$$

where l denotes the convolution layer, j is the j-th channel, $x_j^l$ is the output value from
the j-th channel of convolution layer l, f(·) is the activation function, $M_j$ represents the
subset of feature maps, $k_{ij}^l$ is the convolution kernel matrix, * is the convolution oper-
ation, $b_j^l$ is offset of l.

The sub-sampling layers perform sub-sampling operations on the input feature maps,
which can effectively reduce the computational complexity and extract more repre-
sentative local features. The function of sub-sampling is defined by

$$
x_j^l = f(\beta_j^l \text{down}(x_j^{l-1}) + b_j^l) \quad (2)
$$

where $\beta_j^l$ denotes is the sub-sampling weight coefficient, down(·) is sub-sampling function.
The fully connected layers consist of an input layer, several hidden layers and an output layer. The upper layer's feature maps is spliced into one-dimensional vector as input data, and the final outputs is obtained by weighting and activating.

$$x^l = f(\omega^l x^{l-1} + b^l) \qquad (3)$$

where $\omega^l$ is the neural network weight.

The BP algorithm is a common neural network method in supervised learning. For the convolutional neural network, the main job is to optimization the convolution kernel parameters, the sub-sampling layers weights, the network weights of the full connection layer and the bias parameters of all layers. The essence of the BP algorithm is to allow us to calculate each network layer's effective errors and to derive the learning rules of network parameters, and then drive the actual network outputs are closer to the target values.

## 2.2 Super-pixel Segmentation

Super pixels are divided a pixel-level image into district-level image or sets of pixels by image segmentation. The goal of super pixel segmentation is to change or simplify the representation of an image into something that is more convenient and significant to analyze. The super pixel segmentation method has been under intensive study, now there are many super pixel segmentation algorithms.

Super pixels are obtained by using NCut[9] and SLIC[10] algorithm has a high compactness. Using SLIC and Watershed algorithm for ultra-pixel segmentation is very productive. But if you put more emphasis on edge accuracy and regional merging, you can choose Marker-based Watershed and Meanshift algorithm.

# 3 Improved CNN

CNN have been substantiated to provide a powerful approach for image processing. In this work, we focus on the pooling part of the CNN.

The common CNN pooling methods include average pooling, max pooling and multi-scale mixing pooling. These methods have achieved some good results, they calculate a value representing the local region feature in the pre-divided local area of feature maps. However, those violent segmentation methods have their own unreasonable place, including ignore some local features or neutralize some salient features. This paper presents three pooling methods based on super pixel segmentation. Firstly, take super-pixel segmentation action on the feature maps, and the pixel feature values in each local region (ie, super pixels) we got are similar. Then three kinds of pooling methods are come up with super pixel segmentation.

1) Calculate the average value of pixels within each super pixel called super pixel average pooling.

2) Get the max value of pixels within each super pixel, that is, super pixel max pooling.
3) And super pixel smooth pooling trying to take the value of the point which has the smoothest gradient in each super pixel.

This paper only considers the super pixel level, does not deal with regional merger after image segmentation. Based on SLIC has the low time complexity and high com- pactness, we choose to use SLIC algorithm for super pixel segmentation.

<figure><img src="image_2.png" /><figcaption>Fig. 1. The super-pixel pooling schematic diagram</figcaption></figure>

These segmentation methods can not only extract the necessary features, but also the local stability of the images has no destruction at all.

4 Results

The structure of CNN we used in experiment for MNIST has 7 layers. The network is composed of 2 convolutional layers, 2 pooling layers and 3 fully connected layers. The experiment for CIFAR-10 use AlexNet. While the convolutional layers are fol- lowed by rectified linear operator (ReLU) layers and the pooling layers take the value of local regions with two-pixel strides, the dropout layers having a 0.5 dropout ratio. The last fully connected layer employ softmax function as a multi-class activation func- tion.

Because of the MNIST data have relatively simple characteristics. Here divide the MNIST test set into six parts randomly, each part is transformed into different scales or angles, and then normalize the images in 28\*28 pixels to get a new test set. The training set remains the same. Test on data sets MNIST, New MNIST.

<figure><img src="image_3.png" /><figcaption>Fig. 2. Part of the New MNIST test images</figcaption></figure>

In the following tables: "max", "avg", "sps", "sp-max", "sp-avg" each representing max pooling CNN method, average pooling CNN method, super-pixel smooth pooling CNN method, super-pixel max pooling CNN method and super-pixel average pooling CNN method. The results are shown as follows.
<table><caption>Table 1. Test results of the five methods on the Mnist dataset.</caption><thead><tr><th></th><th>Methods</th><th>Training error(%)</th><th>Test error(%)</th></tr></thead><tbody><tr><td rowspan="5">Table results five</td><td>max</td><td>0.01</td><td>0.8</td></tr><tr><td>avg</td><td>0.06</td><td>0.9</td></tr><tr><td>sps</td><td>0.04</td><td>0.7</td></tr><tr><td>sp+max</td><td>0.05</td><td>0.8</td></tr><tr><td>sp+avg</td><td>0.06</td><td>0.8</td></tr><tr><td rowspan="5">Table results five ods on</td><td>max</td><td>0.04</td><td>10.7</td></tr><tr><td>avg</td><td>0.1</td><td>9.8</td></tr><tr><td>sps</td><td>0.06</td><td>4.7</td></tr><tr><td>sp+max</td><td>0.06</td><td>8.9</td></tr><tr><td>sp+avg</td><td>0.08</td><td>6.6</td></tr><tr><td colspan="4" style="text-align: center;">CIFAR-10 dataset.</td></tr><tr><td></td><th>Methods</th><th>Training error(%)</th><th>Test error(%)</th></tr><tr><td></td><td>max</td><td>2.67</td><td>29.5</td></tr><tr><td></td><td>avg</td><td>12.50</td><td>28.8</td></tr><tr><td></td><td>sps</td><td>5.65</td><td>24.2</td></tr><tr><td></td><td>sp+max</td><td>6.54</td><td>26.4</td></tr><tr><td></td><td>sp+avg</td><td>8.26</td><td>25.1</td></tr></tbody></table>

Based on the above results it can be seen that: the average pooling method is better than the max pooling methods, because of the local regions average values are more representative. Super-pixel pooling method is better than the standard pooling method, which is due to better image segmentation. The super pixel smooth pooling method gets the best results, because this method can find the most representative values.

## 5 Conclusions

A neoteric and universal approach is proposed for improving CNN performance through using the super-pixel pooling to train the network. This approach makes the models have more stable characterization and better generalization, and it can be used for different CNN network structures. Extensive experiments on several standard data sets for the image classification prove that using the super-pixel pooling in the training process can significantly enhance performance of CNN models, in comparison with the same model trained without employing this method.

The super-pixel segmentation technique may be further used for the convolution operation. By introducing the fuzzy segmentation method, adjust the number of extra pixels and the number of pixels per super-pixel contains after segmentation. This method
is suitable for convolution of large-size images, and convolution of image sets contain-
ing different sizes of images. Moreover, we can use the improved CNN to resolve many
practical problems such as pedestrian detection, ship classification, text recognition or
medical images processing.

References

1. LeCun Y, Bottou L, Bengio Y, Haffner P. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 1998, 86(11): 2278-2324.

2. Krizhevsky A, Sutskever I, Hinton G E. ImageNet classification with deep convolutional neural networks. In: Proceedings of Advances in Neural Information Processing Systems 25. Lake Tahoe, Nevada, USA: Curran Associates, Inc., 2012. 1097-1105.

3. Girshick R, Donahue J, Darrell T, Malik J. Rich feature hierarchies for accurate object detection and semantic segmentation. In: Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition. Columbus, USA: IEEE, 2014: 580-587.

4. He K M, Zhang X Y, Ren S Q, Sun J. Spatial pyramid pooling in deep convolutional networks for visual recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015, 37(9):1904-1916.

5. Szegedy C, Liu W, Jia Y Q, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A. Going deeper with convolutions. In: Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition. Boston, MA: IEEE, 2015:1-9.

6. Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition [Online], available: http://arxiv.org/abs/1409.1556, May 16, 2016.

7. Maas A L, Hannun A Y, Ng A Y. Rectier nonlinearities improve neural network acoustic models. In: Proceedings of ICML Workshop on Deep Learning for Audio, Speech, and Language Processing. Atlanta, USA: IMLS, 2013.

8. Ioffe S, Szegedy C. Batch normalization: accelerating deep network training by reducing internal covariate shift. In: Proceedings of the 32nd International Conference on Machine Learning. Lille, France: IMLS, 2015: 448-456.

9. Jianbo Shi, J. Malik. Motion segmentation and tracking using normalized cuts. In: Sixth International Conference on Computer Vision. 1998:1154-1160.

10. Eun-Sung Jung, Sanjay Ranka, Sartaj Sahni. Bandwidth Allocation for Iterative Data-Dependent E-science Applications. In: EEE/ACM International Conference on Cluster, Cloud and Grid Computing. 2010: 233 – 242.