UNSUPERVISED IMAGE SEGMENTATION BY BACKPROPAGATION
Asako Kanezaki
National Institute of Advanced Industrial Science and Technology (AIST)
2-4-7 Aomi, Koto-ku, Tokyo 135-0064, Japan
ABSTRACT
We investigate the use of convolutional neural networks(CNNs) for unsupervised image segmentation. As in thecase of supervised image segmentation, the proposed CNN
assigns labels to pixels that denote the cluster to which the
pixel belongs. In the unsupervised scenario, however, no
training images or ground truth labels of pixels are given
beforehand. Therefore, once when a target image is input,
we jointly optimize the pixel labels together with feature representations while their parameters are updated by gradientdescent. In the proposed approach, we alternately iteratelabel prediction and network parameter learning to meet thefollowing criteria: (a) pixels of similar features are desired to
be assigned the same label, (b) spatially continuous pixels aredesired to be assigned the same label, and (c) the number of
unique labels is desired to be large. Although these criteria
are incompatible, the proposed approach finds a plausible
solution of label assignment that balances well the above criteria, which demonstrates good performance on a benchmark
dataset of image segmentation.
Index Terms— Convolutional neural networks, Unsupervised learning, Feature clustering
1. INTRODUCTION
Image segmentation has attracted attention in computer visionresearch for decades. The applications of image segmentation
include object detection, texture recognition, and image compression. In the supervised scenario, in which a set of pairsof images and pixel-level semantic labels (such as “sky” or“bicycle”) is used for training, the goal is to train a systemthat classifies the labels of known categories for image pixels. On the other hand, in the unsupervised scenario, image
segmentation is used to predict more general labels, such as“foreground” and “background”. The latter case is more challenging than the former, and furthermore, it is extremely hard
to segment an image into an arbitrary number (≥ 2) of plausible regions. The present study considers a problem in which
an image is partitioned into an arbitrary number of salient or
meaningful regions without any previous knowledge.
Once the pixel-level feature representation is obtained,
image segments can be obtained by clustering the feature vectors. However, the design of feature representation remains achallenge. The desired feature representation highly depends
on the content of the target image. For instance, if the goal
is to detect zebras as a foreground, the feature representation
should be reactive to black-white vertical stripes. Therefore,
the pixel-level features should be descriptive of colors andtextures of a local region surrounding each pixel. Recently,
convolutional neural networks (CNNs) have been successfully applied to semantic image segmentation (in supervised
learning scenarios) for autonomous driving or augmented reality games, for example. CNNs are not often used in fully
unsupervised scenarios; however, they have great potential for
extracting detailed features from image pixels, which is necessary for unsupervised image segmentation. Motivated by
the high feature descriptiveness of CNNs, we present a joint
learning approach that predicts, for an arbitrary image input,
unknown cluster labels and learns optimal CNN parameters
for the image pixel clustering. Then, we extract a group ofimage pixels in each cluster as a segment.
Now, we describe the problem formulation that we solvefor image segmentation. Let {xn ∈ Rp }N
n=1 be a set of pdimensional feature vectors of image pixels, where N denotes
the number of pixels in an input image. We assign cluster labels {cn ∈ Z}N
n=1 to all of the pixels by cn = f (xn ), wheref : Rp → Z denotes a mapping function. Here, f can, forinstance, be the assignment function that returns the ID of thecluster centroid closest to xn among k centroids, which are
obtained by, e.g., k-means clustering. For the case in which
f and the feature representation {xn } are fixed, {cn } are obtained by the above equation. On the other hand, if f and{xn } are trainable, whereas {cn } are given (fixed), then theabove equation can be regarded as a standard supervised classification problem. The parameters for f and {xn } in thiscase can be optimized by gradient descent if f and the feature
extraction functions for {xn } are differentiable. However, inthe present study, we predict unknown {cn } while training
the parameters of f and {xn } in a fully unsupervised manner.
To put this into practice, we alternatively solve the following
two sub-problems: prediction of the optimal {cn } with fixed
f and {xn } and training of the parameters of f and {xn }with fixed {cn }.
Let us now discuss the characteristics of the cluster labels {cn } necessary for good image segmentation. Similar toprevious studies on unsupervised image segmentation [1, 2],we assume that a good image segmentation solution matches
well a solution that a human would provide. When a humanis asked to segment an image, he/she would most likely create segments, each of which corresponds to the whole or a(salient) part of a single object instance. An object instancetends to contain large regions of similar colors or texture patterns. Therefore, grouping spatially continuous pixels thathave similar colors or texture patterns into the same cluster
is a reasonable strategy for image segmentation. On the other
hand, in order to separate segments from different object instances, it is better to assign different cluster labels to neighboring pixels of dissimilar patterns. To facilitate the cluster
separation, we also consider a strategy in which a large number of unique cluster labels is desired. In conclusion, we introduce the following three criteria for the prediction of {cn }:(a) Pixels of similar features are desired to be assigned the same label.
(b) Spatially continuous pixels are desired to be assigned the same label.
(c) The number of unique cluster labels is desired to be large.

Note that these criteria are incompatible so that they are neversatisfied perfectly. However, through the gradual optimizationthat considers all three criteria simultaneously, the proposedsystem finds a plausible solution of {cn } that balance wellthese criteria. In Section 2, we describe the proposed iterative
approach to predict {cn } that satisfy the above criteria.
2. METHOD
2.1. Constraint on feature similarity
Let us consider the first criterion, which assigns the same label to pixels of similar features. The proposed solution is
to apply a linear classifier that classifies the features of eachpixel into q classes. In the present paper, we assume the input
to be an RGB image I = {vn ∈ R3 }N
n=1 , where each pixel
value is normalized to [0, 1].
We compute a p-dimensional feature map {xn } from{vn } through M convolutional components, each of which
consists of a 2D convolution, ReLU activation function, and abatch normalization function, where a batch corresponds to N
pixels of a single input image. Here, we set p filters of region
size 3 × 3 for all of the M components. Note that these components for feature extraction are able to be replaced by alternatives such as fully convolutional networks (FCN) [3]. Next,
we obtain a response map {yn = Wc xn + bc }N
n=1 by applying a linear classifier, where Wc ∈ Rq×p and bc ∈ Rq . We
then normalize the response map to {yn } such that {yn }N
n=1
has zero mean and unit variance. The motivation behind thenormalization process is described in Sec. 2.3. Finally, we
obtain the cluster label cn for each pixel by selecting the dimension that has the maximum value in yn . We herein referto this classification rule as argmax classification. Intuitively,
the above-mentioned processing corresponds to the clusteringof feature vectors into q clusters. The ith cluster of the final
responses {yn } can be written as:≥ yn,j, ∀j},
Ci = {yn ∈ Rq | yn,i(1)denotes the ith element of yn . This is equivalent
where yn,ito assigning each pixel to the closest point among the q representative points, which are placed at infinite distance on therespective axis in the q-dimensional space. Note that Ci canbe ∅, and therefore the number of unique cluster labels is arbitrary from 1 to q.

2.2. Constraint on spatial continuity
The basic concept of image pixel clustering is to group similar pixels into clusters (as shown in Sec. 2.1). In image segmentation, however, it is preferable for the clusters of image pixels to be spatially continuous. Here, we add an additional constraint that favors cluster labels that are the sameas those of neighboring pixels. We first extract K fine superpixels {Sk }K
k=1 (with a large K) from the input image
I = {vn }N
n=1 , where Sk denotes a set of the indices of pixels that belong to the kth superpixel. Then, we force all of
the pixels in each superpixel to have the same cluster label.
More specifically, letting |cn |n∈Sk be the number of pixels
in Sk that belong to the cn th cluster, we select the most frequent cluster label cmax , where |cmax |n∈Sk ≥ |cn |n∈Sk for
all cn ∈ {1, . . . , q}. The cluster labels are then replaced by
cmax for n ∈ Sk . In the present paper, we use SLIC [4] with
K = 10, 000 for the superpixel extraction.
2.3. Constraint on the number of unique cluster labels
In the unsupervised image segmentation, there is no clue as tohow many segments should be generated in an image. Therefore, the number of unique cluster labels should be adaptive to the image content. As described in Sec. 2.1, the proposed strategy is to classify pixels into an arbitrary numberq  (1 ≤ q  ≤ q) of clusters, whereas q is the possibly maximum value of q  . A large q  indicates oversegmentation,
whereas a small q  indicates undersegmentation. The aforementioned criteria (a) and (b) only facilitate the grouping of
pixels, which could lead to a naive solution that q  = 1. To
prevent this kind of undersegmentation failure, we introduce
the third criterion (c), which is the preference for a large q  .
Our solution is to insert the intra-axis normalization process for the response map {yn } before assigning cluster labels via argmax classification. Here, we use batch normalization [5] (where a batch corresponds to N pixels of a single
input image), which is described as follows:
yn,i − μi

=  2
yn,i
,
σi + 

(2)where μi and σi denote the mean and standard deviation of{yn,i }, respectively. Note that  is a constant that is added
to the variance for numerical stability. This operation (alsoknown as whitening) converts the original responses {yn } to{yn }, where each axis has zero mean and unit variance. Then(i = 1, . . . , q) has an even chance to be the maxeach yn,iimum value of yn across axes. Even though this operation
does not guarantee that every cluster index i(i = 1, . . . , q)

Algorithm 1: Unsupervised image segmentation
Input: I = {vn ∈ R3 }N// RGB image
n=1//
Label image
Output: L = {cn ∈ Z}N
n=1// Initialize{Wm , bm }M
m=1 ← Init()// Initialize{Wc , bc } ← Init()N{Sk }K
k=1 ← GetSuperPixels( {vn }n=1 )for t = 1 to T doNM{xn }Nn=1 ← GetFeats( {vn }n=1 , {Wm , bm }m=1 )NN{yn }n=1 ← { Wc xn + bc }n=1N{yn }Nn=1 ← Norm( {yn }n=1 ) // Batch norm.
N{cn }n=1 ← { arg max yn }N
n=1 // Assign labels
for k = 1 to K do
cmax ← arg max |cn |n∈Skcn ← cmax for n ∈ Sk
L ← SoftmaxLoss( {yn , cn }N
n=1 )
,
{W
,
b
}
←
Update(L ){Wm , bm }Mccm=1

Fig. 1. Illustration of the proposed algorithm for training the
proposed CNN network.

achieves the maximum value for any n(n = 1, . . . , N ), because of this operation, many cluster indices will achieve themaximum value for any n(n = 1, . . . , N ). As a consequence,
this intra-axis normalization process gives the proposed system a preference for a large q  .
2.4. Learning network by backpropagation
In this section, we describe how to self-train the network for
unsupervised image segmentation. Once a target image is input, we alternatively solve the following two sub-problems:
prediction of cluster labels with fixed network parameters andtraining of network parameters with the (fixed) predicted cluster labels. The former corresponds to the forward process ofa network followed by the superpixel refinement describedin Sec. 2.2. The latter corresponds to the backward processof a network based on gradient descent. As with the case of
supervised learning, we calculate the softmax loss (i.e., thecross-entropy loss) between the network responses {yn } andthe refined cluster labels {cn }. Then, we backpropagate theerror signals to update the parameters of convolutional filters {Wm , bm }M
m=1 as well as the parameters of the classifier {Wc , bc }. In the present paper, we use stochastic gradient descent with momentum for updating the parameters.
The parameters are initialized with Xavier initialization [6],which samples values from the uniform distribution normalized according to the input and output layer size. We iterate this forward-backward process T times to obtain the final prediction of cluster labels {cn }. Algorithm 1 shows thepseudocode for the proposed unsupervised image segmentation algorithm. Figure 1 illustrates the proposed algorithm fortraining the proposed CNN network.
As shown in Fig. 1, the proposed CNN network is composed of basic functions. The most characteristic part of the
proposed CNN is the existence of the batch normalization
layer between the final convolution layer and the argmax classification layer. Unlike the supervised learning scenario, in

Fig. 3. F-measure per image for various methods.
which the target labels are fixed, the batch normalization of
responses over axes is necessary for obtaining reasonable labels {cn } (see Sec. 2.3). Moreover, in contrast to supervised
learning, there are multiple solutions of {cn } with different
network parameters that achieve near zero loss. The value ofthe learning rate takes control over the balance between parameter updates and clustering, which leads to different solutions of {cn }. We empirically found that setting the learning
rate to 0.1 (with momentum 0.9) yielded the best results.
3. RESULTS
We evaluated the proposed method using 200 test images
from the Berkeley Segmentation Dataset and Benchmark(BSDS500) [7, 8]. The test images in this dataset are provided with more than 1,000 hand-labeled segmentations. We
trained the proposed CNN model with T = 500 iterations
for each image, altering the number of convolutional components M as 1, 2, . . . , 5. We fixed p = q = 100 for all of theexperiments. For comparison, we chose to use k-means clustering and the graph-based segmentation method (GS) [9].
For the k-means clustering, we used the concatenation of
RGB values in a α × α window for each pixel representation,
where α = 1, 3, 5, 7. We extracted connected components assegments from each cluster generated by k-means clusteringand the proposed method.
Figure 3 shows the F-measure per image, which is the har-Fig. 2. Example results of the proposed method. Different segments are shown in different colors.

(a) IoU = 0.5(b) IoU = 0.7

Fig. 4. Precision-recall curves.
monic mean of precision and recall, for various methods. We
calculated the intersection over union (IoU) of each estimatedsegment and ground truth segments and regarded the IoU as
correct if the maximum IoU is larger than 0.5. The numbers following the method names in Fig. 3 represent (α, k)
for k-means clustering, the similarity threshold β for merging neighboring segments with GS, and M for the proposed
method. We chose the best results from k = 2, 3, . . . , 20 for
k-means clustering and β = 100, 500, 1,000, 1,500, 2,000
for GS, which are all outperformed by the proposed method.
We also show the precision-recall curves with an IoU threshold 0.5 and 0.7 in Fig. 4. Here, segments were arranged inorder of decreasing size. The best average precision scoresfor each method with IoU = (0.5, 0.7) are (0.049, 0.0129)
with k-means clustering, (0.1161, 0.0449) with GS, and(0.1394, 0.0578) with the proposed method, which demonstrates the effectiveness of the proposed method. Figure 2
shows typical example results obtained using the proposed
method. Many meaningful segments with various colors
and textures (such as a tiger and a giraffe) are successfully
detected by the proposed method.
4. RELATED WORK
Semantic image segmentation based on CNN has been gaining attention in the literature [10, 11, 3, 12]. Existing workoften uses object detectors [13, 14, 15] or user inputs [16, 17]
to determine parameters for segmentation. Since pixel-levelannotations for image segmentation are difficult to obtain,
weakly supervised learning approaches using object boundingboxes [18, 19] or image-level class labels [20, 21, 22, 23] fortraining are widely used. However, to the best of our knowl-edge, no studies have considered CNN for image segmentation in a fully unsupervised manner.
Unsupervised deep learning approaches have focused
mainly on learning high-level feature representations using
generative models [24, 25, 26]. The motivation behind thesestudies is closely related to the conjecture in neurosciencethat there exist neurons that represent specific semantic concepts. Here, we are more interested in the application of deeplearning to image segmentation, and thus emphasize the importance of high-level features extracted with convolutional
layers. Deep CNN filters are known to be effective for texture
recognition and segmentation [27, 28].
Note that the convolution filters used in the proposed
method are trainable in the standard backpropagation algorithm, although there are no ground truth labels. The present
study is therefore related to recent research on deep embeddedclustering (DEC) [29]. The DEC algorithm iteratively refines
clusters by minimizing the KL divergence loss between softassigned data points with an auxiliary target distribution,
whereas the proposed method simply minimizes the softmax
loss based on the estimated clusters. Similar approaches,such as maximum margin clustering [30] and discriminative
clustering [31, 32], have been proposed for semi-supervised
learning frameworks, whereas the proposed method is focused on the fully unsupervised image segmentation task.
5. CONCLUSION
We presented a novel CNN architecture and its self-training
process that enables image segmentation in an unsupervisedmanner. Using the backpropagation of the softmax loss tothe normalized responses of convolutional layers, the proposed CNN jointly assigned cluster labels to image pixels
and updated the convolutional filters to achieve better separation of clusters. We also introduced a superpixel refinement
process to achieve the spatial continuity constraint for theestimated segments. Experimental results on the BSDS500
benchmark dataset demonstrated the effectiveness of the proposed method. An interesting direction for future researchis to investigate constraints other than superpixel refinement,e.g., using edge density for the segmentation priors.
ACKNOWLEDGMENT
The author is thankful to Dr. Masayuki Tanaka for his valuable comment.

6. REFERENCES[1] Ranjith Unnikrishnan, Caroline Pantofaru, and Martial Hebert,“Toward objective evaluation of image segmentation algorithms,” IEEE Trans. on Pattern Analysis and Machine Intelligence, vol. 29, no. 6, 2007.
[2] Allen Y. Yang, John Wright, Yi Ma, and Shankar Sastry, “Unsupervised segmentation of natural images via lossy data compression,” Computer Vision and Image Understanding, vol.
110, no. 2, 2008.
[3] Jonathan Long, Evan Shelhamer, and Trevor Darrell, “Fully
convolutional networks for semantic segmentation,” in CVPR,
2015.
[4] Radhakrishna Achanta, Appu Shaji, Kevin Smith, Aurelien
Lucchi, Pascal Fua, and Sabine Süsstrunk, “Slic superpixels
compared to state-of-the-art superpixel methods,” IEEE Trans.
on Pattern Analysis and Machine Intelligence, vol. 34, no. 11,
2012.
[5] Sergey Ioffe and Christian Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate
shift,” in ICML, 2015.

[16] Wenxian Yang, Jianfei Cai, Jianmin Zheng, and Jiebo Luo,“User-friendly interactive image segmentation through unified
combinatorial user inputs,” IEEE Transactions on Image Processing, vol. 19, no. 9, 2010.
[17] Di Lin, Jifeng Dai, Jiaya Jia, Kaiming He, and Jian Sun,
“Scribblesup: Scribble-supervised convolutional networks forsemantic segmentation,” in CVPR, 2016.
[18] Jun Zhu, Junhua Mao, and Alan L. Yuille, “Learning fromweakly supervised data by the expectation loss svm (e-svm)
algorithm,” in NIPS, 2014.
[19] Feng-Ju Chang, Yen-Yu Lin, and Kuang-Jui Hsu, “Multiple
structured-instance learning for semantic segmentation withuncertain training data,” in CVPR, 2014.
[20] Deepak Pathak, Philipp Krahenbuhl, and Trevor Darrell, “Constrained convolutional neural networks for weakly supervised
segmentation,” in ICCV, 2015.
[21] Niloufar Pourian, Sreejith Karthikeyan, and Bangalore S. Manjunath, “Weakly supervised graph based semantic segmentation by learning communities of image-parts,” in ICCV, 2015.

[6] Xavier Glorot and Yoshua Bengio, “Understanding the difficulty of training deep feedforward neural networks,” in AISTATS, 2010.

[22] Zhiyuan Shi, Yongxin Yang, Timothy Hospedales, and Tao Xiang, “Weakly-supervised image annotation and segmentation
with objects and attributes,” IEEE Trans. on Pattern Analysis
and Machine Intelligence, 2016.

[7] Pablo Arbelaez, Michael Maire, Charless Fowlkes, and Jitendra Malik, “Contour detection and hierarchical image segmentation,” IEEE Trans. on Pattern Analysis and Machine Intelligence, vol. 33, no. 5, 2011.

[23] Wataru Shimoda and Keiji Yanai, “Distinct class-specificsaliency maps for weakly supervised semantic segmentation,”in ECCV, 2016.

[8] David Martin, Charless Fowlkes, Doron Tal, and Jitendra Malik, “A database of human segmented natural images and its
application to evaluating segmentation algorithms and measuring ecological statistics,” in ICCV, 2001.
[9] Pedro F. Felzenszwalb and Daniel P. Huttenlocher, “Efficientgraph-based image segmentation,” Int. J. of Computer Vision,
vol. 59, no. 2, 2004.
[10] Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla,“Segnet: A deep convolutional encoder-decoder architecture
for image segmentation,” IEEE Trans. on Pattern Analysis andMachine Intelligence, 2017.
[11] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos,
Kevin Murphy, and Alan L. Yuille, “Semantic image segmentation with deep convolutional nets and fully connected crfs,”
in ICLR, 2015.
[12] Shuai Zheng, Sadeep Jayasumana, Bernardino RomeraParedes, Vibhav Vineet, Zhizhong Su, Dalong Du, Chang
Huang, and Philip H. S. Torr, “Conditional random fields asrecurrent neural networks,” in ICCV, 2015.
[13] Joseph Tighe and Svetlana Lazebnik, “Finding things: Image
parsing with regions and per-exemplar detectors,” in CVPR,
2013.
[14] Bharath Hariharan, Pablo Arbeláez, Ross Girshick, and Jitendra Malik, “Simultaneous detection and segmentation,” in
ECCV, 2014.
[15] Jifeng Dai, Kaiming He, and Jian Sun, “Instance-aware semantic segmentation via multi-task network cascades,” in CVPR,
2016.

[24] Honglak Lee, Peter Pham, Yan Largman, and Andrew Y. Ng,“Unsupervised feature learning for audio classification using
convolutional deep belief networks,” in NIPS, 2009.
[25] Quoc V. Le, “Building high-level features using large scale
unsupervised learning,” in ICASSP, 2013.
[26] Honglak Lee, Roger Grosse, Rajesh Ranganath, and Andrew Y.
Ng, “Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations,” in ICML, 2009.
[27] Mircea Cimpoi, Subhransu Maji, and Andrea Vedaldi, “Deepfilter banks for texture recognition and segmentation,” in
CVPR, 2015.
[28] Bharath Hariharan, Pablo Arbeláez, Ross Girshick, and Jitendra Malik, “Hypercolumns for object segmentation and finegrained localization,” in CVPR, 2015.
[29] Junyuan Xie, Ross Girshick, and Ali Farhadi, “Unsupervised
deep embedding for clustering analysis,” in ICML, 2016.
[30] Linli Xu, James Neufeld, Bryce Larson, and Dale Schuurmans,“Maximum margin clustering,” in NIPS, 2005.
[31] Francis R. Bach and Zaïd Harchaoui, “Diffrac: a discriminative and flexible framework for clustering,” in NIPS, 2008.
[32] Armand Joulin, Francis Bach, and Jean Ponce, “Discriminative
clustering for image co-segmentation,” in CVPR, 2010.