IPSJ Transactions on Computer Vision and Applications Vol.7 99–103 (July 2015)[DOI: 10.2197/ipsjtcva.7.99]

Express Paper

Cost-alleviative Learning for Deep Convolutional Neural
Network-based Facial Part Labeling
Takayoshi Yamashita1,a) Takaya Nakamura1 Hiroshi Fukui1,b)
Yuji Yamauchi1,c) Hironobu Fujiyoshi1,d)Received: March 13, 2015, Accepted: August 20, 2015, Released: July 27, 2015

Abstract: Facial part labeling which is parsing semantic components enables high-level facial image analysis, andcontributes greatly to face recognition, expression recognition, animation, and synthesis. In this paper, we propose acost-alleviative learning method that uses a weighted cost function to improve the performance of certain classes during
facial part labeling. As the conventional cost function handles the error in all classes equally, the error in a class with aslightly biased prior probability tends not to be propagated. The weighted cost function enables the training coeﬃcientfor each class to be adjusted. In addition, the boundaries of each class may be recognized after fewer iterations, which
will improve the performance. In facial part labeling, the recognition performance of the eye class can be significantly
improved using cost-alleviative learning.
Keywords: facial part labeling, cost-alleviative learning, convolutional neural network, cost-function, back propagation

1. Introduction
Facial part labeling involves parsing semantic components ofthe face such as the mouth, nose, and eyes. This labeling enables high-level facial image analysis, and contributes greatly toface recognition, expression recognition, animation, and synthesis. Facial point detection and face alignment are major components in identifying facial parts. Existing research focuses on
detecting landmarks such as eye and mouth corners, and can generally be categorized into discriminative and regression methods [1], [2], [8], [10] or graphical model methods [7], [14]. Theseapproaches tend to refine each landmark from the initially estimated location. Whereas facial point detection and face alignment obtain facial landmarks with high accuracy, post-processingor human interaction is required to label each pixel with the nameof that facial part. In addition, it is hard to define how many landmark positions are necessary. To address this problem, semanticsegmentation approaches have been proposed [3], [9], [11]. Deep
Learning achieves state-of-the-art performance for the labeling offacial parts [9], [11] and scenes [3]. In particular, the Deep Convolutional Neural Network (DCNN) has attracted attention for
applications in computer vision tasks [5], [6]. However, DCNNbased methods cannot label eye regions correctly, as shown inFig. 1. The DCNN trains the network parameters by iteratively
updating according to the error between prediction and groundtruth. If the prior probabilities are biased to a particular class,1a)b)c)d)

Chubu University, Kasugai, Aichi 487–8501, Japanyamashita@cs.chubu.ac.jp
fhiro@vision.cs.chubu.ac.jp
yuu@vision.cs.chubu.ac.jp
hf@cs.chubu.ac.jp

c 2015 Information Processing Society of Japanthe error in classes with lower probabilities may not be propagated correctly. In Ref. [3], superpixel-based segmentation wasused to reduce the bias and ensure that errors were propagatedappropriately. However, the accuracy is significantly aﬀected by
the superpixel segmentation. A diﬀerent approach uses a Deep
Belief Network to detect and label all facial part regions at thesame time [9]. Although this method avoids the problems of superpixel segmentation by detecting a larger region, it does require
a complex architecture. These techniques can achieve good performance, but do not solve the training problem for the labelingof small regions.
In this paper, we consider this problem and propose a training
approach to improve the accuracy of the labeling of small regions.
As the conventional cost function handles the error in all classes
equally, the error in a class with a slightly biased prior probability

Fig. 1 Facial parts labeling based on conventional DCNN. (a) and (d) are
input images, (b) and (e) are the resulting images, (c) and (f) are theground truth.

99

IPSJ Transactions on Computer Vision and Applications Vol.7 99–103 (July 2015)

tends not to be propagated. We propose a cost-alleviative learning
method that weights the cost function using the prior probabilityof each class. We describe the details of cost-alleviative learning
in Section 3. We show that the cost weight can control the accuracy and convergence speed of certain classes, and compare manually set probabilities with those based on the ratio of facial part
regions in an image. In Section 4, we demonstrate the superiorityof our cost-alleviative learning over the conventional method.

2. Related Work
Conventional applications of facial analysis employ facial
point detection or face alignment as preprocessing techniques tonormalize the facial region. The Active Shape Model (ASM) [1]
is a well-known face alignment approach, and has inspired manymethods [10], [13]. ASM depends on the initial position of themodel, and does not work well in an unconstrained environment. The Markov Shape Model [7] uses multiple initial shapes
as local line segments and appearances. While these methods
reduce the influence of initialization, they can be time consuming. Discriminative approaches adapt to pose variations and cluttered backgrounds, and generally oﬀer reduced computational
cost [2], [8], [14]. The regression ASM detects small patches
related to facial components by boosting the classifiers [2]. As
the detection process for each patch is independent, the matchingprocess may fail if there are several missing components.
Representations that use pixel-wise label maps provide richer
information and more robustness than facial point detection andface alignment [12], [15]. For example, the scene parsing approach in Ref. [15] constructs a topological structure model ofthe face image that is robust in unconstrained environments. As
the model is based on rough priors, it is inaccurate for small
face components such as the eyes. A hierarchical approach toface parsing can improve the accuracy of these small components [9]. First, face parts (e.g., upper face, lower face) aredetected, before a second stage detects face components (e.g.,mouth, eyes) and a final, third stage segments the components
according to pixel-wise labels. This hierarchical structure is constructed by heuristic knowledge, and is not easily generalized.
DCNN-based approaches have been successfully applied to scenelabeling tasks [3]. This method also employs a hierarchical structure that segments and labels superpixels.
For recognition tasks, DCNN updates the network parameters
according to the error obtained from several images. Althoughparticular classes may be biased in each iteration, the update process uses equal prior probabilities for all classes over all itera-tions. On the other hand, for semantic segmentation, the errors
for all pixels are calculated and propagated at the same time. In
particular, in the case of facial part labeling, the prior probabilities of each facial part will be diﬀerent. Whereas a class with ahigh prior probability is likely to propagate the error to the next
step, one with a low probability will find convergence diﬃcult,with a large error that does not occur in the next step. As a result,the performance for this class will be significantly degraded.
In this paper, we propose a cost-alleviative learning techniquethat introduces a new cost function to improve specific classes inthe semantic segmentation task. The cost function is weighted
based on the prior probability of each class. These cost weights
make it possible to alleviate the propagation of error. We now
describe the cost function and the eﬃciency of the cost weights
in detail.

3. Proposed Method
Figure 2 shows the network structure of DCNN for semanticsegmentation. The network is built of three types of layer: convolutional, pooling, and a full connection layer. The convolution
and pooling layers are arranged successively as a deep structure,and are followed by the full connection layer. The output layer
has label maps for each class. In the training phase, the errors
are calculated as the diﬀerence between the response value andlabeled data. The update values are derived from the errors, andare propagated by back-propagation. The error Ec of output unit
c is defined in Eq. (1).
t p log y p(1)
Ec = −p∈PNote that P is the mini-batch size, t p are the labeled data, and y p
is the prediction result. The total training error on one iterationis computed by adding the errors in all classes. The conventional
cost function treats these errors equally. We set the cost weightvc for each class c, and define the total training error E M as theweighted sum in Eq. (2).
Ec · vc
(2)
EM =c∈C

The network parameters W on the t-th iteration are updated using
the gradient of the cost function with a training coeﬃcient η, asshown in Eq. (3).
Wt+1 = Wt − η∂E M∂Wt(3)

From Eq. (2) and our definition that the cost function E M is theFig. 2 Structure of DCNN for segmentation. The network consists of successive convolution and poolinglayers followed by a full connection layer. The output layer has label maps for each class.

c 2015 Information Processing Society of Japan


100

IPSJ Transactions on Computer Vision and Applications Vol.7 99–103 (July 2015)

Table 1 The ratio of each face component in an image.
class
faceeyesnosemoutheye glasses
hathairothers

Table 2Ratio[%]
17.51
0.441.08
0.89
0.06
0.90
10.73
68.34

Network structure and training parameters.

layerinput
1st convolution

2nd convolution

sum of the errors across all classes, Eq. (3) can be written as
Eq. (4).
∂∂
E0 · v0 +E1 · v1
Wt+1 = Wt − η
∂Wt
∂Wt

∂
(4)
+ ··· +EC · vC∂Wt

3rd convolution

4th full connection
outputtraining parameters

gray# of filters (size)
activation function
pooling (size)# of filters (size)
activation function
pooling (size)# of filters (size)
activation function
pooling (size)# of units
activation function
dropout# of units
batch size
learning rate

100 × 100
32 (7 × 7)
maxoutmaxpooling (2 × 2)
64 (6 × 6)
maxoutmaxpooling (2 × 2)
128 (6 × 6)
maxoutmaxpooling (2 × 2)
1,000
sigmoid
0.5
8 × 100 × 100
100.1

As shown in Eq. (4), the gradient of the cost function is the sumof the error gradients in all classes, and the errors in each classare independent. The product of the gradient of the cost function
and the training coeﬃcient η can be considered as the productof the error gradient for each class and the training coeﬃcient.
Therefore, Eq. (4) becomes:
∂E0∂E1+ η · v1 ·Wt+1 = Wt − η · v0 ·
∂Wt
∂Wt

∂EC(5)+ · · · + η · vC ·∂Wt
Both the cost weight of each class vc and the training coeﬃcientη are constants, but the cost weights are generally diﬀerent foreach class. As a result, the weighted cost function is equivalent
to changing the coeﬃcient for each class.
Table 1 gives the ratio of each class in the Labeled Face in theWild (LFW) dataset. face and hair accounts for 17% and 10% of
the whole image set, respectively, whereas the ratio of nose andeye regions are small (1.0% and 0.4%, respectively). The costweight of each class is defined by Eq. (6) using the prior probability p(c) computed from the ratios in Table 1.
C
vc =C − log p(c)(6)

Note that C is the number of classes. The network parameters areupdated according to the weighted cost function, and the convergence speed of a certain class is adjusted by its cost weight.

4. Experiments
We now demonstrate the performance of the proposed costalleviative learning with the weighted cost function. Although
the LFW dataset has pixel-wise labeled data, there are only threeclasses, namely face, hair, and others. To prepare classes withlow prior probabilities, we annotated new pixel-wise labels foreight classes, i.e., eyes, nose, mouth, sunglasses, hat, hair, face,and others. We split the dataset into 9,263 training images and3,970 test images. We augmented the training subset by applying translation, rotation, and scaling to the images, giving a totalof 185,262 images. Table 2 lists the evaluation network structure and training parameters. The network has three convolution
and pooling layers and one full connection layer with dropout.

c 2015 Information Processing Society of JapanFig. 3Labeling accuracy of each facial part.

We employed maxout [4] and sigmoid activation functions in theconvolution layer and full connection layer, respectively. The input images were 100 × 100 pixels in size, and the output units
were 100 × 100 × 8. We set the mini-batch size to 10 and applieda training coeﬃcient of 0.1.
4.1 Face Part Labeling with Cost-alleviative Learning
We compared the recognition performance of our method withthat of conventional DCNN. The cost function of our method
includes weights for each class. The cost weights of the eyes,mouth, nose, face, and hair were found to be 0.78, 0.8, 1.0, 1.05,
and 1.05, respectively. Figure 3 shows the recognition performance after 1 million iterations. The recognition rates for faceand hair are 98% and 78%, respectively. This is similar to theperformance of conventional DCNN. Whereas the DCNN recognition rates for eyes and mouth are 49% and 78%, our methodsignificantly improves these to 83% and 89% for each class. Our
approach also improves the average recognition rate by about 5%.
Figure 4 illustrates the recognition performance during thetraining phase to demonstrate the behavior of the proposed costfunction. The permanence of the face class does not influence
the cost weight of the eye class, and is quite similar to that in
the conventional method. The weight increases rapidly at first,and converges after about 100,000 iterations. In contrast, the eyeclass is not well recognized until the face region recognition hasconverged. The performance of the eye class increases gradually

101

IPSJ Transactions on Computer Vision and Applications Vol.7 99–103 (July 2015)

Fig. 4Recognition rate of face and eye classes on each iteration.

Fig. 6 Precision and recall curves of eye and face classes.

Fig. 7

Fig. 5 Recognition rate of each class at each cost weight.

after 100,000 iterations, but remains at about 50%. The weighted
cost function alleviates the error, and significantly improves theperformance in earlier iterations.
4.2 Evaluation of Cost Weight Values
We define the cost weight from the prior probability of eachclass in Eq. (6). We now demonstrate the recognition behavior
with manually set cost weights. We vary the cost weight of theeye class from 0.6 to 1.6, and train the network over 1 million iterations. Figure 5 shows the recognition performance of each class.
The recognition performance decreases when the cost weight ofthe eye class takes values larger than 1. However, the recognition performance is improved when the cost weight of the eyeclass is set at around 0.8, but gradually worsens with smaller costweights. Thus, a smaller cost weight for the eye class (around0.8) can improve the recognition performance. In Eq. (6), it ispossible to set the cost weight of eyes to 0.78, which is almost
the same as in this manual definition.

5. Discussion
Figure 6 shows the precision and recall curves of the eye andface classes. In addition, the results of face part labeling at each
iteration are shown in Fig. 7. In the conventional method, the eyeclass exhibits high precision, but a low recall rate. This indicatesthat the class occupies a small region and is correctly labeled, butthe labeled region is quite small. The eye class could not be labeled well because it is diﬃcult to determine the boundary. While
the recall rate gradually increases, many iterations are needed toobtain better performance. Our cost-alleviative learning achieves
high precision and recall rates in fewer iterations by applying ac 2015 Information Processing Society of JapanFacial labeling results after diﬀerent numbers of iterations. (a) is theinput image, (g) is the ground truth, (b)–(f) are given by the proposed
method and (h)–(l) are given by the conventional method. (b) and (h)are at 100,000 iterations, (c) and (i) are at 200,000 iterations, (d) and(j) are at 300,000 iterations, (e) and (k) are at 400,000 iterations, and(f) and (l) are at 500,000 iterations.

small cost weight for the eye class. As a result, it is easy to determine the boundaries, leading to a high precision rate. Moreover,
the recall rate converges after fewer iterations. The face region
retains high recall and precision rates, and does not influence thecost weight of the eye class. Thus, cost-alleviative learning enables an eﬃcient improvement in recognition performance andconvergence rate.

6. Conclusion
We have proposed a cost-alleviative learning method that uses
a weighted cost function to improve the recognition performanceof certain classes during semantic segmentation. The weighted
cost function enables the training coeﬃcient for each class to be
adjusted. This alleviates the error in classes that occupy small regions, making it possible to achieve convergence for such classes.
In addition, the boundaries of each class may be recognized after
fewer iterations, which will improve the performance. In facial
part labeling, the recognition performance of the eye class can besignificantly improved using cost-alleviative learning. In future
work, we will apply the weighted cost function to scene parsing
or other semantic segmentation tasks.
References[1][2][3]Cootes, T., Taylor, C. and Graham, J.: Active shape models theirtraining and application, Computer Vision and Image Understanding,Vol.61, No.1, pp.38–59 (1995).
Cristinacce, D. and Cootes, T.: Boosted regression active shape models, BMVC, pp.1–10 (2007).
Farabet, C., Couprie, C., Najman, L. and LeCun, Y.: Learning Hierarchical Features for Scene Labeling, PAMI, Vol.35, No.8, pp.1915–
1929 (2013).

102IPSJ Transactions on Computer Vision and Applications Vol.7 99–103 (July 2015)

[4]
[5]
[6]
[7]
[8]
[9]
[10]
[11]
[12]
[13]
[14]
[15]

Goodfellow, I.J., Farley, D.W., Mirza, M., Courville, A. and Bengio,Y.: Maxout Networks, ICML, pp.1319–1327 (2013).
Krizhevsky, A., Sutskever, I. and Hinton, G.E.: Imagenet classification with deep convolutional neural networks, NIPS, pp.1097–1105(2012).
LeCun, Y., Bottou, L., Bengio, Y. and Haﬀner, P.: Gradient-basedlearning applied to document recognition, IEEE, Vol.86, No.11,pp.2278–2324 (2003).
Liang, L., Wen, F., Xu, Y., Tang, X. and Shum, H.: Accurate facealignment using shape constrained Markov network, CVPR, pp.1313–1319 (2006).
Liang, L., Xiao, R., Wen, F. and Sun, J.: Face alignment via
component-based discriminative search, ECCV, pp.72–85 (2008).
Luo, P., Wang, X. and Tang, X.: Hierarchical Face Parsing via Deep
Learning, CVPR, pp.2480–2487 (2013).
Milborrow, S. and Nicolls, F.: Locating facial features with an extended active shape model, ECCV, pp.504–513 (2008).
Smith, B.M., Zhang, L., Brandt, J., Lin, Z. and Yang, J.: ExemplarBased Face Parsing, CVPR, pp.3484–3491 (2013).
Tu, Z., Chen, X., Yuille, A. and Zhu, S.: Image parsing: Unifying segmentation, detection and recognition, IJCV, Vol.63, No.2, pp.113–140(2005).
Zhou, Y., Zhang, W., Tang, X. and Shum, H.: A Bayesian mixture model for multi-view face alignment, CVPR, Vol.2, pp.741–746(2005).
Valstar, M., Martinez, B., Binefa, X. and Pantic, M.: Facial point detection using boosted regression and graph models, CVPR, pp.2729–2736 (2010).
Warrell, J. and Prince, S.: Labelfaces: Parsing facial features by multiclass labeling with an epitome prior, ICIP, pp.2481–2484 (2009).

(Communicated by Hitoshi Imaoka)

c 2015 Information Processing Society of Japan103