Journal of Robotics, Networking and Artificial Life, Vol. 3, No. 1 (June 2016), 9-12

Feature Acquisition From Facial Expression Image Using Convolutional Neural Networks
Taiki Nishime
Graduate school of Information Engineering, University of Ryukyus
Nishihara, Okinawa, Japan
Satoshi Endo, Koji Yamada, Naruaki Toma, Yuhei Akamine
School of Information Engineering, University of Ryukyus
Nishihara, Okinawa, Japantaiki_one@eva.ie.u-ryukyu.ac.jp, endo@ie.u-ryukyu.ac.jp, koji@ie.u-ryukyu.ac.jp
tnal@ie.u-ryukyu.ac.jp, yuhei@ie.u-ryukyu.ac.jp

Abstract
In this study, we carried out the facial expression recognition from facial expression dataset using Convolutional
Neural Networks (CNN). In addition, we analyzed intermediate outputs of CNN. As a result, we have obtained aemotion recognition score of about 58%; two emotions (Happiness, Surprise) recognition score was about 70%. We
also confirmed that specific unit of intermediate layer have learned the feature about Happiness. This paper details
these experiments and investigations regarding the influence of CNN learning from facial expression.
Keywords: facial expression recognition, convolutional neural networks, deep learning, feature learning

1. Introduction
Facial expression recognition is important to non verbal
communications among the people. Now, opportunities tocommunicate using either voice or text is increasing because of developing mobile phones and Internets. Thus, it
is considered that communication method via some devises has increased than face to face communication.
“UNMASKING THE FACE” by Paul Ekman and W.V.
Friesen [1] described that facial expressions have a closely connection with the emotions. For this reason, It isnatural to think that we can recognized your happiness if
you smiling. Mainstream approaches in facial expression
recognition use Facial Action Coding System (FACS)
labels. FACS was designed to help facial expression
recognition with resolve each expression into several

Action Units (AUs). FACS labels approaches need tolearn from FACS manual and test. As of now, FACS label
can only be given by experts or trained individuals. As aresults, not everybody using easily FACS labels to facial
expression recognition.
The previous studies on facial expression recognition
can be classified into two categories: FACS based method[2] or feature learning method [3]. In FACS based method, they first extracted feature from AUs, then they recognized facial expression from facial images using athese extracted feature and Support Vector Machine. Incontrast to feature learning method [3], recognized facial
expression using Convolutional Neural Networks (CNN)[4].

Published by Atlantis Press
Copyright: the authors
9

Taiki Nishime, Satoshi Endo, Koji Yamada, Naruaki Toma, Yuhei Akamine

Fig 1. CNN structure: consist of three convolutional layers, three pooling layers, two fully connected layers, and output layer

But, this study is not discussed in CNN model that has
finished learning, and there was no argument about learning the feature of CNN.
In this study, firstly, we carried out facial expression
recognition using CNN. In addition, we analyze the intermediate outputs of CNN that obtained from facial
expression images.
2. Convolutional Neural Networks
Convolutional Neural Networks is a type of feed-forward
artificial neural networks that consist of convolutional
layers, pooling layers, fully connected layers and outputlayer. Convolutional layers compute product-sum ofimage and weight. Pooling layers compute the max valueof a particular feature over a region of the image. Theseconvolutional layer and pooling layer were repeated forevery such layer. Fully connected layers applied at theend of or after repeated each layer. Fully connected layers
is the same as regular multilayer perceptron. By propagating these each layer, CNN was feature extracted frominput images.
3. Experiments
3.1. FER-2013 Dataset
We have selected Facial Expression Recognition 2013
(FER-2013) dataset [6]. FER-2013 was created by Pierre
Luc Carrier. This dataset was created using the Googleimage search API to search for images of faces that matcha set of 184 emotion-related keywords like “blissful”,“enraged” etc. Each images included dataset is cropped
around a face, and cropped images were then resized to48x48 pixels and converted to grayscale. Table. 1. present
the details of the dataset. Facial expression We focusedon are Angry(An), Disgust(Di), Fear(Fe), Happiness(Ha),Sadness(Sa), Surprise(Su) and Neutral(Ne).

Table 1. Detail of FER-2013 dataset

3.2. Preprocessing
We preprocess the data using Global Contrast Normalization (GCN) and ZCA whitening [5]. In GCN, subtract bymean and divide by dispersion for each dataset images.
By GCN preprocessing, if the value range of the input isnormalized from -2 to 2, and can be aligned to that range,even if there is a different axis scales. Natural image ischaracterized by strong correlation with neighboring
pixels. ZCA whitening has function to erase such correlation.
3.3. CNN settings
Fig 1. shows CNN model that used in this experiment.
Arrows in Fig 1. show weights, and number of undereach boxes show unit number. The number of input units
setting to same as number of input image pixels. The
number of output unit setting to same as number facial
expression classes. As the facial expression recognition
result, using the maximum value in output layer units.
3.4. Result
The results of this experiments shown in Table. 2. We
have obtained an recognition score 57.02%; Happinessand Surprise facial expression recognition score wasabout 70%. In contrast, Fear, Sad and Neutral score wasbelow 52%. Also these recognitions from only image data
is seemed to be difficult. We have obtained an Disgust
recognition score 0% because of Disgust data was lessthen other facial expression data.

Table 2. Confusion matrix of facial expression recognition

Published by Atlantis Press
Copyright: the authors
10Feature Acquisition From FacialFig 2. Visualized the 25-dimensional intermediate output
of recognized Happiness. The 25-dimensional output wascompressed to 2-dimension output by Principal Components Analysis. Plotting first and third principal components. 48.959% is contribution rate, it is show that48.959% information remained by PCA. Labeled by kmeans.

mediate outputs of cluster 6 was bigger than cluster 5. As
far as we have confirmed detail of outputs in cluster 6, alldata was Happiness face, and is commonly mouth wasopen, as shown in Fig 4. On the other hand, cluster 5
included Happiness face both opened mouth and closed
mouth. Most of cluster 5 data was Happiness face withclosed mouth. We focus on the intermediate output ofeach unit in data of cluster 6 (Fig 3), for example, it canbe seen that the output value of 21th unit is large. Also inFig 4, the Happiness face with opened mouth and closed
mouth were difference in 21th unit output value. In Fig 4,
it is selected some example from cluster 5 and cluster 6.
It was also observed that this result of 21th unit output
value was the same tendency as Happiness face withopened mouth in other cluster. In these results, the 21thunit output have closure connected in shape of mouth
open. We conclude that CNN have learned the feature of
facial expression about mouth.

3.5. Analyzing the intermediate outputs
As mentioned before, CNN was extracted feature ateach layers, and these result of feature extraction wasinfluence on the recognition. Thus, it seems that recognition accuracy is higher, feature extraction is better result.
From this reason, we focused on Happiness, and analyzing the intermediate output values. By this analyzing, wereveal that extracted feature from Happiness data.
In this experiment, there are image that recognized 682
Happiness images in 895 labeled images. We are focusedon the Happiness images and intermediate output values
previous layer for the output layer. Result of compressedintermediate output to two dimension is shown in Fig 2.
From Fig 2, It can be seen that some clusters are overlapped with each other like cluster 1 and cluster 10, but
independent cluster like clusters 5 and cluster 6 is alsoexists. As a result, we investigated the difference in the
distinguishable clusters 5 and cluster 6, we examine theextracted features from the result of this investigation.
Examination method is simple, we checked that whether
the intermediate outputs connoting a pattern.
All of 25 units intermediate output of cluster 5 and cluster 6 are shown in Fig 3, and also examples of image andintermediate outputs include in cluster 5 and cluster 6 areshown in Fig 4. As a results, It can be seen that the inter-

Published by Atlantis Press
Copyright: the authors
11

Taiki Nishime, Satoshi Endo, Koji Yamada, Naruaki Toma, Yuhei Akamine

References

3.6. Future work
In this paper, We carried out the emotion recognition
from facial expression image using a CNN. As a result,
we have obtained an average emotion recognition scoreof 58%; two emotions (Happiness, Surprise) recognition
score was about 70%. From the result of the analyzing
intermediate outputs of CNN, we confirmed specificintermediate outputs was likely to be reflected in shape ofmouth. In this experiments, though using the simple
method that analyze the intermediate outputs pattern, it isconsidered to effective method that analyzing the extracted feature from input data. In the future work, we will try
to analyze larger intermediate outputs than in this experiment. And we revealed that method of feature extractedof deep layer.

[1] Paul Ekman, W.V. Frisen, “UNMASKING THE FACE”(1975)[2] Hiroki NOMIYA, Teruhisa HOCHIN, “Facial ExpressionRecognition using Feature Extraction based on Estimation of Useful Features” (2011)[3] VICTOR-EMIL NEAGOE, ANDREI-PETRU BRAR,
NICUSEBE, PAUL ROBITU, “A Deep Learning Approach for Subject Independent Emotion Recognition
from Facial Expressions”, Recent Advances in Image,
Audio and Signal Processing, pp.93-98 (2013)[4] “Convolutional Neural Networks (LeNet) – DeepLearning 0.1 documentation”, LISA Lab. (2013)[5] Alex Krizhevsky, “Learning Multiple Layers of Featuresfrom Tiny Images”, 2009.
[6] Goodfellow, Ian J, .et al. “Challenges in Representation
Learning: A report on three machine learning contest”
Neural Information Processing. Springer Berlin Heidelberg, 2013.

Fig 4. Example of All of 25 units intermediate
output of cluster 5 and 6.

Published by Atlantis Press
Copyright: the authors
12