Person-invariant Recognition of Subtle Smiles using
Selected Improved LBP Features
Kosuke SASAKI and Manabu HASHIMOTO

Noriko NAGATA

Graduate School of Computer and Cognitive Sciences
Chukyo University
Aichi, Japan
Email: {sasaki, mana}@isl.sist.chukyo-u.ac.jp

Graduate School of Science and Technology
Kwansei Gakuin University
Hyogo, Japan
Email: nagata@kwansei.ac.jp

Abstract—This paper describes a person-invariant method forclassifying subtle smile. The system uses keypoints detected byusing a face tracking tool called Face Tracker. Our method
selects features strategically by calculating their humandependency rate. Its classification accuracy was evaluated using
the facial images of unlearned people. The results showed theaverage F-measure was 0.87 for neutral (expressionless) facial
images, 0.78 for subtle smile images, and 0.87 for exaggerated
smile images. Also, person-invariant accuracy was evaluated byusing the F-measure frequency of unlearned people. The results
revealed that the proposed method has higher person-invariant
accuracy than the previously reported methods.

problem as Matsuhisa and Hashimoto’s method because theplacement of the facial parts is different for each individual.
Local binary pattern features has person-independent
characteristics [3]. In particular, Faisal et al. [9] proposed aperson-independent method to recognize facial expressions byusing compound local binary patterns. However, this method
cannot recognize subtle facial expressions because it selects
only features that change significantly. Thus, previous methods
have two problems. First, they cannot recognize subtle facial
expressions. Second, identification performance is dependenton using learned people as subjects. In particular, we focus onsmile in facial expressions because the smile is closely relatedwith positive emotion such as happiness and satisfaction. To
address these problems, we propose a person-independent
method for classifying smile intensity. In the proposed method,
keypoints are extracted by using a face tracking tool called
Face Tracker [10]. Improved local binary pattern (LBP)
features [11] are calculated by comparing the pixel values of a
keypoint with the pixel values around the keypoint. Personinvariant features are selected by using the human dependency
rates of calculated features. Input facial images are classified
by using weights as human dependency rates. These factors
enable the proposed method to recognize subtle smile ofunlearned people.

Keywords—facial keypoints; person-invariant recognition;
Improved LBPI.

INTRODUCTION

Recently, many people have become interested in
improving the quality of human life. Because people arealways looking for ways to improve their quality of life,
product makers and service providers desire a system that canestimate human emotions so they can develop products andservices towards with customers will feel positively. One study
for estimating human emotions reported a method using anelectroencephalogram (EEG) as a contact sensor [1]. However,the sensor is uncomfortable to wear, and this disturbs theexpression of natural emotion. To avoid such disturbance,
human emotion needs to be estimated in a natural situation.
Since facial expressions are closely related to emotions, many
researchers have studied facial expressions by acquiring themby using a non-contact sensor such as an RGB (red, green, andblue) camera. Two types of methods for classifying facial
expressions have previously been reported. One uses local
facial features such as HoG [2] and Gabor wavelets [4]. The
other uses a small set of keypoints detected from parts of theface [5] [6]. These methods can classify many facial
expressions but cannot recognize very subtle ones.
Consequently, Matsuhisa and Hashimoto [7] proposed amethod to recognize subtle smile by using Gabor filters and the
AdaBoost algorithm. Gabor filters have a response value that issensitive to subtle changes in smile. However, they havedifficulty classifying smile of unlearned people because theresponse value represents a 3D facial shape that is different foreach person. Also, Nomiya and Hochin [8] proposed a method
to recognize subtle facial expressions by using the geometricfeatures of keypoints. However, this method has the sameII.

PROPOSED METHOD FOR CLASSIFYING SMILE
INTENSITY

A. Overview
Figure 1 shows a block diagram of the method we propose
using to classify smile intensity.

Fig. 1. A block diagram of the proposed method.

The proposed method comprises two steps, the first forlearning and the second for classifying. In the first step, facial
keypoints from a local position such as a corners of the eyes ora mouth corner are detected by using a constrained local model(CLM) method that is versatile to humans. Figure 2 shows
detected keypoints obtained from facial images.

B. Improved Local Binary Patterns[11]The LBP features express the spatial structures of the local
textures of gray scale images by thresholding pixel
neighborhoods with the center values. Because these features
capture the brightness difference between focused pixels andthe area around the pixels, they can capture the presence orabsence of brightness around keypoint edges. However, theycannot capture subtle edges. Thus, improved LBP features
were previously reported by Hongliang et al. [11]. These
improved LBP features can capture subtle edges better thanordinary LBP features can. Figure 3 shows the flow ofcomputing ordinary LBP features and the improved LBP
features.

Fig. 2. Detected keypoints from facial images.

This method detects 66 keypoints in all. However, the 17
keypoints obtained for the facial outline are unusable as
background information, so we used only 49 keypoints. In
addition, the size and rotation of a facial image must be
corrected to calculate the features stably. The selected
keypoints are corrected by using the relationships among
keypoints such as the horizontality of the centers of both eyes
and the center of the gravity point that comprises the centers ofboth eyes and the upper lip. Improved LBP features that are
represented as 0 or 1 bits are calculated by comparing the
keypoint pixel values with the values around the keypoint. The
improved LBP features can capture the presence or absence ofedges better than ordinary LBP features. Therefore, they cancapture the presence or absence of facial changes. The features
are person-invariant because the magnitude of pixel value
variation is independent. In addition, the regions around thefacial keypoints are slightly binary. The human dependency
rates are calculated in each region; our method selects theregion having the lowest dependency rate. Bit streams are
generated in smile intensity using the selected regions. In
addition, features are selected for evaluating identities on thebasis of smile intensity. The most effective bits for classifying
smile intensity are selected from bit streams having low humandependency rates. The selected features are utilized to classify
smile intensity on the basis of two requirements. First, thefeatures must be human-independent. Second, they must be
effective for classifying smile intensity. In second step, thekeypoints are detected from input images in the same way as infirst step. The improved LBP features are calculated in each
keypoint. Further, the bit streams are generated by using theregions selected in first step. Our method classifies the smile
intensity of input images by computing the Hamming distance
between the bit streams of first step and second step.

Fig. 3. Computing LBP features and improved LBP features.

The improved LBP features use the average ofneighborhood pixel values. Thus, we can confirm that thesefeatures have a local structure, which is obtained successfully
as shown in Figure 3. Our method uses not only a 3x3
neighborhood region but also 5x5, 7x7, 9x9 and 11x11
neighborhood regions. Thereby, with it we can capture not only
subtle but also substantial facial changes.
C. Feature selection based on human dependency rate andfacial expression identities
Parts of the improved LBP features have personal
characteristics such as wrinkles and sculpted features. We
calculated the human dependency rates from bit frequencies ina learning dataset. Figure 4 shows the flow of calculating
human dependency rates.
Our method focuses on the neighborhood regions aroundkeypoints. We calculated the frequency values of binary bits of
the regions that were focused on in the learning dataset; thedependency rate is an inverse value that is the difference
between frequency values of 0 and 1. The person-independent
regions are selected by thresholding this dependency rate.
Moreover, the selected regions are given to the dependency
rate as weights. The regions are also selected on the basis of
facial expression identities. Our method selects effective
regions to classify facial expressions in each class (neutral faceFig. 4.

Flow of calculating dependency rate on human.

and subtle smile, subtle smile and exaggerated smile, neutral
face and exaggerated). These regions must differ slightly fromone facial expression to another. On this basis, the bit streams
that utilize facial expression classification are selected on thebasis of two requirements. First, they must have low humandependency rate. Second, they must be identical for each facial
expression.
D. Flow of Smile Intensity Classification
Figure 5 shows flow of smile intensity classification.

Fig. 5. Flow of smile intensity classification.

The proposed method detects facial keypoints from input
images in the same way as learning images and calculates
improved LBP features from keypoints of the input images
selected by first step. The bit streams are generated from bits ofthe same regions as those selected in first step. Input images
are classified by using the Hamming distance between the bit
streams of input images and the bit streams generated in firststep. This Hamming distance is calculated by using weightsthat consist of bits having human dependency rates. Because
there are three classifiers, our method classified the input
images by the voting of the classifiers.
III.

Fig. 6. Example images of each facial expression.

B. Smile intensity classification performance for unlearned
peopleBy using a cross-validation approach, a classifier wastrained by 27 people, and unlearned people were classified by it.
Using 30 images for each facial expression, we also compared
our method with those reported elsewhere [7] [8] [9] [12].
Faisal et al. [9] reported a facial expression classification
method that considered person independence. Matsuhisa andHashimoto [7] and Nomiya and Hochin [8] reported arecognition method for subtle facial expressions. Bradley andAdam [12] proposed EmotionNet that is a method ofConvolutional Neural Networks (CNN) to emotion recognitionin facial image. We utilized a learning CNN model thatgenerated by the 15000 iterations. Figure 7 shows the Fmeasure frequency of unlearned people.
This graph shows the classification accuracy becomesperson-independent as the distribution is concentrated on theright side. We confirmed that the results of the proposed
method were distributed more to the right than those ofprevious methods. This shows that the proposed method ismore person-invariant than the previous methods. In addition,
the median value of frequency is calculated for each facial
expression. Table 1 shows the median value of frequency foreach facial expression.

EXPERIMENTAL RESULTS

A. Dataset
We videoed 28 men and women watching a comedy
program and used the captured images in an experiment. Fiveother people watched the taken video at the normal frame rate
and typed in the ground truth to the video. They pressed 1, 2, or
3 when a someone made a neutral face, a subtle smile, or an
exaggerated smile, respectively. We used the images given thesame ground truth by at least four of the five people who
watched the video. Figure 6 shows example images of eachfacial expression in the experiment.

TABLE I.

MEDIAN VALUE OF FREQUENCY FOR EACH FACIAL
EXPRESSION
The median value of frequency in Fig.7.
NeutralSubtleExaggerated

Proposed method

0.9

0.8

1.0

EmotionNet [12]

0.8

0.5

0.7

Geometric feature [8]

0.9

0.6

1.0

Gabor filter [7]

0.6

0.4

0.6

CLBP [9]

0.5

0.6

0.7

exaggerated smile images. This confirmed that our method
provides higher classification performance than previous
methods.
IV.

CONCLUSION

In this paper, we proposed the idea for achieving personinvariant classification of smile intensity. The idea was toselect features by taking person invariance and facial
expression identities into account. Person-invariant accuracy
was evaluated by using the F-measure frequency values of
unlearned people. For unlearned people, the values were 0.87
for neutral (expressionless) facial images, 0.78 for subtle smile
images, and 0.87 for exaggerated smile images. We also foundthat these values were distributed at higher frequencies than theF-measures of previous methods. These results lead us toconclude that the proposed method can accurately recognize
subtle smile and is person-independent.
ACKNOWLEDGMENT
This research is partially supported by the Center of
Innovation Program from Japan Science and Technology
Agency, JST.
REFERENCES[1]

Fig. 7.

F-measure frequency of unlearned people.

Our method provides similar or higher personindependence than previous method in neutral face andexaggerated smile. Our method provides higher personindependence than previous method in subtle smile.
We evaluated the classification accuracy for each facial
expression. Table 2 shows the average F-measure for eachfacial expression.
TABLE II.

F-MEASURE FOR EACH FACIAL EXPRESSION
F-measure value
NeutralSubtleExaggerated

Proposed method

0.87

0.78

0.87

EmotionNet [12]

0.68

0.57

0.57

Geometric feature [8]

0.85

0.57

0.85

Gabor filter [7]

0.58

0.43

0.61

CLBP [9]

0.51

0.56

0.63

The results showed the average F-measure was 0.87 forneutral images, 0.78 for subtle smile images, and 0.87 for

C.A. Kothe, S. Makeig, and J.A. Onton, Emotion Recognition from EEG
During Self-Paced Emotional Imagery, in Humaine Association
Conference on Affective Computing and Intelligent Interaction(ACII2013), pp. 855-858, 2013.
[2] Junkai Chen, Zenghai Chen, Zheru Chi, and Hong Fu, Facial Expression
Recognition Based on Facial Components Detection and HOG Features,
Scientific Cooperations International Workshops on Electrical andComputer Engineering Subfields, pp. 64-69, 2014.
[3] C. Shan, S. Gong, and P. McOwan, Facial Expression Recognition
Based on Local Binary Patterns: A Comprehensive Study, Image andVision Computing, vol. 27, no. 6, pp. 803-816, 2009.
[4] E. Owusu, Y. Zhan, and Q.R. Mao, A neural-AdaBoost based facial
expression recognition system, Expect Syst. Appl. 41(7), pp. 3383-3390,
2014.
[5] A. Majumder, L. Behera, and V. K. Subramanian, Emotion recognition
from geometric facial features using self-organizing map, Pattern
Recognition. 47(3), pp. 1282-1293, 2014.
[6] I. Kotsia, and I. Pitas, Facial expression recognition in image sequencesusing geometric deformation features and support vector machines,
IEEE Trans. Image Process., vol. 16, pp. 172–187, 2007.
[7] H. Matsuhisa and M. Hashimoto, Identifying Subtle Facial Expression
Changes using Optimized Gabor Features, The Journal of the Institute of
Image Information and Television Engineers, vol. 68, no. 6, pp. J252J255, 2014. (In Japanese)[8] H. Nomiya and T. Hochin, Efficient Emotional Video Scene Detection
Based on Ensemble Learning, The IEICE Transactions on Informationand Systems, vol. J95-D, no. 2, pp. 193-205, 2012. (In Japanese)[9] Faisal Ahmed, Hossain Bari, and Emam Hossain, Person-Independent
Facial Expression Recognition Based on Compound Local Binary
Pattern (CLBP), IAJIT, Vol.11, No, 2, pp.195-203, 2014.
[10] J. Saragih, S. Lucey, and J. Cohn, Deformable Model Fitting byRegularized Landmark Mean-Shifts, International Journal of Computer
Vision, vol. 91, no. 1, pp. 200-215, 2011.
[11] Hongliang Jin, Qingshan Liu, Hanqing Lu, and Xiaofeng Tong, Face
Detection Using Improved LBP Under Bayesian Framework, In
Proceedings of Third International Conference on Image and Graphics,
pp. 306–309, 2004.
[12] Bradley Kennedy and Adam Balint, EmotionNet, github:https://github.com/co60ca/EmotionNet