# A New Method of 3D Facial Expression Animation

Authors: Shuo Sun; Chunbao Ge
Journal: Journal of Applied Mathematics
 (2014)

---

## Abstract

Animating expressive facial animation is a very challenging topic within the graphics community. In this paper, we introduce a novel ERI (expression ratio image) driving framework based on SVR and MPEG-4 for automatic 3D facial expression animation. Through using the method of support vector regression (SVR), the framework can learn and forecast the regression relationship between the facial animation parameters (FAPs) and the parameters of expression ratio image. Firstly, we build a 3D face animation system driven by FAP. Secondly, through using the method of principle component analysis (PCA), we generate the parameter sets of eigen-ERI space, which will rebuild reasonable expression ratio image. Then we learn a model with the support vector regression mapping, and facial animation parameters can be synthesized quickly with the parameters of eigen-ERI. Finally, we implement our 3D face animation system driving by the result of FAP and it works effectively.


---

## Body

## 1. Introduction

Facial animation is one alternative for enabling natural human-computer interaction. Computer facial animation has applications in many fields. For example, in the entertainment industry, realistic virtual humans with facial expressions are increasingly used. In communication applications, interactive talking faces not only make the interaction between users and machines more fun, but also provide a friendly interface and help to attract users [1, 2]. Among the issues concerning the realism of synthesized facial animation, humanlike expression is critical. But how to analyze and comprehend humanlike expression is still a very challenging topic for the computer graphics community. Facial expression analysis and synthesis are an active and challenging research topic in computer vision, impacting important applications in areas such as human-computer interaction and data-driven animation. We introduce a novel MPEG-4 based 3D facial animation framework and the animation system driving by FAP that produced from camera videos. The MPEG-4 based framework has the advantages of currency and few data [3–5]. First the system takes 2D video input and recognizes the face area. Then the face image was transformed to ERI [6]. Next, a simple ERI’s parameterized method is adopted for generating an FAP driving model by the support vector regression and it is a statistic model based on MPEG-4. The results of FAPs can be used to drive the 3D face model defined by MPEG-4 standard.The remainder of the paper is organized as follows. We present the related work in Section2. We then describe how to preprocess video data in Section 3. Section 4 presents how to construct the eigen-ERI. Section 5 describes the extraction of the FAP. In Section 6 we propose a novel SVR-based FAP driving model. Finally, we show the experiments and conclude the paper in Sections 7 and 8.


## 2. Related Works

Recently, realistic facial animation has become one of the most important research topics of computer graphics. Many researchers focus on the nature of the facial expression. In [7, 8], a sample-based method is used to make photorealistic expression in detail. In particular, Chang et al. [9] implement a novel framework for automatic 3D facial expression analysis in video. Liu et al. proposed an ERI-based method to extract the texture depth information of the photo in [6], and Figure 2 shows the ERI of the frown expression. For reflecting the changes of a face surface, Tu et al. [10] use the gradients of the ratio value at each pixel in ratio images and apply it to 3D model. In addition, a parameterized ERI method was proposed by Tu et al. [10] and Jiang et al. [11].Zhu et al. [12] used a method of SVR-based facial texture driving for realistic expression synthesis. In Zhu’s work, a regression model was learned between ERI’s parameters and FAPs. According to the inputs of FAPs, the model will forecast the parameters of ERIs. Then a reasonable facial expression image was generated with ERIs. On the contrary, our method is to forecast FAPs from the parameters of the eigen-ERIs; furthermore we realize a 3D face animation system driving by FAPs.In this paper, we realize a 3D face animation system that can generate realistic facial animation with realistic expression details and can apply in different 3D model similar with human. The main problems of our facial animation system are the extraction of the ERI from the camera video and learning the SVR-based model, furthermore building an FAP driving 3D facial expression animation system. Next section will introduce these.


## 3. Video Data Preprocessing

### 3.1. Video Data Capture

For the robustness of the algorithm, we get video data in normal light environment, and video equipment is often used to support continuous capture PC digital camera. Then we set the sampling rate of 24 frames per second and the sampling resolution of 320∗240, and the total 1000 sampling frame was captured, of which 200 expressions defined key frame.In this paper, we adopted the method of the marked points in face to extract facial motion data, and the most difference with other methods is that our method is based on the MPEG-4 standard (Figure1(a)). The advantage of the standard is that you can share daters, and the data can be used with any standards-based grid. By marked points, we can get more accurate facial motion data, while blue circle chip can be pasted in any place, including eyebrows and eyelids, and have no effects on their movement, which makes it possible to get the whole facial motion data and satisfy with the processing of the next step.Mesh model of neutral expression.
(a)
AAM
(b)
Mesh triangleFigure 2
ERI of frown expression.In order to obtain experimental data, first we make a coarse positioning through the blue calibration point using face detection tools and have calibration and reduction transformation operations on the sample data. Then we design a face mesh model in Figure1 to describe the geometry of the facial animation. Mesh vertices were mapped automatically using active appearance model (AAM) and manual fine tuning to meet subsequent demand for data extraction.After obtaining the data of the texture and feature points, all texture will be aligned to the average model.


### 3.2. Features Points

According to MPEG-4, we defined six basic expressions types (see Figure3) and captured them from video camera. Twenty-four key frames demanded for training and one nonkey frame used for testing per expression type are extracted with resolution of 320∗240 pixels. In Figure 1(a), the 68 feature points are marked automatically with AAM [13]. In our experiment, we adopted the 32 feature points belonging to the FPs in MPEG-4.Figure 3
Six basic facial expressions.


## 3.1. Video Data Capture

For the robustness of the algorithm, we get video data in normal light environment, and video equipment is often used to support continuous capture PC digital camera. Then we set the sampling rate of 24 frames per second and the sampling resolution of 320∗240, and the total 1000 sampling frame was captured, of which 200 expressions defined key frame.In this paper, we adopted the method of the marked points in face to extract facial motion data, and the most difference with other methods is that our method is based on the MPEG-4 standard (Figure1(a)). The advantage of the standard is that you can share daters, and the data can be used with any standards-based grid. By marked points, we can get more accurate facial motion data, while blue circle chip can be pasted in any place, including eyebrows and eyelids, and have no effects on their movement, which makes it possible to get the whole facial motion data and satisfy with the processing of the next step.Mesh model of neutral expression.
(a)
AAM
(b)
Mesh triangleFigure 2
ERI of frown expression.In order to obtain experimental data, first we make a coarse positioning through the blue calibration point using face detection tools and have calibration and reduction transformation operations on the sample data. Then we design a face mesh model in Figure1 to describe the geometry of the facial animation. Mesh vertices were mapped automatically using active appearance model (AAM) and manual fine tuning to meet subsequent demand for data extraction.After obtaining the data of the texture and feature points, all texture will be aligned to the average model.


## 3.2. Features Points

According to MPEG-4, we defined six basic expressions types (see Figure3) and captured them from video camera. Twenty-four key frames demanded for training and one nonkey frame used for testing per expression type are extracted with resolution of 320∗240 pixels. In Figure 1(a), the 68 feature points are marked automatically with AAM [13]. In our experiment, we adopted the 32 feature points belonging to the FPs in MPEG-4.Figure 3
Six basic facial expressions.


## 4. Computation of Eigen-ERI

### 4.1. The Computation of the ERI

For building general ERI’s model, we used a large of frontal and neutral expression face as sample library. And we extract the outline of the face features by the method of the active appearance model (AAM). Then the following formula was defined to compute the average face’s ERI, as the standard shape model:(1)Fm=1N∑i=1NFi.HereFm means average face, and Fi means any one face i=1,2,…,N, and N means the number of the frontal and neutral expression faces.For getting facial expression details, we compute the ERI from all key frames sequences continuously. The first frame is denoted byF and regarded as a neutral face image. Fi(1≤i<n) denoted the rest of expressive face images samples, where n is the total number of key frames. Each expressive face sample Fi will be aligned to F. Then, the ERI of the sample sequence can be computed as follows:
(2)Ri(u,v)=Fi(u,v)F(u,v).Here, the(u,v)  denote the coordinates of a pixel in the image, and Fi and F denote the color values of the pixels, respectively.Because the computation of ERI is each point one by one according to [14], the results will be unpredictable if the face features cannot be aligned exactly. So we revise the definition of ERI’s computation
(3)R(u,v)=F′(u,v)Ave(F(u,v)).


### 4.2. Eigen-ERI

The matrix of ERI is the 2D gray imageI(x,y). We represent it with the W×H dimension vector Γ. The training sets are {Φi∣i=1,…,M}; the M means the sum of the images. The average vector of the all images is
(4)ψ=1M∑i=1MΓiThe difference value of the ERI’sΓi and the average of the ERI’s ψ were defined as Φi:
(5)Φi=Γi-ψ;i=1,…,M.The covariance matrix is represented as follows:(6)C=AAT,
where A=[Φ1,…,ΦM].Feature face is composed of the orthogonal eigenvector of the covariance matrix.ERI calculation only takes theM eigenvector corresponding to the largest eigenvalue. The M is decided by the threshold of the θλ:
(7)J=min⁡r{r∣∑i=1rλi∑j=1Mλj>θλ}.
In this paper, we select the maximal 21 variables in the eigen-ERI space to represent 96% variation information in the sample sets.


## 4.1. The Computation of the ERI

For building general ERI’s model, we used a large of frontal and neutral expression face as sample library. And we extract the outline of the face features by the method of the active appearance model (AAM). Then the following formula was defined to compute the average face’s ERI, as the standard shape model:(1)Fm=1N∑i=1NFi.HereFm means average face, and Fi means any one face i=1,2,…,N, and N means the number of the frontal and neutral expression faces.For getting facial expression details, we compute the ERI from all key frames sequences continuously. The first frame is denoted byF and regarded as a neutral face image. Fi(1≤i<n) denoted the rest of expressive face images samples, where n is the total number of key frames. Each expressive face sample Fi will be aligned to F. Then, the ERI of the sample sequence can be computed as follows:
(2)Ri(u,v)=Fi(u,v)F(u,v).Here, the(u,v)  denote the coordinates of a pixel in the image, and Fi and F denote the color values of the pixels, respectively.Because the computation of ERI is each point one by one according to [14], the results will be unpredictable if the face features cannot be aligned exactly. So we revise the definition of ERI’s computation
(3)R(u,v)=F′(u,v)Ave(F(u,v)).


## 4.2. Eigen-ERI

The matrix of ERI is the 2D gray imageI(x,y). We represent it with the W×H dimension vector Γ. The training sets are {Φi∣i=1,…,M}; the M means the sum of the images. The average vector of the all images is
(4)ψ=1M∑i=1MΓiThe difference value of the ERI’sΓi and the average of the ERI’s ψ were defined as Φi:
(5)Φi=Γi-ψ;i=1,…,M.The covariance matrix is represented as follows:(6)C=AAT,
where A=[Φ1,…,ΦM].Feature face is composed of the orthogonal eigenvector of the covariance matrix.ERI calculation only takes theM eigenvector corresponding to the largest eigenvalue. The M is decided by the threshold of the θλ:
(7)J=min⁡r{r∣∑i=1rλi∑j=1Mλj>θλ}.
In this paper, we select the maximal 21 variables in the eigen-ERI space to represent 96% variation information in the sample sets.


## 5. Extraction of FAP

### 5.1. Definition of the FAP (Facial Animation Parameter)

The FAP is defined by a set of facial animation parameters in MPEG-4 standard. FAP based on the face of small actions is very close to the facial muscle movement. In fact, FAP parameter represents a set of basic facial movements, including the head movement control, tongue, eyes and lips, facial expression and lip can reproduce the most natural move. In addition, like those of humans did not exaggerated facial expressions cartoon, FAP parameter can be traced.There are six basic expressions types, defined in MPEG-4 (see Figure3). The MPEG-4 has a total of 68 FAP. The value of FAP is based on FAPU (facial animation parameter unit) as the unit, so that FAP has the versatility. The calculation formulation is
(8)FAPi=FP′-FPFAPU.Among them,i=3,…,68  FP  and  FP′  are neutral, facial feature points on the corresponding parts.


### 5.2. The Implement of Face Animation Based on FAP

Facial animation definitions table defines three parts. First, the FAP range is divided into several segments. Second, we want to know which grid points are controlled by the FAP in the mesh. Third, we must know the motion factor of the control points in each segment. Each FAP needs to find out which parts of the three parts in facial animation definition table, then according to the MPEG-4 algorithm, calculated by the displacement of the FAP control all grid points. For a set of FAPs, each FAP calculated the effective grid point rod size; by the shift-and-add up you will get a vivid facial expression (Figure3). The concrete implement may refer to [15].


## 5.1. Definition of the FAP (Facial Animation Parameter)

The FAP is defined by a set of facial animation parameters in MPEG-4 standard. FAP based on the face of small actions is very close to the facial muscle movement. In fact, FAP parameter represents a set of basic facial movements, including the head movement control, tongue, eyes and lips, facial expression and lip can reproduce the most natural move. In addition, like those of humans did not exaggerated facial expressions cartoon, FAP parameter can be traced.There are six basic expressions types, defined in MPEG-4 (see Figure3). The MPEG-4 has a total of 68 FAP. The value of FAP is based on FAPU (facial animation parameter unit) as the unit, so that FAP has the versatility. The calculation formulation is
(8)FAPi=FP′-FPFAPU.Among them,i=3,…,68  FP  and  FP′  are neutral, facial feature points on the corresponding parts.


## 5.2. The Implement of Face Animation Based on FAP

Facial animation definitions table defines three parts. First, the FAP range is divided into several segments. Second, we want to know which grid points are controlled by the FAP in the mesh. Third, we must know the motion factor of the control points in each segment. Each FAP needs to find out which parts of the three parts in facial animation definition table, then according to the MPEG-4 algorithm, calculated by the displacement of the FAP control all grid points. For a set of FAPs, each FAP calculated the effective grid point rod size; by the shift-and-add up you will get a vivid facial expression (Figure3). The concrete implement may refer to [15].


## 6. SVR-Based FAP Driving Model

### 6.1. Support Vector Regression (SVR)

In a typical regression problem, we are given a training set of independent and identically distributed(i.i.d.) examples in the form of n ordered pairs, {(xi,yi)}i=1n⊂Rd×R, where xi and yi denote the input and output, respectively, of the ith training example. Linear regression is the simplest method to solve the regression problem where the regression function is a linear function of the input. As a nonlinear extension, support vector regression is a kernel method that extends linear regression to nonlinear regression by exploiting the kernel trick [16, 17]. Essentially, each input xi∈Rd is mapped implicitly via a nonlinear regression map ϕ(·) to some kernel-induced feature space F where linear regression is performed. Specifically, SVR learns the following regression function by estimating w∈F and w0∈R from the training data:
(9)f(x)=〈w,ϕ(x)〉+w0,
where 〈*,*〉 denotes the inner product in F. The problem is solved by minimizing some empirical risk measure that is regularized appropriately to control the model capacity.One commonly used SVR model is calledε-SVR model, and the ε-insensitive loss function
(10)|y-f(x)|ε=max⁡{0,|y-f(x)|-ε}
is used to define an empirical risk functional, which exhibits the same sparseness property as that for support vector classifiers (SVC) using the hinge loss function via the so-called support vectors. If a data point x lies inside the insensitive zone called the ε-tube, that is, |y-f(x)|≤ε, then it will not incur any loss. However, the error parameter ε∣0 has to be specified a priori by the user. The primal optimization problem for ε-SVR can be stated as follows:
(11)min⁡W,ξ(*)λ2∥W∥2+∑i=1n(ξi+ξi*)yi-(〈W,ϕ(xi)〉+w0)≤ε+ξi(〈W,ϕ(xi)〉+w0)-yi≤ε+ξi*ξi*≥0.Relatively, (3) transforms into
(12)f(x)=∑i=1l(-αi+α)K(xi,x)+b.Here,αi denotes Lagrange multiplier, K(xi,x) is kernel function, and f(x) is decision function. Forpredicting, it is only a dot product operation and costs very low in the real time. Figure 4 shows the steps of the SVRM mapping ERI to FAP parameters.Figure 4
The steps of SVR processing.


### 6.2. SVR-Based Algorithm

According to the above theory, an FAP driving model for its every parameter is built. Given a set of training data{(x1,y1),(x2,y2),…,(xl,yl)}⊂X×Y, where X denotes the space of the ERI and x1∈R is the ERI’s parameter and Y denotes the space of the feature of FAP and y1∈R is an FAP. We regard it as a regressive problem and represent it by the support vector regression method, including ε-SVR [16] and v-SVR [17]. The offline learning progress is as Figure 5.Figure 5
Offline learning progress.An offline learning process was showed in Figure5. In the above section, we have got the learning parameters of the FAPs and ERIs. Then we can get a regressive model from the ERI’s parameters to the FAPs vectors through the SVR model. In a statistical sense, the results of FAPs can reflect the expression relative with the ERI.There are several steps used to explain how the facial expression animation is driven by video camera.Step 1.
Establish the MPEG-4 based 3D facial expression animation system driving by FAP; see Figure6.3D facial expression animation system.
(a)
3D mesh
(b)
Synthesis faceStep 2.
Capture and detect face image from video camera, and compute its ERI.Step 3.
Forecast the FAPs of the current frame according to its ERI.Step 4.
Compute the motion of the feature points in the face mesh based on the new FAP and animate the 3D human face mesh.


## 6.1. Support Vector Regression (SVR)

In a typical regression problem, we are given a training set of independent and identically distributed(i.i.d.) examples in the form of n ordered pairs, {(xi,yi)}i=1n⊂Rd×R, where xi and yi denote the input and output, respectively, of the ith training example. Linear regression is the simplest method to solve the regression problem where the regression function is a linear function of the input. As a nonlinear extension, support vector regression is a kernel method that extends linear regression to nonlinear regression by exploiting the kernel trick [16, 17]. Essentially, each input xi∈Rd is mapped implicitly via a nonlinear regression map ϕ(·) to some kernel-induced feature space F where linear regression is performed. Specifically, SVR learns the following regression function by estimating w∈F and w0∈R from the training data:
(9)f(x)=〈w,ϕ(x)〉+w0,
where 〈*,*〉 denotes the inner product in F. The problem is solved by minimizing some empirical risk measure that is regularized appropriately to control the model capacity.One commonly used SVR model is calledε-SVR model, and the ε-insensitive loss function
(10)|y-f(x)|ε=max⁡{0,|y-f(x)|-ε}
is used to define an empirical risk functional, which exhibits the same sparseness property as that for support vector classifiers (SVC) using the hinge loss function via the so-called support vectors. If a data point x lies inside the insensitive zone called the ε-tube, that is, |y-f(x)|≤ε, then it will not incur any loss. However, the error parameter ε∣0 has to be specified a priori by the user. The primal optimization problem for ε-SVR can be stated as follows:
(11)min⁡W,ξ(*)λ2∥W∥2+∑i=1n(ξi+ξi*)yi-(〈W,ϕ(xi)〉+w0)≤ε+ξi(〈W,ϕ(xi)〉+w0)-yi≤ε+ξi*ξi*≥0.Relatively, (3) transforms into
(12)f(x)=∑i=1l(-αi+α)K(xi,x)+b.Here,αi denotes Lagrange multiplier, K(xi,x) is kernel function, and f(x) is decision function. Forpredicting, it is only a dot product operation and costs very low in the real time. Figure 4 shows the steps of the SVRM mapping ERI to FAP parameters.Figure 4
The steps of SVR processing.


## 6.2. SVR-Based Algorithm

According to the above theory, an FAP driving model for its every parameter is built. Given a set of training data{(x1,y1),(x2,y2),…,(xl,yl)}⊂X×Y, where X denotes the space of the ERI and x1∈R is the ERI’s parameter and Y denotes the space of the feature of FAP and y1∈R is an FAP. We regard it as a regressive problem and represent it by the support vector regression method, including ε-SVR [16] and v-SVR [17]. The offline learning progress is as Figure 5.Figure 5
Offline learning progress.An offline learning process was showed in Figure5. In the above section, we have got the learning parameters of the FAPs and ERIs. Then we can get a regressive model from the ERI’s parameters to the FAPs vectors through the SVR model. In a statistical sense, the results of FAPs can reflect the expression relative with the ERI.There are several steps used to explain how the facial expression animation is driven by video camera.Step 1.
Establish the MPEG-4 based 3D facial expression animation system driving by FAP; see Figure6.3D facial expression animation system.
(a)
3D mesh
(b)
Synthesis faceStep 2.
Capture and detect face image from video camera, and compute its ERI.Step 3.
Forecast the FAPs of the current frame according to its ERI.Step 4.
Compute the motion of the feature points in the face mesh based on the new FAP and animate the 3D human face mesh.


## 7. Experiment Results

We have implemented all the techniques described above and built an automatic 3D facial expression animation system on Windows environment. The result was showed in Figure7. We represent six basic expressions driving by FAP which come from the forecast through ERI.Five basic 3D expressions’ animation.
(a)
Anger
(b)
Joy
(c)
Disgust
(d)
Sadness
(e)
Surprise


## 8. Conclusion

In this paper, we realize a 3D face animation system that can generate realistic facial animation with realistic expression details and can apply in different 3D model similar to human. It is capable of generating the statistical realistic facial expression animation while only requiring simply camera device as the input data and it works better in any desired 3D face model based on MPEG-4 standard.



---

*Source: 706159-2014-05-20.xml*