# Hybrid Approach for Facial Expression Recognition Using Convolutional Neural Networks and SVM
## 2 Proposed Facial Expression Recognition System Using the Mixture of CNN and SVM Classifier
### SVM-Based Facial Geometry Displacement Classification  
Facial landmark points are used as input data for facial geometric motion classification. In the case of the CK+ database, the landmark points extracted from the AAMs [9] were provided, and we use the shape data in Equation (4) as the input data for the geometric motion analysis. \[s(t)=[x_{1}(t),y_{1}(t),x_{2}(t),y_{2}(t),\cdots,x_{i}(t),y_{i}(t),\cdots,x_{n }(t),y_{n}(t)], \tag{4}\]  
where \(n\) is the number of vertices, and \(t\) is the sampling time. When the landmark points are not provided, AAM [9] or landmark point detection and tracking algorithms [56; 57; 58] can be used to extract the landmark point information from the facial expression sequence. Figure 4: Basic and modified CNN architectures for facial expression recognition from facial motion flow. For facial geometry displacement extraction, we extracted the displacement from neutral facial geometry. Assuming that the facial expression sequence starts from a neutral expression, we extracted the landmark point displacement by subtracting each landmark point sequence by the initial landmark points from discrete sampling frames. \[\Delta_{g}(t)=[x_{1}(t)-x_{1}(0),y_{1}(t)-y_{1}(0),x_{2}(t)-x_{2}(0),\] \[\qquad y_{2}(t)-y_{2}(0),\cdots,x_{n}(t)-x_{n}(0),y_{n}(t)-y_{n}(0 )],\] \[=[\Delta x_{1}(t),\Delta y_{1}(t),\Delta x_{2}(t),\Delta y_{2}(t), \cdots,\Delta x_{n}(t),\Delta y_{n}(t)], \tag{5}\]  
For each landmark point, we further compute the displacement intensity and angle. For example, for the \(i\)th landmark point among \([1,\cdots,n]\) at time \(t\), the displacement can be estimated as follows:  
\[\Delta L_{\mathit{Intensity}_{i}}(t) = \sqrt{\Delta x_{i}(t)^{2}+\Delta y_{i}(t)^{2}}, \tag{6}\] \[\Delta L_{\mathit{Angle}_{i}}(t) = tan^{-1}\bigg{(}\frac{\Delta y_{i}(t)}{\Delta x_{i}(t)}\bigg{)} \tag{7}\]  
Therefore, the feature vector for the SVM at time \(t\) can be represented by  
\[\mathbf{x}(t)=[\Delta L_{\mathit{Intensity}_{1}}(t),\Delta L_{ \mathit{Intensity}_{2}}(t),\cdots,\Delta L_{\mathit{Intensity}_{n}}(t)\] \[\qquad\qquad\qquad\Delta L_{\mathit{Angle}_{1}}(t),\Delta L_{ \mathit{Angle}_{2}}(t),\cdots,\Delta L_{\mathit{Angle}_{n}}(t)], \tag{8}\]  
Figure 5 shows the collection of average landmark flow in different expressions with visualization of facial image patches corresponding to large flow in each expression. The horizontal axis of the graph represents the index of the landmark. The index from 0 to 67 represents \(\delta x_{i}\), and the index from 68 to 135 represents \(\delta y_{i}\) of the landmark points. The average value of each facial expression is plotted in the graph. Relevant facial areas corresponding to their landmark point variations are provided in the graph to show what the average landmark variations represent. The plotted landmark flow shows that landmark flow can be used to characterize local facial action areas such as the mouth, eyes and eye brows in each expression. For the given feature vectors and class labels, SVMs attempt to find the hyperplane that maximizes the positive and negative observation for a specified class [65]. The decision function can be specified by  
\[D(\mathbf{x}) = \sum_{k}y_{k}\alpha_{k}^{*}K(\mathbf{x}(k),\mathbf{x})+b,\alpha_{k}^{*}\geq 0, \tag{9}\] \[y_{k} = \left\{ \begin{array}{ll}1&\text{if }\mathbf{x}(k)\in\text{class A}\\ -1&\text{otherwise}\end{array} \right. \tag{10}\]  
where only support vectors, which are the closest data point to the margin of the classifier, appear in the sum with a nonzero weight, and \(y_{k}=1\) if they are to be classified as positive samples; otherwise, \(y_{k}=-1\), and the symmetric kernels contain finite or infinite series expansions of the form  
\[K(\mathbf{x},\mathbf{x}^{{}^{\prime}})=\sum_{i}\varphi_{i}(\mathbf{x})\varphi_{i}(\mathbf{x}^{{ }^{\prime}}) \tag{11}\]  
The solution for Equation (9) with kernel expansion in Equation (11) can be found by quadratic optimization, and we used the implementation of a library for support vector machines (LIBSVM) [66].