## 2 Materials and Methods
### Experiments
#### 2.3.2 Data Preprocessing  
Face detection and segmentation of upper face and lower face needs to be performed before FER. We use Conditional with Local Neural Fields (CLNF) with OpenFace toolkit [35] to detect the face locate the key points on the face (show in Figure 3a). Every frame is aligned and cropped according to the key points. The upper face region (R1 in Figure 3b) and lower face region (R2 in Figure 3b) are rectangular regions, and are segmented by using location of eyes points, nose point, mouth points (shown as the specific red point position in Figure 3a). Assume that the two eye points are \(\mathrm{E_{1}}(\mathrm{X_{1}},\mathrm{Y_{1}})\) and \(\mathrm{E_{2}}(\mathrm{X_{2}},\mathrm{Y_{2}})\), nose point are \(\mathrm{N(X_{3}},\mathrm{Y_{3}})\), central point of mouth is \(\mathrm{M(X_{4}},\mathrm{Y_{4}})\), left point of mouth is \(\mathrm{ML(X_{6}},\mathrm{Y_{6}})\), right point mouth is \(\mathrm{MR(X_{7}},\mathrm{Y_{7}})\). Then the centroid of R1 is \(\mathrm{R1c(X_{5}},\mathrm{Y_{5}})=\mathrm{R1c((X_{1}+X_{2})/2,(Y_{1}+Y_{2})/2)\), and the centroid of R2 is \(\mathrm{R2c}=\mathrm{M(X_{4}},\mathrm{Y_{4}})\). The height and width of R1 are \(|Y_{5}-Y_{3}|\) and \(\frac{5}{3}|\mathrm{X_{2}}-\mathrm{X_{1}}|\), respectively. The height and width of R2 is \(\frac{5}{3}|\mathrm{Y_{4}}-\mathrm{Y_{3}}|\) and \(|\mathrm{Y_{7}}-\mathrm{Y_{6}}|\), respectively.  
For local networks, we divide the face into two non-overlapping regions including upper and lower regions. The upper region contains eyebrows and eyes and the lower region contains the mouth. We crop the upper region and lower region according to eyes and mouth key points, as shown by the specific red point position in Figure 3a, the cropped upper and lower regions is shown in Figure 3b.  
Each video sequence was normalized for 32 frames by using the linear interpolation method [36]. For global input data flow, each image was resized to \(108\times 88\) by using bilinear interpolation method [37]. For local input data flows, each image was resized to \(36\times 64\). Then all the preprocessed images were converted from RGB into 8-bit gray scale images to reduce computation complexity.  
Data augmentation has always been used as an effective way of reducing over-fitting and improving recognition rates in the case of a small dataset [32]. In the training dataset, we cropped each image along its corner (random location) and changed it from \(108\times 88\) into \(98\times 78\). Then, the cropped image is recovered into original size \(108\times 88\) by using the bilinear interpolation method. After augmentation, we obtain a training set that is 50 times larger than original dataset.  
Figure 3: (a) The 68 facial points tracked by Conditional with Local Neural Fields (CLNF) (b) cropped upper and lower regions.  
Figure 2: Anger, disgust, fear, happiness, sadness, surprise images of one person in apex condition of Oulu-CASIA NIR facial expression database.  
The 10-fold cross-validation is used. Therefore, there are 72 people as the training set and 8 people as the test set. In all experiments, there are no overlapping images between the training sets and test sets.