Spotting micro‐movements in image sequence by introducing intelligent cubic‐LBP

IET Image ProcessingVolume 16, Issue 14 p. 3814-3830 ORIGINAL RESEARCHOpen Access Spotting micro-movements in image sequence by introducing intelligent cubic-LBP Vida Esmaeili, Vida Esmaeili Faculty of Electrical and Computer Engineering, University of Tabriz, 29 Bahman Blvd., Tabriz, IranSearch for more papers by this authorMahmood Mohassel Feghhi, Corresponding Author Mahmood Mohassel Feghhi mohasselfeghhi@tabrizu.ac.ir orcid.org/0000-0002-7193-843X Faculty of Electrical and Computer Engineering, University of Tabriz, 29 Bahman Blvd., Tabriz, Iran Correspondence Mahmood Mohassel Feghhi, Faculty of Electrical and Computer Engineering, University of Tabriz, 29 Bahman Blvd., Tabriz, Iran. Email: mohasselfeghhi@tabrizu.ac.irSearch for more papers by this authorSeyed Omid Shahdi, Seyed Omid Shahdi Department of Electrical Engineering, Qazvin Branch, Islamic Azad University, Qazvin, IranSearch for more papers by this author Vida Esmaeili, Vida Esmaeili Faculty of Electrical and Computer Engineering, University of Tabriz, 29 Bahman Blvd., Tabriz, IranSearch for more papers by this authorMahmood Mohassel Feghhi, Corresponding Author Mahmood Mohassel Feghhi mohasselfeghhi@tabrizu.ac.ir orcid.org/0000-0002-7193-843X Faculty of Electrical and Computer Engineering, University of Tabriz, 29 Bahman Blvd., Tabriz, Iran Correspondence Mahmood Mohassel Feghhi, Faculty of Electrical and Computer Engineering, University of Tabriz, 29 Bahman Blvd., Tabriz, Iran. Email: mohasselfeghhi@tabrizu.ac.irSearch for more papers by this authorSeyed Omid Shahdi, Seyed Omid Shahdi Department of Electrical Engineering, Qazvin Branch, Islamic Azad University, Qazvin, IranSearch for more papers by this author First published: 25 July 2022 https://doi.org/10.1049/ipr2.12596AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinked InRedditWechat Abstract Invisible to naked human eyes, micro-movements are barely noticeable. There are wide ranges of micro-movement applications from spotting subtle changes in a volcano, vascular pulse, and blood vessel to micro-expression detection. In the latter, some evil thoughts can be unveiled, resulting in identifying crooks and lawbreakers and it arises from involuntary subtle and short-duration of facial muscles movements. Precise spotting of these tiny movements is possible only when multiple aspects of temporal images are scrutinized. Meanwhile, since motions often happen in one or two directions, it is rudimentary to extract complete feature sets in textural-based approaches such as cubic Local Binary Pattern (cubic-LBP). Approaches like cubic-LBP also have imposed an unnecessary computation burden. Hence, in this research, a novel method named intelligent cubic-LBP is proposed, which incorporates Convolutional Neural Network (CNN) model. This model learns to select the useful plane(s) automatically. Apex is then detected by applying Partial Autocorrelation Coefficient (PACF) on the selected plane(s). The experimental results show significant improvement in micro-movements identification. The accuracy of the apex frame detection has elevated to 10% and 17% in the Chinese Academy of Sciences Micro-Expressions (CASME) and the CASME II databases, respectively. 1 INTRODUCTION Rich data derived from some of the micro-movements make them noteworthy. One of them is Micro-Expression (ME), in which micro-movements of facial muscles are interpreted as facial expressions. This facial micro-momentary expression is uncontrollable and involuntary [1-3]; hence, it divulges lure, lie, fraud, and wile. As a consequence, it can be applied as an interrogation supplement in police departments, courts, health centres etc [4-7]. Suppose a depressed patient pretends to be happy; while in fact, he is sad and worried. The psychotherapist when examining him/her very carefully perceives that he/she seems sad for a moment. This facial expression occurs unconsciously despite the patient's will to avoid it. Indeed, this expression is very short in duration and subtle in motion [8]. Accordingly, detection of such motion requires high attention due to its low intensity and duration. This brief and rapid involuntary expression is named ME [9], which is considered to be a real and crucial emotion. In another word, the person's heart feeling appears as ME in his face [10]. The other micro-movement is smoke movement and its change in volcanoes. Identification of these subtle changes is an essential task for understanding active volcanoes [11, 12]. Thus, persons can prevent many possible damages. Another important micro-movement is vascular pulse detection and localization of blood vessels. Locating blood vessels is critical during transfusion and surgery. Although it is a difficult task to identify vessels manually, its exact identification automatically avoids inadvertent damage [13]. Micro-saccades are involuntary eye movements when visual fixation [14]. They are subtle and occur rapidly. Their investigation helps in cognition, visual perception, eye motion control, and visual or covert attention [15]. Although exact micro-saccade detection is significant for clinical neuroscience research due to its distinctive visual characteristics, its automatic detection can be arduous [16]. Haggard and Isaacs [17], and then Ekman [9] introduced the concept of ME in the late twentieth century. Through their experiments, they found out that ME occurs in hundredths of a second (less than 0.34 s). ME appeared by the local spontaneous movement of facial muscles around the eyes, chin, eyebrows, or mouth. With regards to the aforementioned challenges, spotting ME needs expertise with cumbersome effort and concentration. In recent decades, as an alternative to trained experts, machine learning proved to be promising in spotting MEs, though some problems still remained unsolved [18]. In addition, the subtle and small movements are not visible well in one image. It seems that the ME is the neutral and ordinary expression in an image. For that reason, a comparison of sequential images is suitable [19]. Many methods are presented to tackle micro-movement revelation problems. These methods extract micro-movement features in sequential frames. A method has the best performance when it is able to represent subtle motions correctly. We classify them into two groups. The first group is handcrafted methods [20, 21]. These methods are the Local Binary Pattern (LBP)-based, Optical Flow (OF)-based, gradient-based, wavelet-based, and motion magnification methods [7, 11, 22–24]. The LBP-based methods such as the LBP on Three Orthogonal Planes (LBP-TOP) [25], and the cubic-LBP [7] have good efficiency on texture [7, 24, 26]. Their characteristics are simple in the calculation, sensitive to illumination, and insensitive to orientation. Although the gradient-based methods, such as the Histogram of Image Gradient Orientation (HIGO) [24] are robust to changing light, they have costly calculations. Wavelet can be used on different resolutions or scales of images; however, it needs a high elapsed time for performance. The OF-based methods take dynamic movement information in adjacent frames. However, they have weak performance, when shaking. Nonetheless, these handcrafted methods depended on previous science and heuristics [21, 27, 28]. The second group is deep learning methods. Unlike handcraft methods, deep learning methods are independent of past data processing. They learn the meaningful features from primary data [29]. Furthermore, these methods are fine-tuned for the demanded outcome, automatically. Nevertheless, they need a large number of data [28, 30, 31]. Over time, researchers try to propose an outperformance method for micro-movements analysis. Undoubtedly, the world is full of small and subtle movements that affect us and our lives in many ways. If we carefully explore consecutive frames related to a subtle movement, we decipher that one frame named apex, out of all other frames, can show the movement better than the others. Thus, it signifies the importance of the apex. Our motivation is to reveal the apex using an outperformed method. Related research has shown that the cubic-LBP [7] can spot all the minute variations in a series of images and analyze them with satisfying outcomes. However, since its corporates large numbers of planes, its total processing time is relatively high. For the sake of optimization, we can pick out the most effective plane(s) automatically. In this paper, we propose a novel method named the intelligent cubic-LBP. We combine a handcrafted method with a deep learning method. We combine the cubic-LBP with Convolutional Neural Network (CNN). It selects particular planes from 15 planes of cubic-LBP. These particular planes show micro-movements in their orientation well. In other words, CNN intelligently offers the best planes of cubic-LBP that it has learned to spot micro-movements. For this reason, we called our proposed method the intelligent cubic-LBP. Using the planes provided, the apex frame, which contains the maximum micro-movements and change, can be detected accurately in many testing samples. Thus, the micro-movements are revealing and visible in this frame. The advantages of the introduced method are facilitating the micro-movements analysis and reducing the processing time. The framework of the proposed intelligent cubic-LBP method is shown in Figure 1. FIGURE 1Open in figure viewerPowerPoint The framework of the proposed intelligent cubic-LBP for apex frame spotting. This sample is sub18-EP15_3 in the CASME database, where frame 5 is its ground-truth apex frame Initially, the images are considered as a 3D array after preprocessing. Afterwards, the cubic-LBP method and Sum of Squared Differences (SSD) are exploited to determine the particular plane(s). The selected plane(s) can show micro-movements well in the images. The images of a sample one by one are fed into CNN, and its corresponding label is the particular plane number. The CNN model is learned to recognize one or multi-plane(s). In other words, The CNN output is the name of the plane(s) that can demonstrate the direction of micro-movement. Thus, the CNN learns the direction of micro-movements. Finally, the Partial Autocorrelation Coefficient (PACF) is applied to the series obtained by that plane(s), and the apex frame is spotted. This frame has a peak of feature difference from the neutral frame. The main contributions of this research are listed as follows: Promoting simple cubic-LBP [7] to be intelligent. As far as we know, this work has not yet been done on the LBP-based method. The cubic-LBP method [7] has 15 different planes in a small cuboid, and it can reveal all micro-movements in every direction. However, relatively large numbers of planes, which in most cases include unnecessary features, impose considerable complexity and computation burden. Manually discarding some planes is not also appropriate as some micro-movements information may be lost. Therefore, automatic selection of some efficient planes is proposed in this research. This task is done using CNN, a widely adopted deep-learning-based method for images. This method improves the results in micro-movement analysis, which is further discussed in detail in Section 4. Unlike Esmaeili and Shahdi's work [7], we automatically select a few specific planes and apply the PACF to locate the apex frame. Also, unlike [2, 10, 18], and [19], which consider only limited planes (four or six planes), every 15 planes that represent each direction of movement are included in our work. Assume an example, where the fifth frame is the ground-truth apex referring to the database tag in a sample (e. g., CASME-sub01-EP16_3). In this example, using the cubic-LBP and the SSD presented in [7], which incorporated all 15 planes, the apex cannot be spotted correctly (see Figure 2). According to Figure 2, the sixth frame has a maximum feature variation among 15 planes. Meanwhile, some planes (such as plane 4, plane 6, and plane 15) accurately demonstrate the fifth frame as the apex. Thus, these planes can precisely reveal micro-movements. In this study, we use the CNN, where inputs and labels to this network are images and efficient plane numbers respectively (here plane 4, plane 6, and plane 15). Therefore, the CNN learns to pick the most efficient planes according to the direction of micro-movements in each sample. In the next step, we consider the effective planes on the sample images. Thereafter, we calculate the max for series production. Finally, the PACF is applied to the series for locating the apex frame. To the best of our knowledge, the apex frame detection has been done only in the ME applications. In this study, we also apply the same procedure on other micro-movement sequential images such as the volcano dataset, the blood vessel dataset, and the micro-saccade data. We prove in Section 4 that using the PACF can increase the accuracy of apex detection. Overall, the combination of this method with the deep-learning approach presents better performance. FIGURE 2Open in figure viewerPowerPoint Determination of apex frame using the cubic-LBP and SSD The remaining parts of this paper are organized in the following manner: The relevant works are discussed in Section 2 and then the suggested method is expressed in Section 3. Our experiments and obtained results are disclosed in Section 4. At last, the conclusion is described in Section 5. 2 RELATED WORKS LBP is a texture feature extractor in a grayscale image. This method is used for micro-movement analysis [6]. It has produced relatively good results in comparison with other methods such as Optical Strain (OS), Constraint Local Model (CLM), Histogram of Oriented Optical Flow (HOOF), and Region HOOF (RHOOF) in micro-movement detection [32]. Also, developed LBP methods can extract key micro-movement features well. The cubic-LBP was first introduced by Esmaeili and Shahdi [7] in 2020. This method considers all possible planes. Ergo, it shows the vertical, horizontal, and oblique movements. Yan et al. [33] applied the LBP on 20% of the CASME II dataset for ME apex detection. The apex frame shows the most facial muscle motion. This frame is located between the ME onset and ME offset frame. The obtained Standard Error (SE) was 1.88 in their work. Since using fewer samples may lead to incorrect results, Liong et al. [34] exploited the LBP in the left and right eye, left and right eyebrow, and mouth on a whole of 247 CASME II samples. According to the advantages of feature extraction from high-frequency regions, they achieved a 20% improvement in the final result. In the previous works, the distance between the ground truth and apex was long. Then, Liong et al. [6] found the apex one frame after ground-truth using the divide-and-conquer strategy. In another word, though the correct ground truth was the sixty-third frame, Liong et al. [6] determined the sixty-fourth frame as the apex frame. However, the apex had to be detected correctly. Because it has the most movement information. The correct apex can be used for other tasks. Therefore, researchers came up with the idea of using the time dimension. Thus, they utilized LBP-TOP, cubic-LBP, and other developed LBP methods [7, 10]. These methods extract the sequential frame features using their corresponding planes. Suppose there are three sequential frames. These methods measure the differences between the before/after frame and the middle frame. Thus, they can display micro-movements transfer fine. The successful results and the SE and Mean Absolute Error (MAE) errors reduction in ME researches confirm this claim [6, 7, 10, 18, 35, 36]. The true apex frame has been determined using cubic-LBP in 38% of CASME samples [7]. These found frames were ground-truth exactly. Also, the MAE and SE errors were significantly reduced against other methods [7]. This indicates that incorrectly detected apex frames are also very close to the ground truth. However, the micro-movement analysis results can be upgraded yet. Also, sometimes the Eulerian video motion magnification is used to reveal subtle micro-movements such as vessel localization, ME, micro-saccades etc. [19, 24, 37–39]. It makes visible micro and subtle movements exaggeratedly in a video. However, we can disclose these movements and changes without exaggeration. Recently, deep learning methods such as CNN have received increasing attention from researchers in many image processing tasks. For micro-movement recognition, in ref. [40], the spatial features of ME were learned using CNN. Then, these features were transferred to temporal feature learning using Long Short-Term Memory Recurrent NN (LSTMRNN) [41]. Since the ME data is limited, Shallow Triple Stream Three-dimensional CNN (STSTNet) [42] is proposed for ME recognition. In this method, three parallel stream feature maps are the network input. The other method which is enhanced by the ME data reduction is part-based deep NN (PB-DNN) [3]. In ref. [20], 2D–3D CNN has been proposed for ME recognition. It combines 2D spatial convolution with multi-scale 1D temporal convolution for ME spatiotemporal feature extraction. Then the extracted features fuse to obtain the short-term features. This is as Net-A. The Net-B has been used to obtain the long-term feature. Finally, two Nets (Net-A and Net-B) features have been mixed to enhance the ME motion features. For the apex frame spotting, Zhang et al. [43] proposed a feature matrix processing method in 2018. This method utilizes a sliding window and captures the features of ME into account to look for the apex. In addition, recently, Tran et al. [44] employed the LSTM network, which is a special kind of RNN. Here, we optimize the deep-learning method to take advantage of this powerful tool for apex detection either. Moreover, the deep learning scheme can apply to any micro-movements like the volcano, surgery, micro-saccades, and other images in order to locate and identify the apex frame. 3 PROPOSED INTELLIGENT CUBIC-LBP METHOD In this section, after some pre-processing steps, we use cubic-LBP with its 15 distinct planes to collect even minute variations. Then, CNN has been used to pick out the most valuable planes in terms of effective data. Meanwhile, it keeps computation as simple as possible. Finally, the PACF is utilized to spot the apex frame. The details on each part are given below separately. The complete block diagram of the proposed method for spotting the apex frame is illustrated in Figure 3. FIGURE 3Open in figure viewerPowerPoint The complete block diagram of the proposed method for spotting the apex frame 3.1 Pre-processing The first step is to prepare video frames for being fetched into the proposed method. For this purpose, initially, all images are converted into a sequence of grayscale images which are thereafter considered as 3D arrays (i.e. x, y, and z). In other words, we convert the video to a set of sequential images. Since this image set contains the RGB channels, we first convert them into grayscale. Then, a trail consisting of a stack of 2D images is considered on the time axis. It builds up a 3D Spatio-temporal volume (x, y, z) or 3D array. The 3D array keeps the grey level of all the pixels in the sequence with height, width, and length. Each of the input images should be resized to get the same standard dimensions as the others. It means that all images in the sequence are rearranged to have equal sizes (width (x) and height (y)). For further normalization, we adopt histogram equalization to both increase contrast and decrease illumination sensitivity at the same time. Figure 4. shows the pre-processing procedure. FIGURE 4Open in figure viewerPowerPoint The pre-processing procedure 3.2 Cubic-LBP As reported in [6], we apply fifteen individual planes of the cubic-LBP on the 3D arrays. These planes could be assembled in a cube as depicted in Figure 5. As it can be perceived, six planes are six facets of the cuboid, and the rest are in the cubic which crisscrossed together in the central pixel. FIGURE 5Open in figure viewerPowerPoint The cubic-LBP planes Supposing that every plane takes the pixels of the sequential images in the corresponding direction, then all pixels are given by running the planes toward image frames. In the LBP process, the grey value of the local neighbouring point (gp) compares to the central pixel (gc) in each plane. If the subtraction result is negative, then the sign is considered to be 0; otherwise, it sets to be 1. The coordinates of the neighbouring point from the first plane up to the fifteenth plane are defined by Equation (1) up to Equation (15) as given below where their corresponding planes are also illustrated in Figure 6: x c + r cos i , y c − r sin i , z c \begin{equation}\left( {xc + r{\rm{cos}}\left( i \right),yc - r{\rm{sin}}\left( i \right),zc} \right)\end{equation} (1) x c + r cos i , y c − r sin i , z c + S \begin{equation}\left( {xc + r{\rm{cos}}\left( i \right),yc - r{\rm{sin}}\left( i \right),zc + S} \right)\end{equation} (2) x c + r cos i , y c − r sin i , z c − S \begin{equation}\left( {xc + r{\rm{cos}}\left( i \right),yc - r{\rm{sin}}\left( i \right),zc - S} \right)\end{equation} (3) x c , y c − r sin i , z c + S cos i \begin{equation}\left( {xc,yc - r{\rm{sin}}\left( i \right),zc + S{\rm{cos}}\left( i \right)} \right)\end{equation} (4) x c + r , y c − r sin i , z c + S cos i \begin{equation}\left( {xc + r,yc - r{\rm{sin}}\left( i \right),zc + S{\rm{cos}}\left( i \right)} \right)\end{equation} (5) x c − r , y c − r sin i , z c + S cos i \begin{equation}\left( {xc - r,yc - r{\rm{sin}}\left( i \right),zc + S{\rm{cos}}\left( i \right)} \right)\end{equation} (6) x c + r cos i , y c , z c + S sin i \begin{equation}\left( {xc + r{\rm{cos}}\left( i \right),yc,zc + S{\rm{sin}}\left( i \right)} \right)\end{equation} (7) x c + r cos i , y c + r , z c + S sin i \begin{equation}\left( {xc + r{\rm{cos}}\left( i \right),yc + r,zc + S{\rm{sin}}\left( i \right)} \right)\end{equation} (8) x c + r cos i , y c − r , z c + S sin i \begin{equation}\left( {xc + r{\rm{cos}}\left( i \right),yc - r,zc + S{\rm{sin}}\left( i \right)} \right)\end{equation} (9) x c + r sin i , y c − r sin i , z c − S cos i \begin{equation}\left( {xc + r{\rm{sin}}\left( i \right),yc - r{\rm{sin}}\left( i \right),zc - S{\rm{cos}}\left( i \right)} \right)\end{equation} (10) x c − r sin i , y c − r sin i , z c + S cos i \begin{equation}\left( {xc - r{\rm{sin}}\left( i \right),yc - r{\rm{sin}}\left( i \right),zc + S{\rm{cos}}\left( i \right)} \right)\end{equation} (11) x c − r cos i , y c − r sin i , z c − S sin i \begin{equation}\left( {xc - r{\rm{cos}}\left( i \right),yc - r{\rm{sin}}\left( i \right),zc - S{\rm{sin}}\left( i \right)} \right)\end{equation} (12) x c + r cos i , y c − r sin i , z c + S sin i \begin{equation}\left( {xc + r{\rm{cos}}\left( i \right),yc - r{\rm{sin}}\left( i \right),zc + S{\rm{sin}}\left( i \right)} \right)\end{equation} (13) x c + r cos i , y c − r sin i , z c − S cos i \begin{equation}\left( {xc + r{\rm{cos}}\left( i \right),yc - r{\rm{sin}}\left( i \right),zc - S{\rm{cos}}\left( i \right)} \right)\end{equation} (14) x c + r cos i , y c − r sin i , z c + S cos i \begin{equation}\left( {xc + r{\rm{cos}}\left( i \right),yc - r{\rm{sin}}\left( i \right),zc + S{\rm{cos}}\left( i \right)} \right)\end{equation} (15)where r is the radius of a circle in which the middle pixel is located at the centre of the circle and other neighbouring pixels are located on its circumference. i is 2πp/P in which the local neighbouring point is p ϵ 0,…, P−1. Hence, P equals the number of neighbouring points. The time interval among consecutive frames is set to be S. FIGURE 6Open in figure viewerPowerPoint The cubic-LBP planes separately Assume that the coordinates of the central pixel are (xc, yc, zc) and P = 4 is equally spaced pixels on a circle of radius r = 1 in the selected plane (e.g. 4th plane), then the coordinates of these four neighbouring points are given by Equation (4) (see also Figure 7). FIGURE 7Open in figure viewerPowerPoint The coordinates of the central pixel and four neighbouring points in the selected plane Then, the feature vector obtains from the normalized cubic-LBP histogram as follows: H m , n = ∑ x , y , z I { L B P m ( x , y , z ) = m } / ∑ k = 0 N n − 1 H k , n \begin{equation}{H}_{m,n} = \sum_{x,y,z} {I\{ LB{P}_m(x,y,z) = m\} /\sum_{k = 0}^{{N}_n - 1} {{H}_{k,n}} } \end{equation} (16)where m is an integer from 0 to the number of LBP operator labels in the plane (N). The n is a number between 1 and 15. I (.) is 1 when (.) is true, and it is 0 otherwise. The SSD technique can be exerted to evaluate the similarity/discrepancy between two feature vectors. For micro-movement detection, we take the advantage of SSD in a manner where each frame vector is compared with the neutral face frame vector. Expectedly, the highest discrepancy with the neutral frame could end up unravelling the apex frame (see Figure 8). FIGURE 8Open in figure viewerPowerPoint Using the cubic-LBP and the SSD for the particular plane(s) spotting However, the heavy burden of computations which is imposed by considering all 15 planes is not simple to be ignored. Besides, 15 numbers of planes in cubic-LBP provide a broad range of features; though, there are still abundant numbers of rudimentary among interesting features. In order to address this issue, we convert conventional cubic-LBP into intelligent cubic-LBP. In this modification, particular planes out of 15 planes can be selected through a fully automatic procedure. The selection of a few particular planes will be explained in the next subsection (Section 3.3). By performing this task, it is not only the accuracy that increased but at the same time, the processing complexity is diminished. Generally, since we have to cope with the particular plane, we use the cubic-LBP and SSD in this subsection. In fact, the plane's number is chosen to be fed to the CNN. For example, if the fifth frame is the ground truth for an apex in a sample, we use the cubic-LBP and the SSD to spot the plane that determines this frame as the apex. As depicted in Figure 8, planes 11, 12, and 15 verify that the fifth frame is the apex in this sample. Thus, they can reveal micro-movements well among the other planes. Hence, in this example, planes 11, 12, and 15 are the particular planes. Also, we can infer that micro-movements are taken place mostly in the direction of planes 11, 12, and 15 in this sample. 3.3 CNN We optimize CNN for the sake of appropriate plane selection. CNN contains locally connected convolutional (CONV) layers, pooling (POOL) layers, and fully connected (FC) hidden layers. Even though its architecture is mainly efficient, it requires relatively high numbers of image sequences for training. Since related available datasets like micro-expression samples are very scarce, we use pre-trained VGG-Face CNN [45]. The architecture of the VGG-Face is illustrated in Figure 9. In this deep architecture, the 3×3 convolutional layers use 3D filters. Subsampling is done by the 2×2 pooling layers with the factor of 2. FIGURE 9Open in figure viewerPowerPoint The architecture of the VGG-Face CNN Images of the micro-expression databases are stored with the name of the particular plane's number. These planes are the ones with the most efficiency in the detection of micro-expression using cubic-LBP. We apply the VGG-face model which takes micro-expression samples to recognize labelled images. In other words, like what happens in supervised learning, samples are trained with their corresponding labels. Hence, particular planes within the cubic-LBP which lead to better apex spotting are chosen for each sample. Finally, we use sigmoid plus cross-entropy for the task of particular plane(s) recognition. The classifier takes a sample (one image) as an input and gives the plane numbers which are encoded as output. Also, we consider pre-trained VGG-16 CNN [46] for micro-movement identification in the volcano, micro-saccades, and vessel sequential images. Again, we do the same procedure to come up with the scarce numbers of available datasets. The size of the input image is set to be 224×224 pixels with RGB channels. The input pixels are then converted to a NumPy array for working in Keras with the Tensor flow backend. To train these novel samples, we employ the transfer learning method. The VGG-16 model is illustrated in Figure 10. This model is characterized by depths of twenty-three and layers of sixteen. The last FC layer is replaced with a new FC one for our classification model. Meanwhile, the images are manually tagged. In addition, the 10-fold cross-validation is used to validate the results. The accuracy is obtained using categorical cross-entropy. FIGURE 10Open in figure viewerPowerPoint The VGG-16 for micro-movement identification In general, each normalized image and its plane number representing the direction of micro-movement are the CNN input. Thus, the CNN learns the direction of micro-movement to identify the plane(s) that show the true micr