# Cost-Effective CNNs for Real-Time Micro-Expression Recognition
## 3 Methodology  
The studies applying deep learning to tackle the ME classification problem [30; 31; 32; 33] usually used pretrained CNNs such as ResNet [18] and VGG [19] and applied transfer learning to obtain ME features. In our work, we first selected off-the-shelf ResNet18 because it provided the best trade-off between accuracy and speed on the challenging ImageNet classification and was recognized for its performance in transfer learning. ResNet [18] explicitly lets the stacked layers fit a residual mapping. Namely, the stacked non-linear layers are let to fit another mapping of \(F(x):=H(x)-x\) where \(H(x)\) is the desired underlying mapping and \(x\) the initial activations. The original mapping is recast into \(F(x)+x\) by feedforward neural networks with shortcut connections. ResNet18 has 20 Convolutional Layers (CLs) (17 successive CLs and 3 branching ones). Residual links after each pair of successive convolutional units are used and the kernel size after each residual link is doubled. As ResNet18 is designed to extract features from RGB color images, it requires inputs to have 3 channels. In order to accelerate processing speed in the deep learning domain, the main current trend in decreasing complexity of CNN is to reduce the number of parameters. For example, [PERSON] et al. [34] proposed a very compact LiteFlowNet which is 30 times smaller in the model size and 1.36 times faster in the running speed in comparison with the state-of-the-art CNNs for optical flow estimation. In [35], [PERSON] et al. explored parameter-reduced residual networks on in-the-wild datasets, targeting real-time head pose estimation. They experimented with various ResNet architectures with a varying number of layers to handle different image sizes (including low-resolution images). The optimized ResNet achieved state-of-the-art accuracy with real-time speed. It is well known that CNNs are created for specific problems and therefore over-calibrated when they are used in other contexts. ResNet18 was made for end-to-end object recognition: the dataset used for training had hundreds of thousands of images for each class and more than a thousand classes in total. Based on that: (i) An ME recognition study considers at most 5 classes, and the datasets of spontaneous MEs are scarce and contain far fewer samples, and (ii) optical flows are high-level features contrary to low-level color features and so require shallower networks; we have empirically reduced the architecture of ResNet18 by iteratively removing residual layers. This allowed us to assess the influence of the depth of the network on its classification capacities in our context and therefore to estimate the relevant calibration of the network. Figure 2 illustrates the reduction protocol: at each step the last residual layer with two CLs is removed and the previous one is connected to the fully connected layer. Only networks with an odd number of CL are therefore proposed. The weights of all CNNs are pretrained using ImageNet. As highlighted in Table 1, the decrease in the number of CLs has a significant impact on the number of learnable parameters of the network, which directly affects the forward propagation time. Once the network depth has been correctly estimated, the dimensions of the input have to be optimized. In our case, CNNs take optical flows extracted between the onset and apex frames of \begin{table}
\begin{tabular}{c c c c c c c c c c} \hline \hline
**CL** & 17 & 15 & 13 & 11 & 9 & 7 & 5 & 3 & 1 \\ \hline
**Nb. of param.** & 10,670,932 & 5,400,725 & 2,790,149 & 1,608,965 & 694,277 & 398,597 & 178,309 & 104,197 & 91,525 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Number of CLs and the number of learnable parameters in the proposed architectures. Figure 2: Depth reduction of a deep neural network: in the initial network, each residual layer contains two Convolutional Layers (CLs) (**left**); the last residual layer is removed (**middle**) to obtain a shallower network (**right**). ME video clips.