## 1 Introduction  
Facial expression as a carrier of emotion conveys rich behavior information [1]. Therefore, facial expression recognition (FER) has been a hot topic, and attracted attention in many fields, including human-computer interaction [2], security [3], and biometrics [4]. In early studies, FER methods focused on the still images, which did not consider the motion information of facial expression [5]. Since facial expression is a dynamic behavior, only employing still images is not sufficient for recognizing facial expressions. Presently, there are some traditional methods of extracting the facial expression dynamic features. For example, Histogram of Oriented Gradient to 3D (3D HOG) [6], as the extension of HOG, extracts the local temporal features. Most current FER systems capture images/videos in the visible light spectrum (VIS) (380 nm-750 nm) [7]. However, different environments produce huge variation for FER [8]. For example, the varying illumination conditions, to some extent, weaken the performance of FER significantly. Though pre-processing algorithms to ease the influence of illumination have been adopted in some works recently [9], it is still difficult to be applied in different circumstances and obtain satisfying results. An alternative method for solving the environment illumination variation problem is to use near-infrared (NIR) (780 nm-1100 nm) camera for facial expression recognition. The NIR camera with integrated NIR illumination sources were mounted in front of the users. The NIR light emitted from the camera has a much higher intensity than the ambient NIR light (because ambient NIR is very week). By using this kind of active light source, the ambient illumination variation problem could be solved, as long as the artificial illumination is constant, and the FER may be performed even in dark illumination conditions [10]. NIR system has applied to many research areas for solving illumination problem. For validating the idea of NIR FER, [PERSON] et al. [11] collected an NIR facial expression database, called Oulu-CASIA NIR facial expression database, and utilized improved LBP-TOP (Local binary patterns from three orthogonal planes) capturing the dynamic local information from the NIR video sequences. [PERSON], [PERSON] et al. [12] proposed a NIR face recognition method based on Zernike moments (ZMs) and Hermite kernels (HKs). [PERSON], [PERSON] et al. [13] modeled the pupil's response to light by using a nonlinear delay differential equation in the NIR spectral band, which can be used to develop robust iris recognition algorithms. [PERSON], [PERSON] et al. [14] captured NIR images and visible color images and presented a new NIR dehazing model based on color regularization, which can solve the color distortion and haze degradation problems at the same time. Like FER in VIS, most NIR FER methods so far have focused on analyzing still images. Though there do exist some traditional methods extracting the facial expression temporal features [51] from NIR video sequences, these methods need to extract features manually. The facial muscle motion contains rich information that represents the facial expression change [15]. According to the authors of [16], video sequences are more suitable than still images for FER. Therefore, it is very necessary to make use of dynamic facial expression representations (temporal features) in video sequences for FER. Nowadays, Convolutional Neural Networks (CNNs) with strong self-learning ability have been proved to be an effective method for action recognition [17], segmentation [18] and detection [19]. CNN can be further extended to 3D CNN [20], which can extract temporal feature in video sequences. [PERSON], [PERSON] et al. [21] designed a network for extracting the spatial and temporal features separately from human action video, and achieved an outstanding performance for recognizing human action on Sports-1M dataset. [PERSON] et al. [22] proposed a 3D CNN for extracting spatio-temporal features from video sequences.