Mutual information guided 3D ResNet for self‐supervised video representation learning

IET Image ProcessingVolume 14, Issue 13 p. 3066-3075 Research ArticleFree Access Mutual information guided 3D ResNet for self-supervised video representation learning Fei Xue, Fei Xue School of Electronic Engineering, Xidian University, Xi'an, People's Republic of ChinaSearch for more papers by this authorHongbing Ji, Corresponding Author Hongbing Ji hbji@xidian.edu.cn School of Electronic Engineering, Xidian University, Xi'an, People's Republic of ChinaSearch for more papers by this authorWenbo Zhang, Wenbo Zhang School of Electronic Engineering, Xidian University, Xi'an, People's Republic of ChinaSearch for more papers by this author Fei Xue, Fei Xue School of Electronic Engineering, Xidian University, Xi'an, People's Republic of ChinaSearch for more papers by this authorHongbing Ji, Corresponding Author Hongbing Ji hbji@xidian.edu.cn School of Electronic Engineering, Xidian University, Xi'an, People's Republic of ChinaSearch for more papers by this authorWenbo Zhang, Wenbo Zhang School of Electronic Engineering, Xidian University, Xi'an, People's Republic of ChinaSearch for more papers by this author First published: 08 October 2020 https://doi.org/10.1049/iet-ipr.2020.0019Citations: 1AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinked InRedditWechat Abstract In this work, the authors propose a novel self-supervised learning method based on mutual information to learn representations from the videos without manual annotation. Different video clips sampled from the same video usually have coherence in the temporal domain. To guide the network to learn such temporal coherence, they maximise the mutual information between global features extracted from different clips sampled from the same video (Global-MI). However, maximising the Global-MI leads the network to seek shared content from different video clips and may make the network degenerate to focus on the background of the video. Considering the structure of the video, they further maximise the average mutual information between the global feature and local patches of multiple regions of the video clip (multi-region Local-MI). Their approach, which is called Max-GL, learns the temporal coherence by jointly maximising the Global-MI and multi-region Local-MI. Experiments are conducted to validate the effectiveness of the proposed Max-GL. Experimental results show that the Max-GL can serve as an effective pre-training method for the task of action recognition in videos. Additional experiments for the task of action similarity labelling and dynamic scene recognition also validate the generalisation of the learned representations of the Max-GL. 1 Introduction Convolutional neural networks (CNNs) have demonstrated state-of-the-art performance for various computer vision tasks, such as image classification [1], action recognition [2, 3], and image segmentation [4]. However, the success of CNNs heavily relies on large amounts of data with a manual annotation such as ImageNet [5]. Such dependence severely limits the use of CNNs, as manual annotations are expensive and time-consuming. In contrast, there are a lot of unlabelled images and videos available on the Internet, which provide rich data for the training of CNNs. Thus, it is of great importance to study representation learning by employing the data without manual annotation. In this work, we focus on representation learning from unlabelled videos. Faced with the challenge of using unlabelled videos in the learning process, self-supervised learning has drawn much attention recently. The key idea of self-supervised learning is using the inherent structures of the raw videos to generate supervisory signals to train the network. Videos contain spatial structures in each frame and dynamic temporal structures between frames, and these structures are indispensable for formulating self-supervised tasks. Current self-supervised approaches leverage sequence order verification [6–8], tracking [9], future frame prediction [10] in videos as regularisation for the representation learning process. Given a video, different video clips sampled from it usually have similar content, such as similar background and consistent motion, which constitute the temporal coherence in videos. In [11, 12], temporal coherence is used as regularisation in the learning process. Though the above approaches do not use semantic labels, they adopt effective supervisory signals to train the network and guide the network to understand the videos. In these approaches, the network is first pre-trained using videos without manual annotation. Then, the learned representations are transferred to other tasks, or be adopted as the initialisation for other supervised tasks such as action recognition. Mutual information is a quantity to measure the relationship between two variables and can also be used to constrain two variables to be more relevant in turn. In this study, we explore the mutual information as a supervisory signal for the self-supervised video representation learning. However, mutual information is challenging to compute, especially for high-dimensional data. Fortunately, recent studies [13, 14] enable the effective estimation of mutual information between high-dimensional vector pairs via deep neural networks. Furthermore, ResNet [15] is extended to 3D ResNet for feature extraction in our approach. To learn the temporal coherence, we first sample two video clips from the same video. Then we extract the global features by feeding the two video clips after data augmentation into the 3D ResNet. Next, we maximise the mutual information between the two global features (i.e. Global-MI). Global-MI maximisation leads the network to seek shared content from different video clips and is not sufficient for the network learning useful representations. Different video clips sampled from the same video usually have a similar background. If we only maximise the Global-MI, the network may focus on the background and ignore the senior semantic information. To solve this problem, we maximise the average mutual information between the global feature and local patches of the corresponding input (i.e. Local-MI). Local-MI maximisation favours the network learning the data that is shared across patches. Furthermore, we also jointly maximise the average mutual information between the global feature and local patches belonging to another region of the video clip, where another region is obtained by repeatedly applying the data augmentation to the video clip. This is inspired by the human vision system [16] that one can infer the full visual scene through part of it. As shown in Fig. 1, given an image of a man playing guitar, we can generate different regions of the image by applying different data augmentation (e.g. multi-scale random cropping). No matter which region we choose, it is easy to determine that there is a man playing guitar in the image. Maximising the Local-MI of other regions encourages the network to distill information, which is shared by different regions of the video clip and further avoids the network degenerate to focus on the background. We denote maximising the average mutual information between the global feature and local patches of multiple regions of the video clip as multi-region Local-MI. Fig. 1Open in figure viewerPowerPoint Inspiration of the multi-region Local-MI. The original image shows a man playing guitar (a), (b), (c), (d) represent different regions of the original image which are obtained through cropping. No matter which sub-image we choose, it is easy to determine that there is a man playing guitar. Maximizing the multi-region Local-MI could make the network discover the relationship between different regions The novel task we proposed that jointly maximising the Global-MI and multi-region Local-MI is referred to as Max-GL. The Max-GL is used to guide the learning process of the 3D ResNet. A series of experiments are conducted to demonstrate the effectiveness of the Max-GL. When used as a pre-training approach for the task of action recognition in videos, the Max-GL improves dramatically compared with random initialisation on UCF101 [17] and HMDB51 [18] action benchmark datasets. We also demonstrate the generalisation of the learned representations through transferring the Max-GL to the task of action similarity labelling on ASLAN [19] dataset and dynamic scene recognition on YUPENN [20] and Maryland [21] datasets. In summary, the main contributions of our work are as follows: We propose a novel self-supervised task Max-GL for the representation learning from the unlabelled videos. In this task, the Global-MI and multi-region Local-MI are jointly maximised to learn the temporal coherence in videos. In addition, a neural network architecture is designed for the Max-GL, including the feature extractor and the network for maximising the Global-MI and multi-region Local-MI. The experimental results demonstrate that our proposed Max-GL can serve as an effective pre-training method for the task of action recognition in videos. Furthermore, experiments of the action similarity labelling and dynamic scene recognition show that the representations learned by the Max-GL can transfer to other visual tasks. The remainder of this paper is organised as follows. Section 2 gives a review of the related works. The proposed method is introduced in Section 3. To evaluate the performance of our method, experiments are performed in Section 4. Section 5 presents the conclusions. 2 Related work In this section, the most relevant works are reviewed in three categories: self-supervised representation learning, information-based methods, and action recognition in videos. 2.1 Self-supervised representation learning CNNs have exhibited dominant performance in high-level computer vision tasks such as image classification, image segmentation, and action recognition in videos. However, training deep networks usually needs enormous amounts of data with manual annotations. To reduce the thirst for annotated data when training the network in a fully supervised manner, there have been lots of research on unsupervised learning. A recently emerging approach is self-supervised representation learning, where the supervision signals are generated automatically by the inner structures of raw images or videos. Various self-supervised approaches utilise spatial context for supervision, including re-ordering disturbed image patches [22, 23], impainting the missing parts of images [24], converting greyscale images to colour images [25], recognising image rotations [26], counting visual primitives [27] etc. Compared with static images, there is an additional time dimension and dynamic information in the temporal domain for videos. Thus, both the spatial context and temporal context could be used to generate the supervision signals for training the networks. Misra et al. [6] train a network to decide whether the sequence of frames is in the right order. Similar to [6], Lee et al. [8] and Fernando et al. [7] also utilise the frame orders from a video to learn representations in a self-supervised manner. In [9], tracking is used to capture the dynamic features of moving objects. In [28], Gan et al. proposed a geometry guided network to learn flow fields between two input frames. Kim et al. [29] proposed to learn representations from unlabelled videos by extending the 2D-puzzle tasks [23] into 3D space. Vondrick et al. [30] proposed a generative adversarial network (GAN) model based on 3D convolution [31] to generate videos, by which representations can be learned without semantic labels. In [11], Mobahi et al. proposed the temporal coherence (TempCoh), which forces the learned representations to be smooth in the temporal domain by minimising the L2 distance of the features extracted from two consecutive frames. In [12], Jayaraman and Grauman proposed a model that requires the representations of different frames from a video change slowly and steady by minimising the higher-order derivatives of the features. The authors of [11, 12] utilise the temporal coherence to design tasks for learning representations in a self-supervised manner. However, they focus on the short-term change between adjacent frames. Given two frames with a long interval, the authors of [11, 12] fail to discover the connection between them. Compared with [11, 12], our work could learn the temporal coherence regardless of the sampling interval. 2.2 Information-based methods Mutual information is a measure of mutual dependence between the two variables in information theory and it can also constrain two variables to be more relevant. Self-supervised approaches based on mutual information have a long tradition. Linsker et al. [32] and Bell and Sejnowski [33] propose the infomax optimisation principle by maximising the mutual information between the input and output, but it is hard to be applied to deep networks. Several recent works have explored mutual information to train the deep networks. In [34], IMSAT maximises the mutual information between the data and its learned features. Mutual information neural estimation (MINE) [13] estimates the mutual information through a neural network and has a strong generalisation ability. In [14], Hjelm et al. propose the deep InfoMax (DIM) by maximising the mutual information between the input and the output of a deep neural network. Since DIM is more concerned with maximising the mutual information than estimating the exact values, an estimator based on Jensen–Shannon divergence (JSD) instead of Kullback–Leibler (KL)-based formulation is used for mutual information estimation in DIM. In [35], Ji et al. proposed the invariant information clustering (IIC) for unsupervised image clustering and segmentation by maximising the mutual information between image association pairs or spatial region pairs. Inspired by the success of mutual information achieved in the field of image processing, we use mutual information to measure the temporal coherence in videos and develop a self-supervised task for video representation learning. 2.3 Action recognition in videos Early approaches for the task of action recognition in videos mainly rely on hand-crafted features, such as trajectories [36] and space-time interest points [37]. Recently, following the great success of CNNs, deep models have drawn much attention in action recognition. Simonyan et al. [3] propose the two-stream network to incorporate the red–blue–green (RGB) frames with the optical flow as. In [2], Wang et al. proposed the temporal segment networks (TSNs) to model the long-term temporal structures of the video. Although the above works achieve state-of-the-art performance, they heavily depend on the weights pre-trained on ImageNet [5]. Different from these works, we fine-tune the network which is pre-trained only on the UCF101 dataset through the Max-GL. Furthermore, our proposed Max-GL and the above works have different goals. The target of the other works is to improve recognition accuracy as high as possible and the Max-GL aims to reduce the dependence of the weights pre-trained on large datasets with manual annotation. 3 Method We develop a novel task for self-supervised representation learning from unlabelled videos by simultaneously maximising the Global-MI and multi-region Local-MI. This task is inspired by the temporal coherence in videos and properties of the mutual information. In this section, we first illustrate the method used to maximise mutual information. Next, we introduce how the global feature and the local feature are extracted. Finally, we describe the details of the Global-MI maximisation and multi-region Local-MI maximisation in our proposed Max-GL. 3.1 Mutual information maximisation The method of maximising mutual information used in our approach follows MINE [13] and DIM [14]. Given two random variables X and Y, mutual information between X and Y is formulated by (1)where H denotes the Shannon entropy and denotes the conditional entropy of Y given X. Mutual information equals to the KL-divergence between the joint and the product of the marginals (2)where is defined as (3)In MINE, a lower-bound based on the Donsker–Varadhan representation [38] of the KL-divergence is used to estimate mutual information as follows: (4)where is defined as (5)where is a discriminator which has parameter and can be implemented by a neural network, x and y are sampled from X and Y. For the tasks in our study, maximising the mutual information is more important than estimating the precise value. However, may generate a large value and it is not conducive to the network optimisation when maximising the mutual information. Equation (2) indicates that the essence of mutual information is the distance between two distributions and it can be measured by JSD. Compared with the KL-divergence, JSD has an upper bound of and is defined as (6) Following the f -divergence representation in f -gan [39], JSD-based mutual information estimator is formulated as (7)where denotes the sigmoid function. We can maximise the mutual information between X and Y by minimising . The goal of mutual information maximisation encourages the discriminator to give a high value for the positive samples and a low value for the negative samples. As described in Algorithm 1 (see Fig. 2), multi-pairs of positive samples and negative samples are required for mutual information maximisation and it is implemented through multiple batches in our work. For the ith batch, is computed. We average the n results and get assumed that the batch size is n. For simplicity, the computation of is written as in the following sections, where (x,y) denotes positive pair and the negative pair. Fig. 2Open in figure viewerPowerPoint Algorithm 1: JSD-based mutual information maximisation 3.2 Feature extraction from 3D ResNet In this section, we introduce how the features are extracted and illustrate the network architecture of the feature extractor. Taking advantage of 3D CNNs for learning spatio-temporal information of the video, we extend ResNet-18 to 3D-ResNet-18 for feature extraction. All 2D convolutional kernels in ResNet-18 are replaced with . Particularly, temporal max pooling is not used in our 3D-ResNet-18. Parameters of the 3D-ResNet-18 are denoted as . Architecture of the 3D-ResNet-18 is shown in Fig. 3. Fig. 3Open in figure viewerPowerPoint Network architecture of the 3D-ResNet-18, whose input is the stack of 16 consecutive RGB frames Given a video with T frames, we denote it as , where represents an RGB image (three channels), w and h denote the width and height of the frames in the video. Similar to convolutional 3D (C3D) [31], we first sample a clip that consists of 16 consecutive frames from the video. Then, we resize the frames after data augmentation to and concatenate them as the input (). Next, the input is fed into the 3D-ResNet-18. We refer to the output of the intermediate layer of the 3D-ResNet-18 as the local feature , where represents the size, d the number of channels, and L the temporal dimension. We denote the local feature S as , . According the spatial location of each feature vector, is denoted as , . Global average pooling across the output of the last layer, with size , is adopted to produce a 512-dimensional feature vector, which is denoted as the global feature. In our approach, the local feature S is used to reflect the spatial structures of the video clip [40]. In the following experiments, the output of the 3D-conv4 layer is chosen, which has the dimension of . Although outputs of other intermediate layers (e.g. 3D-conv5 and 3D-conv3) can also be explored, the layer of 3D-conv4 is considered for efficiency. 3.3 Max-GL Temporal coherence widely exists in various videos and can be used as a supervisory signal for the learning process with unlabelled videos. In this work, our proposed Max-GL, which learns the temporal coherence, is measured by the mutual information by maximising the Global-MI and multi-region Local-MI. In the following, we first describe the Global-MI maximisation (Section 3.3.1) and the multi-region Local-MI maximisation (Section 3.3.2). Then, we introduce how the Global-MI maximisation and multi-region Local-MI maximisation are combined in the Max-GL (Section 3.3.3). 3.3.1 Maximise Global-MI Global-MI is explored to measure the relationship between two video clips. Maximising the Global-MI guides the network to seek shared content between these video clips. However, if the two video clips are randomly sampled, the sampling interval is hard to control and it may cause that the sampled clips are too close each time we sample. This is of no benefit for the network to understand the dynamics of the whole video. To model the long-range temporal structures of the video, we divide the video into three segments of equal durations inspired by [2] and sample one clip from each segment. Then the average Global-MI between the three video clips is maximised. As shown in Fig. 4, are sampled from different segments of one video, and clip f is sampled from another video. Then the four clips are fed into the quadruplet Siamese 3D-ResNet-18 to extract the global features . Next, pairwise features including positive pairs and negative pairs are taken as input to the discriminator independently to predict which pair comes from the same video, where is the parameters of the discriminator . The objective of Global-MI maximisation is to train of the 3D-ResNet-18 and of the together. The loss for maximising the average Global-MI between the three video clips is represented as (8) (9) (10) (11) is implemented with a fully-connected network whose architecture is introduced in Section 4.2. Fig. 4Open in figure viewerPowerPoint Illustration of the Global-MI maximisation. A quadruplet siamese architecture is used to extract the global features 3.3.2 Maximise multi-region Local-MI The objective in (8) can be used to guide the network to learn shared content between the sampled video clips, but this may be not enough for the network to learn the temporal coherence of the video. Different video clips sampled from the same video usually have a similar background, and only maximising the Global-MI may cause the network to focus on the background and ignore the high-level semantic information. To address this issue, we consider the structures of the video clip and maximise the average mutual information between the global feature and local patches of multiple regions (multi-region Local-MI). First, Local-MI maximisation (single-region) encourages the global feature learned from the 3D-ResNet-18 to have high mutual information with every patch of the input. If the network only passes information specific to some parts of the video clip (e.g. background), this will decrease the mutual information between the global feature and other patches of video. Second, there exists a close relationship between different regions of the video clip, which are generated by repeatedly applying data augmentation to the video clip. Through maximising the multi-region Local-MI, the network could not only distill the information shared across patches but also across regions of the video clip. This can further avoid the network focus on the background. As the local features S reflects the spatial locality of the video [40], maximising the Local-MI can be implemented by maximising the mutual information between the global feature G and local feature vector in S. Local feature is produced by feeding another region of the same video clip into the 3D-ResNet-18 and feature vectors in are denoted as . Maximising the Local-MI of another region is fulfilled through maximising the average mutual information between S and . The framework of multi-region Local-MI maximisation is shown in Fig. 5. First, we generate two regions of the video clip through different data augmentation. Global feature G and local feature S belong to the same region of the video clip, and belongs to another region. To maximise the multi-region Local-MI, negative local feature is also extracted from another video. Then, we feed , and into the discriminator to get scores of the three pairwise features, where , , , and denotes the parameters of . Next, we optimise the following loss to maximise the multi-region Local-MI: (12) (13) (14) Fig. 5Open in figure viewerPowerPoint Illustration of the multi-region Local-MI maximisation. See explanations in Section 3.3.2 for more details As shown in Fig. 5, the discriminator needs to distinguish 'true' or 'false' for each pair of feature vectors. Fig. 6 shows that 3D CNNs with kernel size is used to score the pairwise feature vectors. We concatenate the global feature with the local feature at every location. Then, the feature after concatenation is fed into the to get the scores that have a dimension of . The architecture of is introduced in Section 4.2. Fig. 6Open in figure viewerPowerPoint Concatenate-and-convolve architecture for the discriminator in the multi-region Local-MI maximisation. The global feature vector is concatenated with the local feature vector at every location. A convolutional network is employed to score each pairwise feature 3.3.3 Jointly maximise Global-MI and multi-region Local-MI Different from Global-MI, multi-region Local-MI mainly focuses on the learning process within one video clip and any clip of the video can accomplish this task. To reduce the amount of computation, one clip is randomly selected from for the multi-region Local-MI maximisation. We assume that clip2 is selected in Fig. 6. It should be noted that one clip is randomly selected every time we maximise the multi-region Local-MI. The final optimisation objective incorporates the losses in the Global-MI maximisation (8) and multi-region Local-MI maximisation (12) into the following term: (15)where denotes all the parameters of the model, the coefficient of weight decay, and are the weights of different losses. Equation (13) is employed to train the whole model including 3D-ResNet-18, and (see Fig. 7). Fig. 7Open in figure viewerPowerPoint Architecture of the Max-GL, which jointly maximises the Global-MI and multi-region Local-MI. We assume that clip 2 is selected for the multi-region Local-MI maximisation 4 Experiments In this section, the effectiveness of our proposed Max-GL is evaluated through a series of experiments. First, we introduce the datasets used and implemented details of the experiments. Then, we analyse what does the Max-GL guide the network learns through nearest neighbours retrieving. Next, we treat the Max-GL as a pre-training method for the task of action recognition in videos. Besides, we also compare the Max-GL with other self-supervised learning tasks on the UCF101 [17] and HMDB51 [18] datasets. Finally, we conduct experiments on ASLAN [19], YUPENN [20], and Maryland [21] datasets for the task of action similarity labelling and dynamic scene recognition to validate the transferability of the learned representations. 4.1 Datasets Experiments are conducted on five datasets: UCF101 [17], HMDB51 [18], ASLAN [19], YUPENN [20], and Maryland [21]. Unless stated otherwise, the UCF101 dataset is used for the pre-training of our model. UCF101 dataset contains about 13 K videos collected from YouTube with 101 action categories. The number of videos for training is about 9.5 K and testing is about 3.5 K. Action categories in UCF101 range from daily life activities to sports. The task of action recognition for UCF101 is challenging due to the changeable poses of humans, light conditions, different camera views, camera movement etc. UCF101 has three splits for training and testing. HMDB51 dataset consists of 7 K videos with 51 action categories. Videos in HMDB51 are mainly collected from movies. Compared with UCF101, HMDB51 has more complex backgrounds and less training data. HMDB51 also consists of three splits for training and testing. ASLAN dataset contains about 3.6 K videos with 432 action categories, about eight videos per category on average. Since there are too few samples in each category, this dataset is not suitable for the task of action recognition. ASLAN is mainly used for the task of action similarity labelling. YUPENN dataset has 420 videos covering 14 dynamic scene categories and Maryland dataset has 130 videos covering 13 scene categories. Both datasets are not related to human action and used for the task of dynamic scene recognition. 4.2 Implemented details Architectures of discriminators and For the Global-MI maximisation, we first concatenate two global features of different clips as input (1024) and then feed it into a fully-connected network (see Table 1) to get the score. Table 1. Architecture of for the objective of maximising Global-MI. Operation Size Activation input linear layer 512 ReLU linear layer 512 ReLU linear layer 1 — As to the discriminator , we concatenate the global feature with local feature map () at every location as the input (). Then, an convolution is used to get the scores (). The architecture of is shown in Table 2. Layer normal