# Research on Animated GIFs Emotion Recognition Based on ResNet-ConvGRU

Authors: Qian Zhang; Ren Qing-Dao-Er-Ji; Na Li
Journal: Mathematical Problems in Engineering
 (2022)

---

## Abstract

Animated Graphics Interchange Format (GIF) images have become an important part of network information interaction, and are one of the main characteristics of analyzing social media emotions. At present, most of the research on GIF affection recognition fails to make full use of spatial-temporal characteristics of GIF images, which limits the performance of model recognition to a certain extent. A GIF emotion recognition algorithm based on ResNet-ConvGRU is proposed in this paper. First, GIF data is preprocessed, converting its image sequences to static image format for saving. Then, the spatial features of images and the temporal features of static image sequences are extracted with ResNet and ConvGRU networks, respectively. At last, the animated GIFs data features are synthesized and the seven emotional intensities of GIF data are calculated. The GIFGIF dataset is used to verify the experiment. From the experimental results, the proposed animated GIFs emotion recognition model based on ResNet-ConvGRU, compared with the classical emotion recognition algorithms such as VGGNet-ConvGRU, ResNet3D, CNN-LSTM, and C3D, has a stronger feature extraction ability, and sentiment classification performance. This method provides a finer-grained analysis for the study of public opinion trends and a new idea for affection recognition of GIF data in social media.


---

## Body

## 1. Introduction

Emotion recognition is one of the important research directions in affective computing, and its primary task is to identify people’s emotional states through their facial expressions, behaviors, or physiological signals. In recent years, with the rapid development of social media, such as Weibo, forums, and TikTok, people are tending to express their personal feelings through various images and video data on social platforms. It is of great applicable value in box office earnings forecast [1], political elections [2], commodity recommendations [3], medical treatment [4], and education assistance [5] in the ways of mining valuable information from massive data and classifying the expressed affections and emotions.Among them, animated Graphics Interchange Format (GIF) image as a significant data form of emotional expression, has been widely used on various social platforms. GIF is a shorter sequence of image data that is lighter than video and is sometimes referred to as a GIF short video. Compared to the emotion recognition of electroencephalogram (EEG), the GIF data has the advantages of convenient data collection and intuitive affection expression; compared to the emotion recognition of image, it can express more lively and convey much more information. The GIF data of mood changes is shown in Figure1. Undeniably, GIF data has its unique advantages. Therefore, emotion recognition of GIF animated data is of great academic value and broad practical significance [6].Figure 1
GIF graphs of emotional changes (example).The early research on image emotion recognition mainly used the method based on manual features, which was composed of traditional image processing, pattern recognition, and classifiers, including the targeted determination such as histograms of oriented gradients (HOG), K-nearest neighbor algorithm (KNN), support vector machine (SVM), and other classification methods [7]. With the wide application of deep learning in affective computing, deep learning is applied to emotion recognition tasks by many experts and scholars. The emotion recognition method based on deep learning mainly extracts data features through convolution operation in the neural networks, including architectures such as GoogleNet, ResNet, VGGNet, and AlexNet [8]. This method has become the mainstream method of image emotion recognition because of its strong generalization ability and high accuracy of emotion recognition.There are fewer related research studies at present because GIF data is widely used later than pictures, and the data features are more complex, the traditional convolution cannot make full use of the unique attributes of GIF to extract features. It has inspired us to build a model based on ResNet-ConvGRU to complete the emotion recognition task of animated GIFs referring to the algorithm idea of combining convolution feature extraction in pictures with spatial-temporal characteristics extraction in the video. The advantages of the ResNet-ConvGRU model compared with the conventional CNN-GRU network structure are that the residual network has stronger feature extraction capability and the ConvGRU network has stronger sequence prediction and generalization capability. The combined framework of the two is more conducive to learning animated GIF spatio-temporal features, which in turn improves the efficiency of emotion recognition. In this paper, we introduce the theory of animated GIFs emotion recognition and research progress from the aspects of GIF data emotion recognition classification system, feature extraction, network construction, attribute extraction, and classification recognition and finally summarize and prospect the existing problems and future challenges.


## 2. Related Work

### 2.1. Emotional Classification System

Affections and emotions are people’s attitude experience and corresponding behavior reactions to objective things. Emotion classification is also called refined affection classification, which is because the early affection classification is usually divided into two categories of positive and negative, or three categories of positive, negative, and neutral, failing to meet people’s needs for the simulation of complex emotion changes. Compared with affection classification, the emotion classification can reflect people’s real psychological state in a certain period to a certain extent, hence, the study of emotion classification emerges from an opportunity. With the advancement of society, people’s cognition of emotion is constantly updated. There is no unified standard for the classification of emotion in the academic community, but it can be divided into discrete emotion and dimensional emotion according to the different classification methods. The researchers of discrete emotions believed that people can have specific emotions that vary according to mood changes, but the basic emotions are rare and can be defined in a variety of ways. In 1992, American psychologist Ekman and his colleagues proposed a basic system of six emotions, including happiness, sadness, anger, fear, disgust, and surprise, based on the in-depth study of facial expressions [9]. Then, Robert Plutchik proposed a multidimensional emotional model in 2001, which contains eight basic two-way emotions, including happiness, trust, fear, surprise, sadness, disgust, anger, and expectation [10]. Izard [11] put forward the theory that human beings had ten basic emotions including anger, contempt, happiness, shame, disgust, interest, surprise, sadness, fear, and guilt employing factor analysis. Researchers of dimensional emotion theory tried to identify two emotions or define emotions in more dimensions. They believed that it was more reasonable to analyze emotional states in the emotional space. Russell and Barrett [12] adopted a two-dimensional valence-arousal model aiming at emotion recognition results. Wherein the valence corresponds to the pleasure value of the emotion felt by the individual, and the arousal corresponds to the emotional intensity.We selected and sorted out seven emotion categories including happiness, like, surprise, sadness, anger, disgust, and fear based on the business requirements of most image emotion classification tasks and constructed the emotion corpus.


### 2.2. GIF Data Emotion Recognition Method

Jou et al. [13] firstly predicted the emotion in GIF and compared the influence of different types of features in GIF, including color histogram, aesthetics, intermediate semantic features inspired by affection modeling, and facial features on emotion recognition. The experimental results showed that the facial expression features had the highest accuracy in predicting 17 emotions compared with other features. Inspired by video analysis, Chen and Picard [14] employed a three-dimensional convolution neural network (CNN) to extract spatial-temporal characteristics from GIFs for emotion recognition, achieving good results. Yang et al. [15] proposed a key point attend visual attention network (KAVAN), and improved the traditional LSTM layer into an (HSLSTM) module to further improve the affection recognition performance of GIF. Liu et al. [16] adopted the network model of a combination with C3D and ConvLSTM for affection analysis of GIF short videos, achieving good experimental results. Lin [17] proposed an affection analysis method (RFCM) based on a 3D residual network (Res Net3D) and improved convolution long and short time memory network (FConvLSTM). As a result, it could effectively improve the effect of affection classification. Ma et al. [18] proposed a new method for animated GIFs content classification based on keyframe attention and entropy loss, reducing the interference of redundant frames to a certain extent, thus improving the classification efficiency of the model. The above-given algorithms had achieved good results in some aspects, but the GIF features were not fully utilized to extract its spatial features affecting the final GIF recognition efficiency. Although the transfer learning of the 3D convolution method in video recognition can be simultaneous to the extraction of the spatial-temporal characteristics of the GIF data, the 3D convolution can not obtain a longer time dependence. In addition, there are some problems in GIF data such as the large difference in image sequence length distribution and the inconsistent image size, bringing challenges to GIF emotion recognition. An animated GIFs emotion recognition algorithm based on ResNet-ConvGRU is proposed in this paper to solve the above-given problems. The efficiency of emotion recognition of GIF data is further improved by learning the spatial and temporal characteristics of GIF data, respectively.


## 2.1. Emotional Classification System

Affections and emotions are people’s attitude experience and corresponding behavior reactions to objective things. Emotion classification is also called refined affection classification, which is because the early affection classification is usually divided into two categories of positive and negative, or three categories of positive, negative, and neutral, failing to meet people’s needs for the simulation of complex emotion changes. Compared with affection classification, the emotion classification can reflect people’s real psychological state in a certain period to a certain extent, hence, the study of emotion classification emerges from an opportunity. With the advancement of society, people’s cognition of emotion is constantly updated. There is no unified standard for the classification of emotion in the academic community, but it can be divided into discrete emotion and dimensional emotion according to the different classification methods. The researchers of discrete emotions believed that people can have specific emotions that vary according to mood changes, but the basic emotions are rare and can be defined in a variety of ways. In 1992, American psychologist Ekman and his colleagues proposed a basic system of six emotions, including happiness, sadness, anger, fear, disgust, and surprise, based on the in-depth study of facial expressions [9]. Then, Robert Plutchik proposed a multidimensional emotional model in 2001, which contains eight basic two-way emotions, including happiness, trust, fear, surprise, sadness, disgust, anger, and expectation [10]. Izard [11] put forward the theory that human beings had ten basic emotions including anger, contempt, happiness, shame, disgust, interest, surprise, sadness, fear, and guilt employing factor analysis. Researchers of dimensional emotion theory tried to identify two emotions or define emotions in more dimensions. They believed that it was more reasonable to analyze emotional states in the emotional space. Russell and Barrett [12] adopted a two-dimensional valence-arousal model aiming at emotion recognition results. Wherein the valence corresponds to the pleasure value of the emotion felt by the individual, and the arousal corresponds to the emotional intensity.We selected and sorted out seven emotion categories including happiness, like, surprise, sadness, anger, disgust, and fear based on the business requirements of most image emotion classification tasks and constructed the emotion corpus.


## 2.2. GIF Data Emotion Recognition Method

Jou et al. [13] firstly predicted the emotion in GIF and compared the influence of different types of features in GIF, including color histogram, aesthetics, intermediate semantic features inspired by affection modeling, and facial features on emotion recognition. The experimental results showed that the facial expression features had the highest accuracy in predicting 17 emotions compared with other features. Inspired by video analysis, Chen and Picard [14] employed a three-dimensional convolution neural network (CNN) to extract spatial-temporal characteristics from GIFs for emotion recognition, achieving good results. Yang et al. [15] proposed a key point attend visual attention network (KAVAN), and improved the traditional LSTM layer into an (HSLSTM) module to further improve the affection recognition performance of GIF. Liu et al. [16] adopted the network model of a combination with C3D and ConvLSTM for affection analysis of GIF short videos, achieving good experimental results. Lin [17] proposed an affection analysis method (RFCM) based on a 3D residual network (Res Net3D) and improved convolution long and short time memory network (FConvLSTM). As a result, it could effectively improve the effect of affection classification. Ma et al. [18] proposed a new method for animated GIFs content classification based on keyframe attention and entropy loss, reducing the interference of redundant frames to a certain extent, thus improving the classification efficiency of the model. The above-given algorithms had achieved good results in some aspects, but the GIF features were not fully utilized to extract its spatial features affecting the final GIF recognition efficiency. Although the transfer learning of the 3D convolution method in video recognition can be simultaneous to the extraction of the spatial-temporal characteristics of the GIF data, the 3D convolution can not obtain a longer time dependence. In addition, there are some problems in GIF data such as the large difference in image sequence length distribution and the inconsistent image size, bringing challenges to GIF emotion recognition. An animated GIFs emotion recognition algorithm based on ResNet-ConvGRU is proposed in this paper to solve the above-given problems. The efficiency of emotion recognition of GIF data is further improved by learning the spatial and temporal characteristics of GIF data, respectively.


## 3. Model Introduction

An animated GIFs emotion recognition method based on ResNet-ConvGRU is proposed in this paper. The model structure is shown in Figure2 below. It mainly includes four modules: the input layer, the ResNet layer, the ConvGRU layer, and the emotion classification layer.(1)
The input layer preprocesses the source data including converting the GIF data to a static discrete image, i.e., the steps of gif converting to .png and adjusting the size of the image.(2)
The ResNet layer is used to extract the spatial features of static images.(3)
The ConvGRU layer is used to extract the temporal features of the animated GIFs image sequence. It takes the image spatial features extracted from the ResNet network layer and stitches them by column and inputs them to the ConvGRU layer to further extract the image sequence features.(4)
The emotion classification layer is used for classifying the total characteristics of the data with the Softmax function and outputting the result.Figure 2
Structure diagram of ResNet-ConvGRU model.


### 3.1. Inputs Layer

The input layer mainly preprocesses the input animated GIFs data, making it more standard and convenient for model training. First, the GIF short video is converted into an image format for saving, i.e., converting the suffix name of the data .gif to .png. The naming of static images is arranged according to time series convenient for the model to extract its temporal features. Second, all images are converted to RGB format. It is because the data in the data set is the gray image and the data decoding mode is gray image mode, the RGB image format needs to be further converted into three channels. Third, the image size is adjusted to be 224 × 224. Finally, canonical image data are input into the model.


### 3.2. Spatial Feature Learning

ResNet [19] is usually used to explore the local features in an image. The spatial feature of the image is extracted from the animated GIF with ResNet in this paper. The spatial feature learning of GIF images is adopted by the ResNet18 network because the deep network model may lead to insufficient training and degradation due to the small data set in this paper. Figure 3 shows the ResNet image spatial features extraction network structure diagram, including 17 convolution layers, 1 maximum pooling layer, 1 average pooling layer, and 1 full connection layer. The ResNet network layer in this paper is convolved by a small convolution kernel of 3 × 3 to extract the deep spatial features of the image.Figure 3
Structure diagram of ResNet network.Each residual unit in Figure3 is composed of two residual blocks. Where the residual block contains identity mapping and ReLU connection. The function of identity mapping is to establish a direct correlation channel between input and output. A function expression of ReLU connection:(1)Fx=W1σW2x,where σ is the nonlinear activation function ReLu, W1 and W2 represent the model parameters. When the numbers of channels of the input characteristic pattern and output characteristic pattern of the residual block are equal, the output of the residual block is(2)y=Fx,Wi+x.


### 3.3. Temporal Feature Learning

The traditional recurrent neural network (RNN) cannot process long sequence information well because of the problems of gradient disappearance and gradient explosion in training that is easier to occur. The Gated recurrent unit (GRU) is obtained during the improvement of the traditional recurrent neural network. As a variant of the Long Short-Term Memory(LSTM) networks, GRU only retains the reset gate and the update gate with the advantages of fewer parameters, a simple model, and fast convergence speed. Its unit structure is shown in Figure4.Figure 4
GRU structure.The ConvGRU [20, 21] network is used to extract the temporal features of the animated GIFs image sequence. This is because GIF images, unlike static images, have time-series features that change and can dynamically represent changes in human emotion. That is also the reason that makes it very difficult to extract the emotion features of GIF images. In this paper, the image features extracted by the ResNet network are stitched together and fed into the ConvGRU network, so as to learn the changes in the emotion features of GIF images in the temporal dimension. ConvGRU is an improvement based on retaining the advantages of the GRU structure. The activate function σ and full connection operation in tanh in its structure are replaced by a convolution operation. By doing so, the RNN’s ability to explore the inherent dependencies of sequence data can be inherited, and the underlying temporal characteristics of the data can be captured. The basic calculation formula of ConvGRU is as follows:(3)Zt=σWxz∗Xt+Whz∗Ht−1,Rt=σWxr∗Xt+Whr∗Ht−1,Ht′=fWxh∗Xt+Rt∘Whh∗Ht−1,Ht=1−Zt∘Ht′+Zt∘Ht−1,where Zt, Rt, and Ht stand for update gate, reset gate, and memory state, respectively. The update gate defines the amount of information that the emotional features preserved by the memory state are finally preserved to the ConvGRU network. The reset gate determines how the image space features output by the ResNet network are combined with the sentiment features stored in the memory state of the ConvGRU network. The memory state is used to preserve the image features learned by the ResNet network. Ht′ is the new image features obtained by aggregating the image features outputted by the ResNet network with the output of the previous hidden layer. ∗ represents convolution operation, ∘ stands for Hadamard product [22], f stands for activation function. Wxz, Whz, Wxr, Whr, Wxh, and Whh represent the training parameter matrix. In this paper, the output result of the ResNet network is used as the input of the ConvGRU layer through the superposition of the multilayer ConvGRU, and the time sequence relationship is established. The convolution kernel size of each layer of ConvGRU is 3×3.


### 3.4. Emotional Classification System

The emotion classification layer is mainly to classify the final emotion characteristics and output the results. The loss function is cross-entropy in this paper, and the Softmax function is used as the classification function in the full connection layer to output the prediction results.(4)L=1N∑iLi=−1N∑i∑c=1Myiclogpic,where M is the number of categories, here in this paper it presents a seven classified emotion task, thus M=7. yic is the symbolic function, the value is 0 or 1. When the real class of the sample i is equal to c, it is 1, otherwise, it is 0. pic is the prediction probability that the observed sample i belongs to the class c.(5)yi=softmaxWsofttanhWlFi∗+bl+bsoft,where Wl, bl stand for the weight and offset of the full connection layer, Wsoft and bsoft stand for the weight and offset of the classification layer softmax, yi represents the emotion classification result of the final data.


## 3.1. Inputs Layer

The input layer mainly preprocesses the input animated GIFs data, making it more standard and convenient for model training. First, the GIF short video is converted into an image format for saving, i.e., converting the suffix name of the data .gif to .png. The naming of static images is arranged according to time series convenient for the model to extract its temporal features. Second, all images are converted to RGB format. It is because the data in the data set is the gray image and the data decoding mode is gray image mode, the RGB image format needs to be further converted into three channels. Third, the image size is adjusted to be 224 × 224. Finally, canonical image data are input into the model.


## 3.2. Spatial Feature Learning

ResNet [19] is usually used to explore the local features in an image. The spatial feature of the image is extracted from the animated GIF with ResNet in this paper. The spatial feature learning of GIF images is adopted by the ResNet18 network because the deep network model may lead to insufficient training and degradation due to the small data set in this paper. Figure 3 shows the ResNet image spatial features extraction network structure diagram, including 17 convolution layers, 1 maximum pooling layer, 1 average pooling layer, and 1 full connection layer. The ResNet network layer in this paper is convolved by a small convolution kernel of 3 × 3 to extract the deep spatial features of the image.Figure 3
Structure diagram of ResNet network.Each residual unit in Figure3 is composed of two residual blocks. Where the residual block contains identity mapping and ReLU connection. The function of identity mapping is to establish a direct correlation channel between input and output. A function expression of ReLU connection:(1)Fx=W1σW2x,where σ is the nonlinear activation function ReLu, W1 and W2 represent the model parameters. When the numbers of channels of the input characteristic pattern and output characteristic pattern of the residual block are equal, the output of the residual block is(2)y=Fx,Wi+x.


## 3.3. Temporal Feature Learning

The traditional recurrent neural network (RNN) cannot process long sequence information well because of the problems of gradient disappearance and gradient explosion in training that is easier to occur. The Gated recurrent unit (GRU) is obtained during the improvement of the traditional recurrent neural network. As a variant of the Long Short-Term Memory(LSTM) networks, GRU only retains the reset gate and the update gate with the advantages of fewer parameters, a simple model, and fast convergence speed. Its unit structure is shown in Figure4.Figure 4
GRU structure.The ConvGRU [20, 21] network is used to extract the temporal features of the animated GIFs image sequence. This is because GIF images, unlike static images, have time-series features that change and can dynamically represent changes in human emotion. That is also the reason that makes it very difficult to extract the emotion features of GIF images. In this paper, the image features extracted by the ResNet network are stitched together and fed into the ConvGRU network, so as to learn the changes in the emotion features of GIF images in the temporal dimension. ConvGRU is an improvement based on retaining the advantages of the GRU structure. The activate function σ and full connection operation in tanh in its structure are replaced by a convolution operation. By doing so, the RNN’s ability to explore the inherent dependencies of sequence data can be inherited, and the underlying temporal characteristics of the data can be captured. The basic calculation formula of ConvGRU is as follows:(3)Zt=σWxz∗Xt+Whz∗Ht−1,Rt=σWxr∗Xt+Whr∗Ht−1,Ht′=fWxh∗Xt+Rt∘Whh∗Ht−1,Ht=1−Zt∘Ht′+Zt∘Ht−1,where Zt, Rt, and Ht stand for update gate, reset gate, and memory state, respectively. The update gate defines the amount of information that the emotional features preserved by the memory state are finally preserved to the ConvGRU network. The reset gate determines how the image space features output by the ResNet network are combined with the sentiment features stored in the memory state of the ConvGRU network. The memory state is used to preserve the image features learned by the ResNet network. Ht′ is the new image features obtained by aggregating the image features outputted by the ResNet network with the output of the previous hidden layer. ∗ represents convolution operation, ∘ stands for Hadamard product [22], f stands for activation function. Wxz, Whz, Wxr, Whr, Wxh, and Whh represent the training parameter matrix. In this paper, the output result of the ResNet network is used as the input of the ConvGRU layer through the superposition of the multilayer ConvGRU, and the time sequence relationship is established. The convolution kernel size of each layer of ConvGRU is 3×3.


## 3.4. Emotional Classification System

The emotion classification layer is mainly to classify the final emotion characteristics and output the results. The loss function is cross-entropy in this paper, and the Softmax function is used as the classification function in the full connection layer to output the prediction results.(4)L=1N∑iLi=−1N∑i∑c=1Myiclogpic,where M is the number of categories, here in this paper it presents a seven classified emotion task, thus M=7. yic is the symbolic function, the value is 0 or 1. When the real class of the sample i is equal to c, it is 1, otherwise, it is 0. pic is the prediction probability that the observed sample i belongs to the class c.(5)yi=softmaxWsofttanhWlFi∗+bl+bsoft,where Wl, bl stand for the weight and offset of the full connection layer, Wsoft and bsoft stand for the weight and offset of the classification layer softmax, yi represents the emotion classification result of the final data.


## 4. Experiment

The experimental codes are implemented on GPU-Nvidia Tesla P100, training with Python3.7 and Torch1.10.0.


### 4.1. Experimental Data

The GIFGIF data set was used in the experiment in this paper. This data of 17 emotion classifications were provided by MIT multimedia labs, including pleasure, disgust, happiness, excitement, embarrassment, surprise, fear, satisfaction, pride, anger, and guilt, a total of 6170 animated GIFs. The animated GIFs content covered many fields such as movies, TV programs, cartoons, and sports. The data label was pair-compared and voted from users’ emotions on more than 2.7 million animated GIFs content.We selected 7 out of 17 emotion categories including happiness, like (satisfaction), surprise, sadness, anger, disgust, and fear for a better experiment. The first 300 animated GIFs in each category were used, and a total of 2100 GIF data were used in the experiment. In the experiment, 80% of each category of data in the data set was selected as the training set, and the remaining 20% was selected as the test set.


### 4.2. Experimental Parameters and Evaluation Indicators

The experimental parameters are set as shown in Table1.Table 1
Experimental parameters.
ParametersValueParametersValuekernel_size(3,3)Loss functionCross entropyLearning rate0.0001Epoch100weight_decay1e-4Batch size30The function optimizedAdamNumber of categories7The evaluation indexes of the model were accuracy (Acc), precision (P), Recall (R), and F1 Score (F1) to measure the experimental effect of the model. Accuracy represented the proportion of the number determined to be correct to the total number:(6)Acc=TP+TNTP+TN+FP+FN,where TP was the number of true positive classes, TN was the number of true negative classes, FP was the number of misjudging the negative classes as positive classes, and FN was the number of misjudging the positive classes as negative classes. The following formula was the same. Precision was the proportion of the true positive class to the predicted positive class.(7)P=TPTP+FP.The recall rate refers to the proportion of positive class judged to the true positive class.(8)R=TPTP+FN.The F1 score was the harmonic average of accuracy and recall.(9)F1=2⋅P⋅RP+R.As shown above, the higher the accuracy, precision, recall, andF1 score, the better the performance of the emotion recognition model.


### 4.3. Experimental Analysis and Results

Animated GIF is a kind of image sequence data. Its image sequence number is unevenly distributed due to the different sources of the animated GIFs in the data set. As shown by the relevant experiments, the problems of uneven distribution of the number of image sequences and too long sequences all have different degrees of influence on the emotional recognition of animated GIFs. We made the following statistics on the number frequency distribution of all animated GIF image sequences in the experimental data to explore the distribution characteristics of animated GIF image sequences in the data set:Figure5 showed that most of the animated GIF sequences number used in the experiment were within 100, mostly, between 0 and 50. The shortest animated GIF image sequence was 2, and the maximum could be up to 303 in this paper according to the experimental data statistics. It could be seen that the number of animated GIF image sequences in this data set was relatively concentrated despite the large differences in the number of image sequences.Figure 5
Number frequency distribution diagram of the animated GIF image sequence.To alleviate the problem above that the number of animated GIF image sequences had a greater difference which influenced the emotion recognition efficiency, in this paper, after the animated GIF images were converted to static images for saving every 4 images were input into the model as an image sequence according to the time sequence and their spatial-temporal characteristics were analyzed, respectively. Data of image sequences that were less than 4 images were supplemented randomly, while longer image sequences were split, and input in multiple batches. Various animated GIF emotion recognition methods were compared to verify the proposed effectiveness of the animated GIF emotion recognition method based on ResNet-ConvGRU.Table2 showed the comparison of the evaluation indexes of GIF data of each emotion category in the emotion classification model based on ResNet-ConvGRU. It could be seen that the category of anger has a higher score on precision, indicating that the category had higher prediction accuracy in the positive sample results. It might be because the facial expressions and body movements of the animated GIF images of the anger emotion category were more obvious and easier to be distinguished than other emotions. While like had a higher score on recall, indicating that the probability of the actual emotion category being predicted was higher. It might be because the GIF image of like had a higher correlation between features, such as a tendency to contain obvious love and easier to be recognized. There were still gaps in the F1 score among the seven emotional categories, which indicated that there was a certain difference in the difficulty of recognizing various emotional features. Of course, it might be related to the animated GIF images themselves in the model training.Table 2
Animated GIF emotion recognition confusion matrix.
LabelPrecisionRecallF1 scoreAnger0.79760.63310.7059Disgust0.59590.64380.6189Fear0.63220.65770.6447Happiness0.63840.62150.6299Like0.69120.80790.7450Sadness0.71210.64030.6743Surprise0.65920.69400.6762In order to verify the advantages of the proposed model backbone network ResNet feature extraction, the proposed model is experimentally compared with a similar deep network VGGNet in this paper. It is found that the ResNet network used in this paper contains 17 convolutional layers and 1 fully connected layer, while the VGG19 network contains 16 convolutional layers and 3 fully connected layers Table3.Table 3
Comparison of ResNet-ConvGRU and VGGNet-ConvGRU.
ModelPrecisionRecallF1-scoreResNet-ConvGRU0.67520.67120.6707VGGNet-ConvGRU0.66720.63030.6350As can be seen from the above experimental results, it is clear that the precision, recall, and F1-score of the ResNet-ConvGRU model proposed in this paper are higher than those of the VGGNet-ConvGRU model for animated GIF emotion recognition by 0.8%, 4.09%, and 3.57%, respectively. The results show that the network structure with residual connections is more capable of feature extraction. This is because ordinary convolutional networks experience gradient disappearance as the number of layers deepens, which results in a loss of information about the learned image features. In contrast, the ResNet network uses jump connections internally, which to some extent avoids the phenomenon of information loss as the network deepens. Therefore, the use of the backbone network ResNet to extract animated GIF image features has obvious advantages in terms of structure.In addition, it was explored that the length of the input image sequence has an impact on the accuracy of the model. Specifically, the models were trained with 2, 4, 6, and 8 images as an input sequence. The experiments showed that the two-image sequence was the least effective, with a model accuracy of 61%, the 4- and 6-image sequences were more effective, with an accuracy of 67% and 66.56%, respectively, while the 8-image sequence had a reduced model training effect, with an accuracy of 65.21%. This may be related to the dataset on the one hand, and the higher correlation of consecutive 4–6 images on the other hand, which is more conducive to the model to extract the sentiment features of short image sequences.The proposed model and the classical video analysis model were compared on the emotion recognition task to further verify the effectiveness of the proposed model.There were 4 classic video analysis models:(1)
VGGNet [23]. It is a deep network structure consisting of 3 × 3 small convolutional kernels, commonly used in VGG16 and VGG19. It can extract image features and is commonly used in areas related to image recognition.(2)
ResNet3D [24, 25]. 3D convolution kernel was used to extract the spatial-temporal characteristics, and the network structure was deeper (18 layers and 34 layers) than the C3D model. This model could be used in video analysis and motion recognition.(3)
CNN-LSTM model [26]. CNN model was used to learn visual features, the output of CNN was used as the input of the LSTM module to learn, data sequence features, and the parameter weights of the two were shared across time. This model could be used in the image sequence and video recognition.(4)
C3D model [27, 28]. The network constructed with 3D convolution and 3D pooling could be used to extract the spatial-temporal characteristics of video data for video recognition and other fields.It could be seen from Table4 that the classification accuracy of the ResNet-ConvGRU model was improved by 2.5 %–4.52% compared to the classic emotion recognition model in the animated GIFs emotion recognition task. The experimental results show that the feature extraction capability of the residual network is better than that of the ordinary convolutional network, and the effects of different residual network models are also different, the accuracy of the ResNet-ConvGRU model is improved by 2.5% compared with the ResNet3D model. In general, the proposed animated GIFs affection recognition model based on ResNet-ConvGRU could improve the performance of feeling classification.Table 4
Comparison of emotion recognition accuracy of different methods.
ModelAccuracyResNet-ConvGRU0.6700VGGNet-ConvGRU0.6349ResNet3D0.6450CNN-LSTM0.6427C3D0.6248


## 4.1. Experimental Data

The GIFGIF data set was used in the experiment in this paper. This data of 17 emotion classifications were provided by MIT multimedia labs, including pleasure, disgust, happiness, excitement, embarrassment, surprise, fear, satisfaction, pride, anger, and guilt, a total of 6170 animated GIFs. The animated GIFs content covered many fields such as movies, TV programs, cartoons, and sports. The data label was pair-compared and voted from users’ emotions on more than 2.7 million animated GIFs content.We selected 7 out of 17 emotion categories including happiness, like (satisfaction), surprise, sadness, anger, disgust, and fear for a better experiment. The first 300 animated GIFs in each category were used, and a total of 2100 GIF data were used in the experiment. In the experiment, 80% of each category of data in the data set was selected as the training set, and the remaining 20% was selected as the test set.


## 4.2. Experimental Parameters and Evaluation Indicators

The experimental parameters are set as shown in Table1.Table 1
Experimental parameters.
ParametersValueParametersValuekernel_size(3,3)Loss functionCross entropyLearning rate0.0001Epoch100weight_decay1e-4Batch size30The function optimizedAdamNumber of categories7The evaluation indexes of the model were accuracy (Acc), precision (P), Recall (R), and F1 Score (F1) to measure the experimental effect of the model. Accuracy represented the proportion of the number determined to be correct to the total number:(6)Acc=TP+TNTP+TN+FP+FN,where TP was the number of true positive classes, TN was the number of true negative classes, FP was the number of misjudging the negative classes as positive classes, and FN was the number of misjudging the positive classes as negative classes. The following formula was the same. Precision was the proportion of the true positive class to the predicted positive class.(7)P=TPTP+FP.The recall rate refers to the proportion of positive class judged to the true positive class.(8)R=TPTP+FN.The F1 score was the harmonic average of accuracy and recall.(9)F1=2⋅P⋅RP+R.As shown above, the higher the accuracy, precision, recall, andF1 score, the better the performance of the emotion recognition model.


## 4.3. Experimental Analysis and Results

Animated GIF is a kind of image sequence data. Its image sequence number is unevenly distributed due to the different sources of the animated GIFs in the data set. As shown by the relevant experiments, the problems of uneven distribution of the number of image sequences and too long sequences all have different degrees of influence on the emotional recognition of animated GIFs. We made the following statistics on the number frequency distribution of all animated GIF image sequences in the experimental data to explore the distribution characteristics of animated GIF image sequences in the data set:Figure5 showed that most of the animated GIF sequences number used in the experiment were within 100, mostly, between 0 and 50. The shortest animated GIF image sequence was 2, and the maximum could be up to 303 in this paper according to the experimental data statistics. It could be seen that the number of animated GIF image sequences in this data set was relatively concentrated despite the large differences in the number of image sequences.Figure 5
Number frequency distribution diagram of the animated GIF image sequence.To alleviate the problem above that the number of animated GIF image sequences had a greater difference which influenced the emotion recognition efficiency, in this paper, after the animated GIF images were converted to static images for saving every 4 images were input into the model as an image sequence according to the time sequence and their spatial-temporal characteristics were analyzed, respectively. Data of image sequences that were less than 4 images were supplemented randomly, while longer image sequences were split, and input in multiple batches. Various animated GIF emotion recognition methods were compared to verify the proposed effectiveness of the animated GIF emotion recognition method based on ResNet-ConvGRU.Table2 showed the comparison of the evaluation indexes of GIF data of each emotion category in the emotion classification model based on ResNet-ConvGRU. It could be seen that the category of anger has a higher score on precision, indicating that the category had higher prediction accuracy in the positive sample results. It might be because the facial expressions and body movements of the animated GIF images of the anger emotion category were more obvious and easier to be distinguished than other emotions. While like had a higher score on recall, indicating that the probability of the actual emotion category being predicted was higher. It might be because the GIF image of like had a higher correlation between features, such as a tendency to contain obvious love and easier to be recognized. There were still gaps in the F1 score among the seven emotional categories, which indicated that there was a certain difference in the difficulty of recognizing various emotional features. Of course, it might be related to the animated GIF images themselves in the model training.Table 2
Animated GIF emotion recognition confusion matrix.
LabelPrecisionRecallF1 scoreAnger0.79760.63310.7059Disgust0.59590.64380.6189Fear0.63220.65770.6447Happiness0.63840.62150.6299Like0.69120.80790.7450Sadness0.71210.64030.6743Surprise0.65920.69400.6762In order to verify the advantages of the proposed model backbone network ResNet feature extraction, the proposed model is experimentally compared with a similar deep network VGGNet in this paper. It is found that the ResNet network used in this paper contains 17 convolutional layers and 1 fully connected layer, while the VGG19 network contains 16 convolutional layers and 3 fully connected layers Table3.Table 3
Comparison of ResNet-ConvGRU and VGGNet-ConvGRU.
ModelPrecisionRecallF1-scoreResNet-ConvGRU0.67520.67120.6707VGGNet-ConvGRU0.66720.63030.6350As can be seen from the above experimental results, it is clear that the precision, recall, and F1-score of the ResNet-ConvGRU model proposed in this paper are higher than those of the VGGNet-ConvGRU model for animated GIF emotion recognition by 0.8%, 4.09%, and 3.57%, respectively. The results show that the network structure with residual connections is more capable of feature extraction. This is because ordinary convolutional networks experience gradient disappearance as the number of layers deepens, which results in a loss of information about the learned image features. In contrast, the ResNet network uses jump connections internally, which to some extent avoids the phenomenon of information loss as the network deepens. Therefore, the use of the backbone network ResNet to extract animated GIF image features has obvious advantages in terms of structure.In addition, it was explored that the length of the input image sequence has an impact on the accuracy of the model. Specifically, the models were trained with 2, 4, 6, and 8 images as an input sequence. The experiments showed that the two-image sequence was the least effective, with a model accuracy of 61%, the 4- and 6-image sequences were more effective, with an accuracy of 67% and 66.56%, respectively, while the 8-image sequence had a reduced model training effect, with an accuracy of 65.21%. This may be related to the dataset on the one hand, and the higher correlation of consecutive 4–6 images on the other hand, which is more conducive to the model to extract the sentiment features of short image sequences.The proposed model and the classical video analysis model were compared on the emotion recognition task to further verify the effectiveness of the proposed model.There were 4 classic video analysis models:(1)
VGGNet [23]. It is a deep network structure consisting of 3 × 3 small convolutional kernels, commonly used in VGG16 and VGG19. It can extract image features and is commonly used in areas related to image recognition.(2)
ResNet3D [24, 25]. 3D convolution kernel was used to extract the spatial-temporal characteristics, and the network structure was deeper (18 layers and 34 layers) than the C3D model. This model could be used in video analysis and motion recognition.(3)
CNN-LSTM model [26]. CNN model was used to learn visual features, the output of CNN was used as the input of the LSTM module to learn, data sequence features, and the parameter weights of the two were shared across time. This model could be used in the image sequence and video recognition.(4)
C3D model [27, 28]. The network constructed with 3D convolution and 3D pooling could be used to extract the spatial-temporal characteristics of video data for video recognition and other fields.It could be seen from Table4 that the classification accuracy of the ResNet-ConvGRU model was improved by 2.5 %–4.52% compared to the classic emotion recognition model in the animated GIFs emotion recognition task. The experimental results show that the feature extraction capability of the residual network is better than that of the ordinary convolutional network, and the effects of different residual network models are also different, the accuracy of the ResNet-ConvGRU model is improved by 2.5% compared with the ResNet3D model. In general, the proposed animated GIFs affection recognition model based on ResNet-ConvGRU could improve the performance of feeling classification.Table 4
Comparison of emotion recognition accuracy of different methods.
ModelAccuracyResNet-ConvGRU0.6700VGGNet-ConvGRU0.6349ResNet3D0.6450CNN-LSTM0.6427C3D0.6248


## 5. Conclusions

An animated GIFs emotion recognition method based on ResNet-ConvGRU is proposed in this paper. The model can improve further the training efficiency by converting GIF short videos into static image sequences and extracting the spatial and temporal features of image sequences using ResNet and ConvGRU networks, respectively. In the experiment, we first showed the frequency distribution of the number of animated GIF images in the data set. Then, the recognition efficiency of various emotion categories of the proposed method was verified. Finally, different emotion recognition models were compared by experiments to show that the proposed model has higher accuracy than the experimental results. The model was validated on only one data set in this paper. The effective performance of emotion recognition on different data sets and the robustness and universality of the model still need further research.



---

*Source: 3143748-2022-09-30.xml*