# Lightweight Facial Expression Recognition Based on Class-Rebalancing Fusion Cumulative Learning
## 3 Models and Methods
### The Dual-Branch Network (RLR-CNet)  
With the advancement of study in CNNs, there has been remarkable progress in image classification, which is inseparable from lots of scientific, reasonable, and high-quality datasets, such as MS COCO, ImageNet, and other datasets. These high-quality datasets have a relatively uniform distribution of sample sizes for each category, which is conducive to improving the representation ability of the network for feature extraction. However, most classification datasets (such as facial expression dataset) in real scenes have an uneven  
Figure 6: The structure of the SA module. distribution of sample sizes between classes, and exhibit a long-tail distribution [32] (a few classes have larger sample sizes and most classes have smaller sample sizes), to some extent, so the accuracy of model classification and recognition is affected. When extracting complex and subtle features such as facial expression features, it is especially critical to promote the quality of the dataset and the rationality of the sample distribution. Therefore, in order to resolve the issue of the uneven inter-class distribution of facial expression data and further promote the accuracy of FER in lightweight networks, RLR-CNet is proposed in this paper. This network has two branches based on fusion cumulative learning. Specifically, the two branches proposed in this paper are called the \"feature learning branch\" and \"class-rebalancing branch\". Figure 7 shows a schematic diagram of the dual-branch network. The Clip_K5_ShuffleNet inverted residual structure is used into two branches [33], the lightweight ShuffleNet as the backbone network is used in the dual-branch network, and all blocks in Stage2, Stage3, and Stage4 are replaced with the improved Clip_K5_ShuffleNet modules in this paper. All weights of the network before Stage4 are shared between the two branches. In each convolutional layer, the input images are convolved with the same filter to achieve weight sharing and cut down the number of parameters. Good feature learning is beneficial for rebalancing the learning of class-rebalancing branches and can greatly reduce the network's computational load. For the two branches, the conventional uniform sampler and the reverse sampler are used, respectively, to obtain two data samples \((x_{1},y_{1})\) and \((x_{2},y_{2})\), which are used as the input of the feature learning branch and class-rebalancing branch, respectively, and the feature vectors \(f_{1}\) and \(f_{2}\) are obtained after the corresponding branch convolution and global average pooling. Considering that class-rebalancing can significantly boost classifier learning, it also impairs feature learning to some extent [34]. So, this paper introduces a specific method of cumulative learning, which staggers the learning \"attention\" of the two branches during network training. In other words, focusing on the learning of the feature learning branch in the early stage and the class-rebalancing branch in the later stage, to eliminate the effect of class-rebalancing on feature learning, achieves the goal of promoting the recognition accuracy of the network. Specifically, an adaptive trade-off parameter \(\alpha\) is used to control the weights of \(f_{1}\) and \(f_{2}\), and the weighted feature vectors \(\alpha*f_{1}\) and \((1-\alpha)*f_{2}\) are input into the classifiers \(W_{1}\) and \(W_{2}\), which are finally integrated together by channel merging. The output formula is as follows:  
\[z=\alpha W_{1}^{T}f_{1}+(1-\alpha)W_{2}^{T}f_{2} \tag{6}\]  
where \(z\) is the final output and the predicted probability for each class \(i\in\{1,2,\cdots,C\}\) can be calculated by the following formula:  
\[\hat{p}_{i}=\frac{e^{z_{i}}}{\sum_{j=1}^{C}e^{z_{j}}} \tag{7}\]  
Notated as \(\hat{p}=|\hat{p}_{1},\hat{p}_{2},\cdots,\hat{p}_{C}|^{T}\), the final category with the maximum probability is used as the last recognition result of the dual-branch network. The dual-branch network designed in our paper mainly has the following three characteristics:  
1.