## 3 Method
### Knowledge Distillation with Teacher Bound  
Even though the inference speed is greatly improved, reducing the MAC from 4 G to 0.3 G, the proposed method minimizes the decrease in accuracy using knowledge distillation and a teacher-bounded loss function. As a result of performance comparisons with existing related studies, SOTA-level accuracy was confirmed despite the light weight of 0.3 G MACs (excepting the teacher model). Figure 4 shows an overview of the proposed method. The teacher model uses Emonet, which has exhibited the best performance in recent emotion recognition, valence, and arousal regression studies. The network structure for knowledge distillation is commonly used in teacher and student model structures. However, when calculating loss, the regression task is learned by applying a teacher bound. We confirmed that this combination of knowledge distillation and teacher bound improves both regression performance and classification performance. The overall performance improvement results realized with the proposed method are summarized in Section 4. \[L_{total}(R_{s},\ R_{t},\ y)=\lambda L_{b}(R_{s},\ R_{t},\ y)+\lambda L_{CE}(P_ {s},\ P_{t})+(1-\lambda)L_{MSE}(R_{s},\ y)+(1-\lambda)L_{CE}(P_{s},\ y) \tag{5}\]  
Figure 3: Scatter plot of AffectNet according to intensity of valence and arousal. Figure 2: AffectNet image histogram according to facial expression class. The teacher-bounded added knowledge distillation loss function can be expressed as Equation (5) and Figure 4, where \(\lambda\) denotes the weight of knowledge distillation loss measured based on the teacher model and student model output. Here, the \(\lambda\) value of the proposed method is set to 0.3. Therefore, the weights of the loss measured based on the teacher model and the loss measured by the correct answer label are 0.3 and 0.7, respectively. \[L_{b}(R_{s},\ R_{t},\ y)\equiv\left\{ \begin{array}{ll}\|R_{s}-y\|_{2}^{2}, \ \ if\ m\|R_{s}-y\|_{2}^{2}>\|R_{t}-y\|_{2}^{2}\\ 0\,\ otherwise\end{array} \right. \tag{6}\]  
Here, \(L_{b}(R_{s},\ R_{t},\ y)\) is a loss function designed to train the regression task of the model. Based on the regression loss of the teacher model, if the output value of the loss is lower than a certain threshold, it is set to \(m\); otherwise, it is set to 0, as expressed in Equation (6) and Figure 4. In the proposed method, \(m\) is set to 1.3. In this study, it is confirmed that the amount of model computation can be reduced as much as possible while minimizing degradation of accuracy using the knowledge distillation loss function combined with  
\[L_{b}(R_{s},\ R_{t},\ y).\ L_{CE}(P_{s},\ P_{t})=-\sum_{x\in X}P_{t}(x)\log P_{ s}(x) \tag{7}\]  
The loss function for the classification task, \(L_{CE}(P_{s},\ P_{t})\), uses cross-entropy, wherein knowledge distillation is calculated based on Equation (7) using the probability of the teacher model \(P_{t}\) and the probability of the student models \(P_{s}\). \[L_{MSE}(R_{s},\ y)=\frac{1}{N}\sum_{x\in X}\left(y-R_{s}(x)\right)^{2} \tag{8}\]  
The loss function for the regression task, \(L_{MSE}(R_{s},\ y)\), is based on the mean squared error (MSE); \(R_{s}\) denotes the output of the student model, and \(y\) is a label of valence and arousal. \[L_{CE}(P_{s},\ y)=-\sum_{x\in X}y(x)\log P_{s}(x) \tag{9}\]  
The loss function for the classification task, \(L_{CE}(P_{s},\ y)\), is based on cross-entropy; \(P_{s}\) denotes the probability output of the student model, and \(y\) is a label of the facial expression class. **Implementation details**. The proposed method uses Emonet as the teacher model and MobileNetV2 and EfficientFormer as student models. To confirm that the proposed  
Figure 4: Proposed knowledge distillation method based on teacher-bounded loss. knowledge distillation method improves performance, experiments were conducted on two CNN models using MobileNetV2 and EfficientFormer. The pre-trained results were used on the ImageNet dataset [22] to initialize the CNN models and fine-tune the entire network on the AffectNet dataset.