# Multi-Modal Emotion Recognition for Online Education Using Emoji Prompts
## 3 The Proposed Method
### Multi-Modal Emotion Analysis Method  
The multi-modal emotion analysis method was divided into two parts: text pre-training and emoji data recognition. Text pre-training: First, we performed basic operations such as word segmentation and the removal of stop words from the collected text data. The Jieba ([[https://github.com/fxsjy/jieba](https://github.com/fxsjy/jieba)]([https://github.com/fxsjy/jieba](https://github.com/fxsjy/jieba)), accessed on 1 January 2023) Chinese word segmentation package was used for word segmentation. The Chinese word segmentation package can accurately segment words with semantic relevance into word groups, with an accuracy rate of more than 99.8%. Second, part of the speech tagging was performed on the text data after word segmentation. The purpose of this step is to identify adjectives and adverbs with rich emotional polarity. Of course, entity and functional verbs were also tagged. These words also play a role in semantic recognition. For training, we used the current efficient BERT model based on an attention mechanism. Specifically, we used random MASK to cover up adjectives or adverbs that represent emotional polarity. In the prediction stage, we predicted the words that were covered to judge the accuracy of model prediction. If the predicted emotional word polarity did not change, the prediction was considered accurate. The final accuracy was calculated from all of the prediction results. The calculation formula (Formula (1)) is as follows:  
\[p=\frac{1}{2}(\frac{n_{pos}}{n_{pos\_all}}+\frac{n_{neg}}{n_{neg\_all}}), \tag{1}\]  
where the accuracy rate \(p\) comprises two parts: the prediction accuracy rate of positive emotion words and the prediction accuracy rate of negative emotion words. The final accuracy rate \(p\) is the average of the two. \(n_{pos}\) represents the number of correctly predicted positive emotion texts, \(n_{pos\_all}\) represents the number of all positive emotion texts in the test set, \(n_{neg}\) represents the number of correctly predicted negative emotion texts, and \(n_{neg\_all}\) represents the number of all negative emotion texts in the test set. Because the predicted emotional words are not always the ideal masked words, in most cases, the predicted emotional words will be the most likely emotional words, selected by means of the calculated probability, which means that the predicted emotional words and the masked emotional words cannot match completely. Therefore, in the actual experiment, we measured the similarity between the predicted emotional words and the masked emotional words, and the number of accurate predictions was calculated by setting the threshold of similarity. The calculation method for the threshold \(\eta\) is shown in Formula (2):  
\[\eta=\cos(\omega_{predict},\omega_{mask}), \tag{2}\]  
where \(\omega_{predict}\) and \(\omega_{mask}\) represent the predictive and masked sentiment words, respectively. \(\cos\) (0) calculates the cosine similarity between the two values. The threshold value \(\eta\) was initially set to 0.5. When \(\eta\) > 0.5, the current prediction is correct. When \(\eta\) < 0.5, this indicates that the current prediction is incorrect. Finally, we calculated the number of correctly predicted texts. Figure 3: Two groups of emoji data with strong emotional polarities. Emoji prompt: The emoji prompt improves the accuracy of multi-modal recognition through the prompt method. In most cases, an emoji prompt is used as an aid for emotion classification. The first case includes text and emoji modal data. If the prediction polarity of the text and emoji is the same, the final emotional result is the polarity of the text and emoji. If the predicted polarities of the text and emoji are different, the final emotional result depends on the predicted polarity of the text. In the first case, the polarity of the emoji has no effect. In the second case, the data only include text data but lack emoji data.