## 4 Study 2. Knowledge-Driven and Data-Driven Competence
### Annotation user interface
#### 4.1.6 Extracted Features  
The extracted features in each modality are as follows (for all experiments in Section 4.2, we extracted the same features):  
1. Visual features are traits that can be captured visually. To extract visual features, we cut frames from the video clips at 30 Hz. First, with the multitask cascaded convolutional neural network (MTCNN) face detection algorithm [(97)], we obtained aligned faces. In addition, we applied the facial action coding system (FACS) to influencers' faces to detect facial action units and better understand their facial expressions. Finally, we obtained 35 facial action units and 8-dimensional gazes using the OpenFace2.0 toolkit. Forty-three-dimensional visual features were extracted, as shown in Figure 7.  
Figure 6: Competence annotation interface. 2. Vocal  
Vocal features are the acoustic features of an influencer's voice. Vocal features were obtained from the MP3 audio files that were separated from the video clips by calling the moviepy module in Python. Then, we used the Librosa speech toolkit to extract voice features from these audio files at 22,050 Hz. Finally, 3-dimensional frame-level voice features, including 20-dimensional mel-frequency cepstral coefficients (MFCCSs), RMS, zero crossing rate, spectral rolloff, and spectral centroid, were extracted. Twenty-four-dimensional vocal features were extracted, as shown in Figure 8.  
Figure 8: Mel spectrogram of the vocal modality. Figure 7: Extracted features from the visual modality. Textual  
Textual features are the characteristics of individuals' language content when they introduce products. All text was obtained by transferring individuals' voices into text with the ASR technique of the Alibaba Cloud. Since all the videos were obtained from a livestream platform, the transcripts were in Chinese. We used two tokens to label the beginning and end of each text. Then, we generated word vectors by using pretrained Chinese BERT-based word embeddings. Finally, each word was represented as a 768-dimensional word vector, as shown in Figure 9 and Table 5.  
To address this challenge, we applied a cutting-edge deep learning algorithm, LMF, for individual competence evaluation at the feature level [98]. This algorithm outperforms other deep learning algorithms in three ways: (a) LMF is characterized by using tensor representation, enriching the intermodal information in the model and leading to a more convincing prediction. (b) LMF is an efficient light computation method characterized by using low-rank tensors instead of explicitly creating a high-dimensional tensor. Therefore, LMF requires relatively low computing power, making it suitable for application and generalization in various scenarios. (c) LMF has also been proven to achieve high performance in prior studies, with relatively high accuracy and low MAE [98]. Given the above advantages of LMF, we chose it for competence prediction. We divided the 2374 video samples into an 80% training set, a 10% validation set, and a 10% test set for the LMF model. The operating principle of this method is the transformation of the input representation into a high-dimensional tensor \(Z\) and then mapping it back to a lower-dimensional output vector space. Here, \(\{Z_{m}\}_{m=1}^{M}\) denotes the unimodal vectors of M different modalities, and \(h\) denotes the multimodal output representation. However, in the following steps, LMF does not explicitly create a tensor \(Z\). Instead, three steps are taken to simplify the calculation:  
First, the weight factor \(W\) is decomposed. Generally, to map tensor \(Z\) back to a lower-dimensional output vector space, it needs to be multiplied by a weight factor \(W\), which is naturally a tensor of order \(-(M+1)\) and denotes the weight of a linear layer. It can be partitioned into \(\tilde{W}_{k}\) (\(\tilde{W}_{k}\in R^{d_{1}\times d_{2}\times...\times d_{M}}\), \(k=1,\ 2,\ \ldots,d_{h}\)). Moreover, every \(\tilde{W}_{k}\) can be decomposed by Equation (8).