# A ResNet-Based Audio-Visual Fusion Model for Piano Skill Evaluation
## 3 Methodology
### Feature Extraction and Fusion
Moreover, the extracted auditory features include information such as the audio spectrum's shape, pitch, and rhythm, which can reflect the content and characteristics of the audio data. Finally, we employ the averaging method as our aggregation scheme (see Aggregation option). **Multimodal branch:** By utilizing the ResNet-3D and ResNet-2D networks, we have obtained visual and aural features. To better capture the semantic association and complementary information between the video and audio modalities, we adopt a joint representation approach for the features extracted from the video and audio data. This helps to create a more comprehensive and accurate feature representation. Let \(\mathbf{X_{\mathrm{v}}}\) and \(\mathbf{X_{\mathrm{a}}}\) represent two sets of deep feature vectors extracted for the visual and aural modalities, where \(\mathbf{X_{\mathrm{v}}}=\{x_{\mathrm{v}}^{1},x_{\mathrm{v}}^{2},\dots,x_{\mathrm{v }}^{M}\}\in\mathbb{R}^{M\times d}\) and \(\mathbf{X_{\mathrm{a}}}=\{x_{\mathrm{a}}^{1},x_{\mathrm{a}}^{2},\dots,x_{\mathrm{ a}}^{N}\}\in\mathbb{R}^{N\times d}\). \(d\) represents the dimension of the visual and aural feature representations, and \(M\) and \(N\) denote the number of extracted visual and aural features, respectively. The multimodal features, \(\mathbf{X_{\mathrm{m}}}\), are obtained by splicing \(\mathbf{X_{\mathrm{v}}}\) with \(\mathbf{X_{\mathrm{a}}}\) in the corresponding dimensions using Algorithm 2:  
\[\mathbf{X_{\mathrm{m}}}=[\mathbf{X_{\mathrm{v}}};\mathbf{X_{\mathrm{a}}}]\in\mathbb{R}^{L \times d} \tag{3}\]  
where \(L=M+N\) denotes the number of fused features, as shown in Figure 3.  
**Algorithm 2** Feature Fusion Algorithm  
```
0:\(X_{v}\): the video features extracted by ResNet-3D; \(X_{a}\): the audio features extracted by ResNet-2D;
0:\(X_{m}\): multimodal features;
1:functionFeatureFusion(\(X_{v}\), \(X_{a}\))
2:\(v\_shape\gets X_{v}\_shape[0]\);
3:\(a\_shape\gets X_{a}\_shape[0]\);
4:if\(v\_shape=a\_shape\)then
5:\(X_{m}\_shape\gets list(X_{v}\_shape)\);
6:\(X_{m}\_shape[1]\gets X_{m}\_shape[1]+X_{a}\_shape[1]\);
7:\(X_{m}\leftarrow\text{torch\_zeros}(X_{m}\_shape)\);
8:\(X_{m}[. ;\,X_{v}\_shape[1]]\gets X_{v}\);
9:\(X_{m}[. ;\,X_{v}\_shape[1]]\gets X_{a}\);
10:endif
11:return\(X_{m}\);
12:endfunction
```  
**Algorithm 2** Feature Fusion Algorithm  
**Aggregation option:** During the piano performance, the score obtained by the players can be perceived as an additive operation. It is often advantageous to perform linear operations on the learned features, which enhances the interpretability and expressiveness of the learned features. Linear operations can also be utilized to reduce the dimensionality of the features, which enhances the efficiency and generalization capabilities of the model. Consequently, we propose the utilization of linear averaging as the preferred aggregation scheme. The application of linear averaging is detailed in Algorithm 3 and depicted in Figure 4.  
Figure 4: Feature average option. Figure 3: The structure of feature fusion. ```
0:\(feats\_list\): The list of features obtained from the network;
0:\(feats\_avg\): Features after the averaging process;
1: Initialize \(feats\_cat\) to Tensor format;
2:for each \(feats\) in \(feats\_list\)do
3:\(feats\_temp\gets feats\);
4:\(feats\_temp\gets feats\_temp\_unsqueeze\_(0)\);
5:\(feats\_temp\gets feats\_temp.transpose\_(0,1)\);
6:\(feats\_cat\leftarrow torch.cat((feats\_cat,feats\_temp),1)\);
7:endfor
8:\(feats\_avg\gets feats\_cat.mean(1)\);
9:return\(feats\_avg\);
```  
**Algorithm 3** Feature Average Algorithm