# A ResNet-Based Audio-Visual Fusion Model for Piano Skill Evaluation
## 4 Experiments
### Results of Experiments  
Table 4 and Figure 6 present the results of the unimodal and audio-visual fusion models on the PISA dataset. Both models achieved good results; however, the audio-visual fusion model achieved better results than the unimodal model. This indicates that the multimodal method can effectively compensate for the inability of the audio model alone to utilize visual information, leading to more accurate and comprehensive evaluation results. In addition, our audio-visual fusion model obtained the best experimental results, achieving an accuracy rate of 70.80%. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Model** & **V:A** & **Accuracy (\%)** \\ \hline C3D + ResNet18-2D [16] & 8:1 & 68.70 \\ \hline ResNet18-3D + ResNet34-2D & 1:1 & 66.39 \\ \hline ResNet34-3D + ResNet18-2D & 1:1 & 66.81 \\ \hline ResNet34-3D + ResNet34-2D & 1:1 & 65.97 \\ \hline ResNet50-3D + ResNet50-2D & 1:1 & 59.45 \\ \hline ResNet18-3D + ResNet18-2D (Ours) & 1:1 & 70.80 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Performance (% accuracy) of multimodal evaluation. V:A: the ratio of the number of video features to audio features. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Stage** & **18-Layer** & **34-Layer** & **50-Layer** \\ \hline Block1 & \
\begin{tabular}{c} _Conv_ : 64, \(7\times 7\), \(2\times 2\) \\ _Max pool_ : \(3\times 3\), \(2\times 2\) \\ \end{tabular} \\ \hline Block2\_x & \(\left[\begin{array}{c}Conv:64,3\times 3\\ Conv:64,3\times 3\end{array}\right]\times 2\) & \(\left[\begin{array}{c}Conv:64,3\times 3\\ Conv:64,3\times 3\end{array}\right]\times 3\) & \(\left[\begin{array}{c}Conv:64,1\times 1\\ Conv:64,3\times 3\end{array}\right]\times 3\) \\ \hline Block3\_x & \(\left[\begin{array}{c}Conv:128,3\times 3\\ Conv:128,3\times 3\end{array}\right]\times 2\) & \(\left[\begin{array}{c}Conv:128,3\times 3\\ Conv:128,3\times 3\end{array}\right]\times 4\) & \(\left[\begin{array}{c}Conv:512,1\times 1\\ Conv:128,3\times 3\end{array}\right]\times 4\) \\ \hline Block4\_x & \(\left[\begin{array}{c}Conv:256,3\times 3\\ Conv:256,3\times 3\end{array}\right]\times 2\) & \(\left[\begin{array}{c}Conv:256,3\times 3\\ Conv:256,3\times 3\end{array}\right]\times 6\) & \(\left[\begin{array}{c}Conv:256,1\times 1\\ Conv:256,3\times 3\end{array}\right]\times 6\) \\ \hline Block5\_x & \(\left[\begin{array}{c}Conv:512,3\times 3\\ Conv:512,3\times 3\end{array}\right]\times 2\) & \(\left[\begin{array}{c}Conv:512,3\times 3\\ Conv:512,3\times 3\end{array}\right]\times 3\) & \(\left[\begin{array}{c}Conv:512,1\times 1\\ Conv:512,3\times 3\\ Conv:2048,1\times 1\end{array}\right]\times 3\) \\ \hline Block6 & \multicolumn{2}{c}{
\begin{tabular}{c} _Average pool_ : \(7\times 7\), \(1\times 1\) \\ \hline Block7 & _Linear_ : in = 512, out = 128 \\ \hline Block8 & _Linear_ : in = 128, out = 10 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: ResNet-2D for aural branch. All conv. layers are followed by Batch Normalization (BN) and Rectified Linear Units (ReLu). “[ ] \(\times\) m\({}^{\prime}\) means repeat this block m times. As shown in Table 5, our audio-visual fusion model outperformed the other models in terms of computational efficiency. This was due to the residual structure that accelerated the convergence speed of the network, as well as the small convolutional kernel that reduced the complexity of the convolutional operation, thus enhancing the computational efficiency of the model. The results in Table 6 show that the accuracy improvement of the audio-visual fusion model was smaller when the ratio of the number of video features to audio features was relatively large. However, when the number of features in both modalities was closer, the accuracy improvement was relatively larger. The reason may be that the number of features of one modality had a significantly larger number of features than the other, which caused the model to place more emphasis on the modality with more features and overlook the modality with fewer features.