# A ResNet-Based Audio-Visual Fusion Model for Piano Skill Evaluation
## 3 Methodology
### Feature Extraction and Fusion  
**Visual branch:** There are some skills of performers that can only be observed and evaluated visually rather than by hearing such as the fingering skills of the performer. For example, a professional pianist playing a piece at high speed may use their index and ring fingers to play octaves, which is difficult for an average pianist. For a pianist, the lack of such skills does not necessarily indicate their proficiency level. However, if a pianist possesses these skills, it can prove that they have reached a high level of technical ability. The movements of fingers captured in videos involve both the appearance and temporal dynamics of the video sequences. The efficient modeling of the spatio-temporal dynamics of these sequences is crucial for extracting robust features, which, in turn, improves the performance of the model. Three-dimensional CNNs are efficient in capturing the spatio-temporal dynamics in videos. Specifically, we consider ResNet-3D [4] to extract the spatio-temporal features of the performance clips from a video sequence. Compared to conventional 3D CNNs, ResNet-3D can effectively capture the spatio-temporal dynamics of the video modality with higher computational efficiency. For ResNet-3D, we stack multiple 3D convolutional layers to model motion features in the temporal dimension and utilize 3D pooling layers and fully connected layers for feature descent and combination. In this way, we can extract rich visual features from the video data, including information such as the object's shape and color, to capture finger motion patterns. In addition, we can utilize a pre-trained model to improve the model's performance, as shown in Algorithm 1. Finally, we employ the averaging method as our aggregation scheme (see Aggregation option). Figure 2: Samples from the dataset. Images after cropping (only part of the complete sample is shown). ```
0:\(model\_dict\):the dictionary of model parameters;pretrained_dict:the dictionary of pre-trained model parameters;
0:\(model\_dict\):the model dictionary after completing the update;
1:function MODELINIT(model_dict,pretrained_dict)
2:for each pair(\(k,v\))\(\in\)pretrained_dict do
3:if\(k\)\(\in\) model_dict.keys then
4:\(model\_dict[k].value\)\(\leftarrow\)pretrained_dict[\(k].value\);
5:endif
6:endfor
7:return\(model\_dict\);
8:endfunction
```  
**Algorithm 1** Model Initialization Algorithm  
**Aural branch:** We can also obtain a lot of information about the piano from the audio. The rhythm, notes, and pitches of the piano can be perceived through listening, and this is a common, simple, and practical way to evaluate a piano piece. In fact, different scores vary greatly in terms of style, rhythm, etc. This requires the judges to have a high level of proficiency in the piece being played by the performer in a piano competition or performance, and judges who are not familiar with the piece may encounter difficulties in evaluating the skills of the performer. Information such as the pitch and rhythm of a piano performance is contained in the audio data, and both raw audio waveforms [24; 25] and spectrograms [26; 27] can be used to extract the auditory features. However, the spectrogram can provide more detailed and accurate audio features. Specifically, we convert the raw audio data into the corresponding Mel-spectrogram, which can be regarded as image data, due to its two-dimensional matrix form. We then feed the obtained Mel-spectrogram to the auditory network. Further, ResNet-2D [5] outperforms the traditional 2D CNN in terms of computational efficiency and feature extraction. Additionally, it can utilize a pre-trained model to improve performance. Therefore, we prefer to use ResNet-2D for feature extraction from the Mel-spectrogram. By stacking 2D convolutional layers, we can capture the patterns and variations in the audio data in the frequency and time dimensions.