# Audio-Visual Tensor Fusion Network for Piano Player Posture Classification
## 4 Performance Evaluation
### Implementation Details  
Table 2 shows a summary of the experiments conducted in this paper. To the best of our knowledge, there are no methods that use video and audio information to classify postures in the music domain. Thus, we have adopted several methods from different domains. Specifically, we compared the proposed AV-TFN with VN [7] and AN that utilize only visual and audio networks, respectively. Besides, we compared the proposed method with AVN_Concat [8] and AVN_Atten [9] that use audio-visual information with various data representation techniques. Figure 7 demonstrates the structure of competing models. Each model was formed with the structure outputting the best performance (i.e., number of layers, number of filters, and optimizer). For VN and AN, the skeleton extracted from the video and MFCC extracted from the audio were used as input values. Through hyperparameter setting experiments (refer to Section 4.2), LSTM was selected to process time-series data for VN. Here, four LSTM layers and one hundred LSTM cells were used, where the epoch was set to 1000. As for AN, four CNN layers were used, where the number of filters was set to 32, the batch size was set to 200, the kernel size was set to (3,3), and the epoch was set to 100. Furthermore, after flattening the data to one dimension, ten dense unit layers are formed. In the last layer, the Softmax operation is performed to classify the posture. For both VN and AN, we use Adagrad optimizers with a learning rate of 0.01 and decay of 0.0. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Ex** & \multicolumn{3}{c}{**Parameters**} \\ \hline \multirow{3}{*}{Ex. 1} & & Image resolution & \(80\times 120\) \\  & & Kernel size & (6,6) \\  & with other methods & Models & VN, AN, AVN-Concate, \\  & & & AVN-Atten, AV-TFN \\  & & Tensor fusion ratio & 50:50 \\ \hline \multirow{3}{*}{Ex. 2} & & Image resolution & \(20\times 30\), \(40\times 60\), \(80\times 120\), \(120\times 180\) \\  & & Epoch & 1500 \\ \cline{1-1}  & & Kernel size & (1,1), (2,2), (6,6) \\ \cline{1-1}  & & Models & AN, AVN-Concate, AV-TFN \\ \cline{1-1}  & & Tensor fusion ratio & 50:50 \\ \hline \multirow{3}{*}{Ex. 3} & & Image resolution & \(20\times 30\) \\ \cline{1-1}  & & Epoch & 1500 \\ \cline{1-1}  & & Kernel size & (1,1) \\ \cline{1-1}  & & Models & AV-TFN \\ \cline{1-1}  & & Tensor fusion ratio & 0:100, 50-50, 100:0 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Summary of experiments and hyperparameters. We constructed AVN-Concat as follows. The input values for AVN-Concat are the skeleton data and MFCC. For this, LSTM and CNN were used, which are suitable for time series data processing and audio classification, respectively. For the visual information, four CNN layers were used for processing. Here, the number of filters was set to 32, the batch size was set to 200, the kernel size was set to (3,3), and the epoch was set to 100. Furthermore, after flattening the data to one dimension, the Softmax function is executed. For the audio information, four LSTM layers comprising 100 cells are used for processing. Furthermore, after flattening the data to one dimension, the Softmax function is executed. Each stream of CNN and LSTM is concatenated, and ten dense unit layers are formed. In the last layer, Softmax operation is performed one more time to classify the postures. We use SGD optimizers for AVN-Concat, where a learning rate was set to 0.001, decay was set to 1e-6, and momentum was set to 0.9. As for AVN-Atten, the input data are concatenated into one dimension through a bi-direction Gated Recurrent Unit (GRU) layer with 200 attention units. Furthermore, AVN-Atten performs classification of postures using a dense layer and Adam optimizers, where a learning rate was set to 0.001, the beta_1 was wet to 0.999, and the decay was set to 0.0. Regarding the training and test ratio, we used 80% of the total data as a training dataset and 20% as test data, where data were randomly chosen. In experiments 1-3 (Table 2), the same training dataset and test data were used.