# DExter: Learning and Controlling Performance Expression with Diffusion Models
## 5 Evaluation
### Qualitative Study  
We evaluated the naturalness and expressiveness of the rendered performances through a listening test. For samples from eight selected pieces, we compared the following: (1) two human performances with relatively distant interpretations; (2) renderings made with Basis Mixer and VirtuosoNet, as described in Section 5.1; and (3) a rendering from the proposed model, DExter. The performances (including the ground truths) were rendered to audio using a Yamaha Disklavier, which produced similar pedal/articulation-related artifacts in both the human and machine performances. Eighty-two participants listened to the performances and evaluated them on a 100-point Likert scale, rating the overall naturalness and expression of the output as one score. The performances used for the test can be found on the demo page.  
The results of the listening test, shown in Figure 5, provide a nuanced view of the performance-rendering capabilities of the models in comparison to the ground truth. In terms of the mean rating, DExter demonstrated better performances of the pieces by [PERSON], even sometimes comparable to the ground truth (_Barcarolle_). However, they were outperformed via Basis Mixer or VirtuosoNet in the case of older compositions (_[PERSON]'s fugue_ and _[PERSON]'s sonatus_). Overall, in terms of the mean rating scores, there was still a gap between the generative outputs and GT (51.81), while DExter (48.54) slightly outperformed VirtuosoNet (48.31) and Basis Mixer (46.33). It was also surprising to observe that GT did not always secure the highest ratings. In the case of [PERSON]'s etude, at least, it might be explained by the fact that human pianists suffer from physical limitations in technically demanding passages, while the generative models do not.  
\begin{table}
\begin{tabular}{c c c c c} \hline \(\mathbf{w}\) & _dev:_ **tem (\(\downarrow\))** & _dev:_ **vel (\(\rightarrow\)0)** & _cor:_ **tem(\(\uparrow\))** & _cor:_ **vel (\(\uparrow\))** \\ \hline
0.5 & \(1.11\pm 2.46\) & \(-2.47\pm 1.10\) & \(0.11\pm 0.16\) & 0.02 \(\pm\) 0.22 \\
1.2 & \(\mathbf{0.72\pm 2.65}\) & \(\mathbf{1.48\pm 2.12}\) & \(\mathbf{0.19\pm 0.16}\) & **0.28 \(\pm\) 0.21** \\
2 & \(1.33\pm 2.37\) & \(3.02\pm 1.53\) & 0.04 \(\pm\) 0.15 & 0.13 \(\pm\) 0.24 \\
3 & \(1.86\pm 1.81\) & \(4.63\pm 1.15\) & 0.04 \(\pm\) 0.14 & 0.10 \(\pm\) 0.23 \\ \hline \end{tabular}
\end{table}
Table 4: Deviation and correlation of test set relative to ground-truth space (same analysis as in Section 5.1) of tempo and velocity curves. \(\rightarrow\)0 means a value close to 0 is better. \(\downarrow\) indicates a lower value is preferred while \(\uparrow\) is otherwise. The numbers in bold are the best performance on the metric.  
Figure 5: Mean ratings for the eight pieces evaluated in the listening test for each model and human (ground-truth, GT) performances.  
Besides the numerical ratings, we also asked three musically trained participants for explicit feedback on their ratings. In addition to some positive comments, we also received quite specific and useful negative feedback, such as (specifically referring to polyphonic music, such as that of [PERSON]) _no clear voicing among the lines_ and _poor balance between the hands_. Given currently dominating 'flattened' representations, such as our p_codec or tokens in transformer models, learning the vertical structure of music remains a challenge for rendering models [24].