# Exploring the Effectiveness of Evaluation Practices for Computer-Generated Nonverbal Behaviour
## 6 Discussion  
For this paper, we trained two models on a data set containing dyadic conversations. We used the synthesised stimuli in six subjective studies. Two of the three evaluation paradigms are existing evaluation strategies from the field of co-speech gesture generation. We compared these two to a newly made questionnaire based on the usage of Likert scales and constructs in a previous work on gesture generation [1]. In study one, the human-likeness for gesturing was evaluated. No significant difference was found between StyleGestures (SG) and the baseline. However, the ground truth scored significantly higher on average, and both SG and the baseline differed significantly from the ground truth. The finding that SG and the baseline did not differ significantly is interesting since the baseline model incorporates semantic information. Human-likeness evaluations focus on motion quality rather than appropriateness of gestures with speech audio. Previous works on gesture generation evaluation yielded more robust differences using this methodology [13]. Interestingly, the ground truth condition scored 70 on average, even though the stimuli in this condition were based on human recorded motion. This has been found previously in other works as well [13,17] and is most likely because of the different embodiment of the avatar in relation to that of humans, as well as the lack of other human-like features (such as a face). Another angle of evaluating the appropriateness of gesticulation can be conducted through the use of matching and mismatching videos [20]. In study 2, we took the same 30 segments and combined them into one video with the interlocutor. In one of the two videos presented to the participant, the gesture sequence of the avatar on the left was not related to that part of the conversation. We found significant differences for both the baseline and SG with the ground truth condition but not for the baseline versus SG. When we look at Figure 6, we see that more videos were identified as 'equal' for the SG condition. As expected, the ground truth videos were identified as matching more than 70% of the time. Since the baseline model also had access to the text beside the speech audio, one would expect this model to generate more appropriate (and even semantic-related) gestures, but the results from the appropriateness study do not seem to confirm this. We wanted to know whether these two paradigms of human-likeness and appropriateness testing could be used with more subtle forms of human nonverbal behaviour, such as listening behaviour. In the third study, we evaluated the human-likeness of the generated listening behaviour and compared it to the ground truth. It is important to mention that for the human-likeness evaluation, we excluded the audio to only assess the quality of the motion. We found significant differences between all conditions, with SG scoring past the ground truth. However, the overall rating for each condition was not particularly high. We think that human-likeness testing for motion for listening behaviour is difficult since appropriate listening behaviour is really dependent on the conversation. Omitting the audio could also have led to the participants not being able to see that this motion was supposed to be part of a conversation. Another reason could be that in terms of motion quality, it was all similar and therefore scored the same because of the lack of context. Appropriateness testing of listening behaviour could help figure out whether it actually matters what listening behaviour is tied to a conversation and whether participants can spot differences in generated listening behaviour. Here, we cannot report any significantdifference between the three conditions. It appears that the participants had difficulty identifying the right listening behaviour. One reason for that could be that listening behaviour takes place more with facial expressions than with body language, and that the body pose alone is not enough to say that someone is attending a conversation.