# Exploring the Effectiveness of Evaluation Practices for Computer-Generated Nonverbal Behaviour
## 4 Materials and Methods
### User Studies
#### 4.4.2 Study 1: Human-Likeness for Gesticulation  
The purpose of this study was to investigate how human-like a gesture motion was when generated by SG in comparison with the baseline and ground truth motion. We recruited 22 participants. From the test set, 30 segments were randomly selected in which the speaker was talking, and for each stimulus, gesture motion was synthesised from SG and the baseline or taken from the ground truth. Each segment had a length of between 6 and 12 s. The participants were asked the following question: 'How human-like does the gesture motion appear?' We did not include audio when rendering the videos, as the inclusion of audio would affect the ability of the participants to rate the stimuli for human-likeness. Three videos were placed on one evaluation screen using the HEMVIP framework for evaluating the stimuli, which has been validated and used by others before [13; 17; 18; 42] based on webMUSHRA [48]. The participants could press play to start one of the videos, and at all times, only one video was visible (see also Figure 2 for the interface). The order of the videos on the screen was randomised, as well as the order in which the screens were  
\begin{table}
\begin{tabular}{p{113.8 pt} p{113.8 pt} p{113.8 pt}} \hline \hline
**Study** & **Study Goal** & **Question Posed** \\ \hline \multirow{2}{*}{1} & Human-likeness of the generated gesture motion & How human-like does the gesture motion appear? \\ \hline \multirow{2}{*}{2} & Appropriateness of the generated gesture motion for the conversation & Indicate which character’s motion best matches speech in terms of rhythm, intonation, and meaning. \\ \hline \multirow{2}{*}{3} & Human-likeness of the generated listening motion & How human-like does the listening motion appear? \\ \hline \multirow{2}{*}{4} & Appropriateness of the generated listening motion for the conversation & Indicate which character’s motion shows the most appropriate listening behaviour, considering the speaker’s motion. \\ \hline \multirow{2}{*}{5} & Agreement on three constructs for the generated gesture motion: appropriateness, human-likeness, and intelligibility & See Figure 4. \\ \hline \multirow{2}{*}{6} & Agreement on three constructs for the generated listening motion: appropriateness, human-likeness, and intelligibility & See Figure 4. \\ \hline \hline \end{tabular}
\end{table}
Table 1: Overview of the studies, the study goal per study, and the question that the participants were asked.  
presented to the participant. Each participant was presented with two attention check videos inserted randomly during the experiment. Both attention checks would ask the participant to rate the video with a specific score. The text for the attention check would only appear halfway through the video (so that it would take out participants that did not watch the complete video or that were not paying attention to the video at all). The participants were asked to rate the human-likeness on a scale from 0 to 100, where a score of 100 would mean the gesture motion was completely human-like and excellent. The rating scale was accompanied by five anchors that were equally spaced, namely bad, poor, fair, good, and excellent. Each participant rated 14 screens with 3 stimuli per screen, totalling 42 ratings per participant and 308 ratings per condition.