# Exploring the Effectiveness of Evaluation Practices for Computer-Generated Nonverbal Behaviour
## 2 Related Work
### Evaluation Strategies for Generated Human Behaviour  
The previous two sections described the state of the art in the fields of both gesture generation and listening behaviour generation. But how is the generated behaviour in these works compared, and how is quality assessed? A recent review by [PERSON] et al. [1] discusses subjective evaluation methodologies used in the field of gesture generation for embodied conversational agents. Their work identified that a majority of studies make use of questionnaires to assess the quality of generated motion. Questionnaires, employing Likert scales, are a widely used tool to assess one's attitude towards a concept [39]. One such example is the Godspeed questionnaire [15], which originates from and is used in the field of human-robot interaction and measures the concepts of anthropomorphism, animacy, likeability, perceived intelligence, and perceived safety. [PERSON] et al. [1] called for standardisation given the large variety in reported constructs for the field of gesture generation (which consist of multiple items). In support of this, a 2019 review by [PERSON] et al. [40] that looked into questionnaire usage at intelligent virtual agent conferences (IVAs) found that for 76% of the studies, questionnaires were unique for those studies and were not reused in other studies. Having matching constructs and questionnaire items would make it easier to compare between studies. Efforts for standardisation are underway for the evaluation of virtual agents, but these efforts do not specifically include the evaluation of nonverbal generated behaviour for ECAs [41]. Among the constructs that they reported are human-likeness, appropriateness, naturalness or effectiveness, and understanding. Even though there is a degree of overlap in the constructs between studies, this does not mean that they are measured in the same way or that they contain the same items (statements) and response scales. There have been other, more direct methods proposed for measuring these constructs that do not involve questionnaires or answering multiple statements. For appropriateness, recent studies introduced a new evaluation paradigm that makes use of matched and mismatched gesture motion [13; 19; 20]. In appropriateness for speech, the concept involves combining two motion fragments and one audio segment. Both motion fragments should come from the same condition or system. One of the motion fragments is directly associated with the audio and represents the intended match, while the other fragment is randomly selected and does not have a direct connection to the corresponding audio, resulting in a mismatch. The measured subjective preference for matched motion stimuli over mismatched ones then quantifies how specifically appropriate the motion in question is for the speech. For naturalness or human-likeness, recent studies have relied on directly asking participants to indicate human-likeness using a fine-grained slider [13; 17; 42]. These studies accomplished this using the HEMVIP framework [18], which labels the scale with five anchors (bad, poor, fair, good, and excellent) and associates each position of the slider with a score between 0 and 100 [18]. This framework makes it possible to evaluate multiple different gesture videos for the same speech in parallel. [PERSON] et al. compared the HEMVIP framework with pairwise comparisons and found that under certain circumstances, pairwise comparisons could be a faster way of assessing quality, but it has the downside of scaling badly with multiple conditions [42]. As an example of studies to evaluate aspects of gesture understanding, [PERSON] et al. conducted a study to evaluate the effectiveness and comprehension of generated gestures in an interactive scenario using a virtual avatar [14]. Due to the challenges of online assessments, the researchers employed various methods, including dimensions from the Godspeed questionnaire and direct participant evaluations of human-likeness, along with gaze data.