# Exploring the Effectiveness of Evaluation Practices for Computer-Generated Nonverbal Behaviour
## 1 Introduction  
In this paper, we address the subjective evaluation of generated nonverbal behaviour for embodied conversational agents. We specifically focus on two common methods for assessing speech gesture behaviour and listening behaviour. Additionally, we introduce a newly designed questionnaire for the evaluation of speech gesture behaviour as well as listening behaviour based on recommendations provided in a recent review [1]. Nonverbal behaviour, encompassing elements such as eye gaze, blinking, and co-speech gestures, plays a vital role in facilitating effective human communication [2]. Among the various forms of nonverbal behaviour, co-speech gestures take up a major role [3]. Research has shown that incorporating co-speech gestures can improve human-human communication. For example, [PERSON] et al. [4] discovered that when questions were accompanied by gestures, the time between questions and answers was reduced. Similar effects can be observed when humans interact with embodied conversational agents (ECAs), as the inclusion of nonverbal behaviours such as co-speech gestures and full-body motion has been found to improve the interaction between humans and ECAs [5, 6]. Furthermore, researchers found that including nonverbal behaviours increased the willingness to cooperate with an ECA [7]. Due to the potential of nonverbal behaviour to enhance the human-likeness and communicative efficacy of ECAs, the automatic generation of nonverbal behaviour has become a major research focus. For the field of human motion generation, data-driven methods have gained popularity over recent years [8; 9; 10; 11]. Since the aim of nonverbal behaviour generation for ECAs is to enhance ECAs' interactions with people, proper assessment of these technologies must involve people as well (i.e., use subjective evaluation methods). This is often accomplished through Likert scales [1], in interactive scenarios through listening comprehension [12], through direct questions [13], or using behavioural metrics [14]. In related fields such as human-robot interaction (HRI), aspects such as anthropomorphism, animacy, and likeability are evaluated through the use of questionnaires, of which the Godspeed questionnaire is the most common one [15]. Providing these scales with indirect items instead of asking users to evaluate stimuli directly has the benefit that more nuanced information becomes available, and it becomes easier to compare the outcomes of different studies due to the questionnaire being used across different studies. This paper is an extension of a recently accepted late-breaking report [16] where we focused on generated listening behaviour, while the focus in this paper is more on evaluation methodologies. In comparison with the conference publication, we introduce four new user studies. In this study, we explore three different approaches to evaluating generated co-speech gestures and listening behaviour, aiming to assess their effectiveness and understand their qualities. Previous research commonly relied on questionnaires and Likert scales to evaluate motion and behaviour quality [1]. Following the recommendations and insights from that review, we introduce a new questionnaire that enables us to compare methodological results to previously used methodologies for assessing human-likeness (introduced in [17; 18]) and appropriateness (through mismatching, as introduced in [19; 20]). Our objective is to determine the potential benefits of using a standardised questionnaire in this field and examine how well it correlates with the commonly used direct subjective measurements. A particular goal of this article is to study the effect of different evaluation methods in scenarios where the behaviours evaluated and the differences between them are quite subtle. To this end, we trained two existing motion generation models using dyadic conversational data to generate head, arm, and body motion corresponding to listening, which typically is less vivid than gesture motions performed during active speaking.