## 3 Methods
### Third-Party Emotion Evaluation  
To establish ground-truth for the physiological and speech features derived from user responses to the voice user interface, third-party evaluations were conducted by six evaluators. To perform the evaluation, evaluators watched 188 clips of participant webcam videos corresponding to each interaction in order to simultaneously consider both physical and oral expressions of emotion. Each clip was coded to commence from the moment the voice user interface's question was posed and ended 500 ms following the participant's response. Each participant had a range of 7 to 17 interaction clips to be evaluated, presented in a randomized order. Evaluations were recorded using the online survey tool Qualtrics. The survey used to record the evaluations was built into the platform and embedded on the page using custom HTML code. Each survey recorded the evaluations of the same participant, resulting in 16 unique Qualtrics links.  
To ensure standardized evaluations, all evaluators were trained. Within this training, evaluators were guided within their manual assessment of the four studied dimensions of affective state: valence, arousal, control, and short-term emotional episodes (STEE). As its name suggests, the STEE evaluation point was indented to capture momentary fleeting glimpses into the participant's emotional state. The temporal nature of these events did not make them any less important. On the contrary, these split moments depicted authentic emotion, especially amongst subjects who tended to suppress public displays of emotion.  
Evaluators were instructed to watch each interaction clip twice and assess the emotional reaction using both visual and voice behavior of the participant, while taking into consideration the semantic context of the voice user interface speech. A series of instructions and guidelines addressing the emotional dimensions to be assessed were provided and explained to the evaluators. For each dimension, the spectrum of extremes was defined. In addition to these definitions, a series of vocal and visual cues were provided as examples of elements to look out for.  
Evaluators were provided instructions with regards to the Self-Assessment Manikin (SAM) scale proposed by [PERSON] and [PERSON] in 1994 [83]. Valence, arousal, and control are classic dimensions of affective state, measured ubiquitously in IS research by users through  
Figure 4: Graphical summary of the experiment procedures.  
self-assessment questionnaires. The SAM scale measures three emotional dimensions, that of pleasure, arousal, and control or dominance, using a series of graphic abstract characters displayed horizontally using a nine-point scale, although five and seven-point variants may also exist [84]. For this experiment, we opted for the nine-point scale in order to offer further precision and remain consistent with previous observer-based studies utilizing this measure [85,86].  
In contrast to the valence, arousal, and control dimensions, the STEEs were observed using a binary evaluation. To assess STEEs, evaluators were asked to select the best suited option (non-present, positive STEE, or negative STEE) applicable to the interaction. Solely its presence, rather than its frequency and intensity, was observed within this evaluation point. In addition to the SAM-based and binary-based scale ratings, evaluators were asked to note the vocal and visual cues supporting their evaluations.  
In order to assess the evaluators' grasp of the dimensions, all six analyzed the same participant. Following this primary evaluation, the results were analyzed and further guidance was provided in order to ensure uniformity. The process was repeated, resulting in greater consistency. Once this consistency was achieved, evaluators were instructed to pursue the remaining evaluations. The remaining Qualtrics links, featured in random and individualized orders, limited the risk of bias, as evaluator fatigue upon the same final evaluation was avoided.