# Evaluation of Emotional Satisfaction Using Questionnaires in Voice-Based Human-AI Interaction
## 2 Materials and Methods
### Voice-Based Intelligent System Design  
The currently launched VIS (Bixby (Samsung Electronics), Siri (Apple Inc.), etc.) are complete systems, but the design parameters cannot directly be controlled; therefore, we have conducted research to develop an evaluation system using the Wizard of Oz (WoZ) experiment [22]. In the field of HCI, a WoZ experiment is a research experiment in which subjects interact with a computer system that they believe to be autonomous, but the system is actually being operated or partially operated by an unseen human being [23; 24]. [PERSON] et al. [25] developed and evaluated a WoZ-type voice interface to implement an interaction situation with a voice-based interface during driving conditions. In addition, [PERSON] and [PERSON] [26] conducted a study to mitigate the problems arising from interaction with users using a WoZ-type VIS. In this study, a WoZ-based adjustable VIS was constructed to evaluate the changes in user satisfaction according to the design parameters. The final four design parameters were selected based on the user requirements revealed in previous studies [27; 28; 29] and are listed in Table 1.  
The level of the design parameters was selected based on existing research and Bixby. The \"response time to get to the next interaction\" is defined as the time taken by the VIS to respond to the user's request after the user requests information. The range of this parameter was selected based on Bixby's average response time of 2 s in order to check if there was a difference in user satisfaction when a certain amount of time was spent on the information to be presented. The experiment was designed such that the information was presented randomly between 1 and 5 s, so that the subject was unaware of the presented time. For the analysis, this parameter was classified as 1, 2, 3, and 4 s.  
The \"number of trials to get the right answer for a question\" is defined as the number of times a user tries to obtain the desired answer. This parameter was set based on a study in which the system is reset if an error occurs up to two times; it was designed to obtain the correct answer after two attempts [30]. The \"pace of the answer\" is defined as the number of syllables per second of the answer; here, 4, 6, and 8 syllables/s were selected and set in three steps. Four syllables and eight syllables were added to determine the highest level of satisfaction, focusing on the six syllables per second applied to the existing Bixby. Finally, the \"sentence structure of the answer\" is defined as the amount of information present in the response based on the user's needs. The \"sentence structure of the answer\" \begin{table}
\begin{tabular}{c c} \hline \hline
**Design Parameters** & **Scope** \\ \hline Response time to get to the next interaction & Between 1 and 5 s \\ \hline Number of trials to get the right answer for & 1 trial \\ a question & 2 trials \\  & 3 trials \\ \hline  & 4 syllables/s \\ Pace of the answer & 6 syllables/s \\  & 8 syllables/s \\ \hline  & Answer only \\ Sentence structure of the answer & Repeat the question and then answer with a \\  & clear reference source \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Scope of voice-based intelligent system design parameters. was divided into three levels--\"answer only\", \"repeat the question and then answer\", and \"repeat the question and then answer with a clear reference source\"--considering the format of the answer of [PERSON]. When the structure was designed by combining these four parameters, 108 interactions per subject were determined. However, considering the time, cost, and fatigue of the subject, \"response time to get to the next interaction\" and \"pace of the answer\" were designed to appear randomly in all the experiments, and nine interaction scenarios were determined by combining the remaining two parameters (\"number of trials to get the right answer for a question\" and \"sentence structure of the answer\").