# Evaluation of Emotional Satisfaction Using Questionnaires in Voice-Based Human-AI Interaction
## 4 Discussion  
### Validity of the Kansei Engineering Approach  
Identifying the user's emotional satisfaction that occurs in the process of interacting with an intelligent system is a very important point in designing high-quality HAII. In particular, it is necessary to understand what kind of emotions are generated and develop in a direction that can increase the satisfaction level of emotions. In this study, we confirmed the emotions generated by users in voice-based HAII using the Kansei engineering approach. By quantitatively evaluating the emotions through 30 KWs, two representative emotions were derived--'pleasurability\" and \"reliability\"--and emotion classification was conducted. Human emotions are mostly classified based on [PERSON]'s six basic human \begin{table}
\begin{tabular}{c c c c c c c c c c c} \hline \hline  & \multicolumn{3}{c}{**Pleasurability**} & \multicolumn{6}{c}{**Reliability**} \\ \hline
**Number of Trials** & **Sentence Structure** & **Avg** & **Cluster** & **Number of Trials** & **Sentence Structure** & **Avg** & **Cluster** & & \\ \hline
1 Trial & Q + A + Ref & 0.88 & A & & 1 Trial & Repeat Q + A & 0.611 & A & & \\
1 Trial & Repeat Q + A & 0.501 & A & B & 1 Trial & Q + A + Ref & 0.561 & A & B & & \\
2 Trials & Q + A + Ref & 0.077 & A & B & C & 2 Trials & Q + A + Ref & 0.326 & A & B & C & \\
1 Trial & Answer only & 0.052 & A & B & C & 2 Trials & Repeat Q + A & 0.235 & A & B & C & D \\
2 Trials & Repeat Q + A & \(-\)0.035 & B & C & 1 Trial & Answer only & \(-\)0.053 & A & B & C & D \\
3 Trials & Q + A + Ref & \(-\)0.111 & B & C & 2 Trials & Answer only & \(-\)0.325 & B & C & D \\
2 Trials & Answer only & \(-\)0.219 & B & C & 3 Trials & Q + A + Ref & \(-\)0.386 & B & C & D \\
3 Trials & Repeat Q + A & \(-\)0.349 & B & C & 3 Trials & Repeat Q + A & \(-\)0.382 & & C & D \\
3 Trials & Answer only & \(-\)0.776 & & C & 3 Trials & Answer only & \(-\)0.615 & & & D \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Emotional classification using Tukey analysis. Figure 5: Classification map of emotional satisfaction. emotions (joy, sadness, anger, fear, disgust, and surprise) [33] or valence-arousal (positive/negative and active/passive, VA) [34]. This method is simple for setting emotion standards; however, it may have the disadvantage of being limited in expressing the complex emotions of human beings specifically. The Konsei engineering approach used in this study can measure complex emotions because it is free to select any KW. In the case of pleasurability, it is possible to confirm it as a basic emotion, but reliability is difficult to confirm with basic emotions or the VA model. Several studies have been conducted to evaluate satisfaction with a VIS in advance. [PERSON] et al. [5] conducted a qualitative evaluation of Amazon Echo, focusing on interaction satisfaction and personalization based on user reviews. [PERSON] and [PERSON] [35] qualitatively evaluated Google Home using a survey on usability and user experience. Qualitative evaluation is difficult to express clearly in expressing satisfaction in a way that draws conclusions through subjective evaluation of experts based on individual opinions of users. In contrast, this study designed a VIS using the WoZ method based on user requirements and implemented an environment in which users can interact directly. The data were collected and evaluated using KWs, which express the user's emotions. This study can be seen as a more detailed study in that it goes beyond the verification of the effect of the VIS on the user's emotions, and identifies design parameters of the system that affect the user's emotions. Thus, it is believed that this study is the first to quantitatively confirm the direct emotion of the user in the interaction. In addition, our study can be viewed as more detailed in that it is possible to identify design parameters that influence the user's emotion among design parameters of the VIS and to quantitatively grasp the degree of its influence.