## 5 Experimental Setup
### Results
#### 5.3.1 Qualitative Evaluation: SASSI Questionnaire  
To provide a qualitative evaluation of the user studies reported in this work, the SASSI questionnaire was used, since it allows to comprehensively evaluate different aspects of KIDE4I in their different adaptations. Furthermore, so as to cover other areas that are relevant for this study regarding the system's industrial application, additional questions have been added to SASSI (_vid._ Appendix A). Thus, the aspects covered by the questionnaires are the following [41]:  
* Response accuracy (SASSI). It refers to whether the system is able to interpret an user command and generates an appropriate response. * Likeability (SASSI). It refers to the user perception of the system in terms of usefulness, pleasantness and friendliness. * Cognitive demand (SASSI). It stands for the mental effort required by the user to interact with the system. In industrial scenarios, this aspect is especially relevant, since one of the main objectives of KIDE4I is to simplify the performance of specific tasks. * Annoyance (SASSI). This aspect evaluates how repetitive or annoying is to interact with the system. * Habitability (SASSI). It refers to whether the user knows what to say to the system. * Speed (SASSI). It evaluates if the system response given a user interaction is fast. * Verbosity. It determines whether the system interactions are too long. In this sense, the system should give the correct amount of information so as users can perform their tasks in the minimum amount of time. * Productivity. This question aims to determine if the fact of using this system would increase the user's productivity, as it is also one of the main objectives of KIDE4I. \begin{table}
\begin{tabular}{c c c c c c} \hline \hline (a) & Gender & (b) & Age & (c) & Interaction with Everyday Devices \\ \hline M & 50\% & **24–34** & 42\% & **Never** & 8.3\% \\ \hline F & 50\% & **35–44** & 33.3\% & **Sometimes** & 83.3\% \\ \hline  & & **45–54** & 16.6\% & **Frequently** & 8.3\% \\ \hline  & & **55–59** & 8\% & & \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Demographic data for participants in the KIDE4 BinPicking user study. (a) Gender information. (b) Age information. (c) Frequency of voice interaction with everyday devices. * Security. As user security is of utmost importance in industrial scenarios, this item evaluates if the design of the system allows users to perform the intended tasks by preserving a secure distance. Each question in SASSI consists of a 6-point Likert scale, where 1 stands for _Strongly disagree_ and 6 to _Strongly agree_. Figures 7 and 8 provide the results obtained from the questionnaire for the SASSI questions and for the additional questions. In general, it can be observed that the results are very positive in both use cases, with median scores between 5 and 6 for common questions between use cases and 4 and 6 for additional questions for the bin-picking use case. The results in Figure 7 show that some aspects had more consensus between participants than others. The aspects that denoted more variability were **annoyance**, **habitability** and **speed**. For **annoyance**, it can be observed that most answers are among the highest ratings, but also a significant amount of answers--in the sense that these observations are not outliers--have obtained the lowest ratings. This is caused by the results obtained  
Figure 8: Results obtained from user questionnaires: additional questions for the bin-picking use case. Figure 7: Results obtained from user questionnaires. (**a**) Results obtained for the guide use case. (**b**) Results obtained for the bin-picking use case. (**c**) Results obtained considering both experiments. for question number 24 (\"_The interaction with the system is repetitive_\"), which was rated with an average score of 2.38. Since the tasks to perform in the user studies were strongly related to industrial scenarios, which inherently consist of very specific actions on specific elements, this was an expected outcome.