# An Abstraction Layer Exploiting Voice Assistant Technologies for Effective Human--Robot Interaction
## 5 Use Case Evaluation
### User Experience  
The functionalities of the robot were demoed for 30 min to 10 participants chosen among students and researchers on different topics of the University of Cagliari (6 males and 4 females). The average of their age was 31.2 with a standard deviation of 3.37. We asked them to perform the following tasks with the robot and for the following amount of time: sentiment analysis (3 min), conversation with the robot (5 min), ask the robot to perform some actions (4 min), ask the robot to identify some objects (3 min), play Bingo (5 min) and chess (5 min). Each user filled a survey: the first part included seven open questions (the first five also included five levels of reply, _very bad, bad, normal, good, very good_), whereas the second part was a standard System Usability Scale (SUS) ([[https://www.usability.gov/how-to-and-tools/methods/system-usability-scale.html](https://www.usability.gov/how-to-and-tools/methods/system-usability-scale.html)]([https://www.usability.gov/how-to-and-tools/methods/system-usability-scale.html](https://www.usability.gov/how-to-and-tools/methods/system-usability-scale.html)) (accessed on 30 June 2021)) questionnaire aimed to evaluate the application's usability. In the following, we summarize their answers to the open questions. **Q1. How do you find the interaction with the robot? ** Six users gave a _good_ score and four _very good_. **Q2. How effectively did the robot classify positive and negative sentences? ** Five users gave a score of (_very good_), whereas five more gave _normal_, as they found some errors in the ability to recognize the correct sentiment. **Q3. How effectively did the robot perform the action commands you gave? ** Seven users gave (_good_) mentioning that the robot was able to perform the action from a list. Three users gave _normal_ complaining about the robot that was it not always able to recognize their commands. It was easy to fix this problem because users employed peculiar words not covered within the dictionary. The solution was to update the dictionary with the new words. **Q4. How effectively did the robot identify objects? ** Eight users gave _good_ as the robot correctly identified the object they showed. Two users gave _normal_ because they used the application at an incorrect distance from the robot. We fixed this by letting the robot say, when the object detection module was triggered, to maintain a certain distance from it. **Q5. How effective was the robot in playing Bingo or chess? ** All users gave _good_ as the robot played both the games smoothly and the users had fun playing with it. **Q6. What are the main weaknesses of the resulting interaction? ** The only complaint was pointed out by two users mentioning that tiredness of the user might appear when used extensively. **Q7. Can you think of any additional features to be included in the robot interaction? ** The suggested features were: (1) to have the robot say instructions better when it was turned on and each time a certain module was triggered; (2) the ability to save all the data pertaining to the interaction with the user. The SUS questionnaire confirmed the good opinion of the users, scoring 83.2/100, equivalent to an A grade, and placed our use case in the 93% percentile rank (percentiles of SUS [[https://measuringu.com/interpret-sus-score/](https://measuringu.com/interpret-sus-score/)]([https://measuringu.com/interpret-sus-score/](https://measuringu.com/interpret-sus-score/)) (accessed on 30 June 2021)). All users felt very confident in playing with the proposed use case (with an average score of 4.1 \(\pm\) 0.4) and thought that it was easy to use (4.3 \(\pm\) 0.3). In addition, they were happy to use it frequently (4.2 \(\pm\) 0.2) and did not think that it was complex (1.2 \(\pm\) 0.3) or that they would need the help of a technical person to use it in the future (1.5 \(\pm\) 0.7).