# Evaluation of Methodologies and Measures on the Usability of Social Robots: A Systematic Review
## 3 Results
### Evaluation Methods
#### 3.2.1 Type of Stimuli  
Generally, it is best to provide a product which has already completed the development process to the users and use it as a stimulus for evaluation. It is helpful to improve a product when the users are able to give feedback after using a final product in their natural environment. However, in the case of social robots, it is inefficient to evaluate the product after its actual physical development in consideration of its long and complicated development process. When we want to evaluate a product that is still in the stage of development, we provide users with various type of stimulus that can make up for unimplemented features of the product. Considering the particularity of this field, this paper analyzed which type of stimuli are usually provided and evaluated in the assessment, and what are the pros and cons of each type. Among the 36 articles reviewed in this study, we found four types of stimuli used in evaluating the usability social robots: (1) a textual description of the robot, (2) a still image showing the robot, (3) a video showing the appearance and interaction of robot, (4) live interaction between the participant and the robot. Founded types of stimuli are summarized in Table 1.  
Text. There was only one study that used a stimulus delivered in the textual description and it was illustrated with photographs of the robot [9]. They used a text stimulus for comparing the effect of scenario media. It is difficult to describe the appearance of a robot in text, so a text scenario could be more appropriate to deliver the function of a robot but not its appearance. It is also mentioned in [9] that a text scenario is effective when evaluating the robot's functionality and users' general attitude toward a robot. Images. There was four research that used a still image as a stimulus for evaluation [10; 11; 12; 13]. Those research aimed to evaluate the robot's appearance to identify appropriate design or facial expression of robots. Images are relatively efficient in terms of time and cost, and it can also be designed for a specified purpose. However, in using images, it is hard to fully grasp different experiences of interaction between the user and a robot. In recent research, an image that represents a robot can't be understood by people as being with the subject in a shared context and it yields different behavioral responses than \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Type of Stimuli** & **Description** & **Count** & **Ratio** \\ \hline \multirow{2}{*}{Text} & A text stimulus can be a short description & \multirow{2}{*}{1} & \multirow{2}{*}{3\%} \\  & summarizing the product concept. & & \\ \hline \multirow{2}{*}{Image} & An image stimulus is used to show the & \multirow{2}{*}{4} & \multirow{2}{*}{10\%} \\  & appearance of product in perspective. & & \\ \hline \multirow{4}{*}{Video} & A video stimulus is more realistic and dynamic & \multirow{2}{*}{4} & \multirow{2}{*}{10\%} \\  & version of storyboard to communicate a possible & & \\ \cline{1-1}  & use case clearly. & & \\ \hline \multirow{2}{*}{Live Interaction} & Giving a working prototype or a finished product & \multirow{2}{*}{31} & \multirow{2}{*}{77\%} \\  & to user \(\backslash\) to interact with freely. & & \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Summary of stimuli type. physical robots in real-world [7]. Image is static and can't describe the delicate difference among robots' movements or changes of facial expression. Taking this limitation into account, using an image as a stimulus for social robot evaluation can only be effective if the subject for evaluation is the users' first impression towards the appearance of the robot. Video. In many research, a video analysis was chosen commonly and was shown to be an effective research method to meet the goal of understanding user's perceptions of robotic technologies [10]. Despite known limitations [14], a video stimulus has advantages, including consistent experience across participants.