# Process-Aware Enactment of Clinical Guidelines through Multimodal Interfaces
## 5 User Evaluation
### Evaluation Setting and Results of the Second User Study  
The results and findings of the first user study were leveraged to refine the weak aspects of the first prototype, in order to realize a (more) robust second working prototype of the system. If compared with the first prototype, the second one provided a more elaborated design of the GUI, together with a redefinition of the interaction principles underlying the vocal interface. For example, in the first prototype, the vocal features of the system were always active during the enactment of a CG. This resulted in many \"false positives\", i.e., the system wrongly recognized as proper vocal commands (by consequently activating unwanted functionalities) some words pronounced by the doctor during the patient's visit. In the second prototype, to prevent false triggers, we decided to activate the vocal interface only after a specific (and customizable) key vocal instruction pronounced by the doctor. Leveraging the second working prototype, we performed a second user study employing the same chest pain CG used in the first user study. In addition, the second user study took place at the DEA of Policlinico Umberto I in Rome. Seven users (different from the ones that were involved in the first user study) participated in the second user study, including six postgraduate medical students and one doctor. Like in the first user study, participants attended the patient simulator and were asked to use the second prototype of the system to enact the CG (see Figure 8). They also completed the same questionnaire employed in the first user study to assess the effectiveness of the system. The results of this second user study are shown in Table 2.  
From the analysis of the results, it is evident that the good feelings obtained in the first user study about the working of the system have been confirmed by the results of this second user study. If we refer to Figure 10, we can note that participants' ratings in the second user study increased for all statements if compared with the first user study. Moreover, the results highlighted that the design of the second prototype made considerable progress, in particular because we precisely followed for its development the traditional design guidelines for building multimodal GUIs [10]. \begin{table}
\begin{tabular}{l c c c c c c c c c c c} \hline \hline  & **Q1** & **Q2** & **Q3** & **Q4** & **Q5** & **Q6** & **Q7** & **Q8** & **Q9** & **Q10** & **Q11** \\ \hline
**User1** & 4 & 4 & 4 & 4 & 5 & 3 & 4 & 4 & 4 & 4 & 4 \\
**User2** & 4 & 4 & 5 & 3 & 5 & 2 & 3 & 4 & 2 & 4 & 3 \\
**User3** & 3 & 3 & 3 & 4 & 4 & 3 & 4 & 4 & 3 & 3 & 4 \\
**User4** & 5 & 4 & 3 & 2 & 4 & 5 & 5 & 4 & 4 & 5 & 4 \\
**User5** & 1 & 5 & 5 & 5 & 5 & 5 & 5 & 5 & 5 & 5 & 5 \\
**User6** & 3 & 5 & 4 & 5 & 4 & 5 & 4 & 4 & 5 & 5 \\
**User7** & 3 & 4 & 5 & 5 & 4 & 4 & 5 & 4 & 5 & 4 & 4 \\ \hline
**Avg** & 3.2 & 4.16 & 4.14 & 4 & 4.3 & 3.86 & 4.29 & 4.14 & 3.86 & 4.29 & 4.14 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Results of the second user study One critical aspect was the interaction with the vocal interface (cf. statement Q4), which resulted as being quite unsatisfactory in the first user study, with an average rating of 2,7. Conversely, the improved vocal interface employed in the second working prototype was really appreciated by the participants in the study, with an average rating of 4. To confirm that the improvement of the vocal interface was not the result of a coincidence, we analyzed the ratings for the statement Q4 collected in the first and second user studies leveraging the 2-sample \(t\)-test. This statistical test is applied to compare whether the average difference between two population means is really statistically significant or if it is due instead to random chance. The results of the 2-sample \(t\)-test are summarized in Figure 11. Statistical significance of the results is determined by looking at the p-value, which gives the probability that the collected results have been obtained randomly.