# Process-Aware Enactment of Clinical Guidelines through Multimodal Interfaces
## 5 User Evaluation
### Evaluation Setting and Results of the First User Study  
The two developed working prototypes have been tested with patients suffering from chest pain (cf. Section 2.3), and therefore the related CG has been modeled, configured and deployed on the system to perform the testing. The initial user study was performed in Policlinico Umberto I hospital in Rome with the help of the Department of Emergency and Admissions (DEA). In Figure 8, a doctor is shown using the TESTMED system to enact the CG on a patient simulator. In this experiment, five postgraduate medical students and two doctors participated in the study. Given a patient simulator that was supposed to reflect a real patient with chest pain symptoms, the participants were asked to use the TESTMED system to visit the patient according to the related CG (see Figure 8). After the completion of the user testing, a questionnaire was provided to the participants with the aim to gather their background information and collect data about how they perceived the interaction with the system. Specifically, the questionnaire consisted of 11 statements covering aspects like easiness of use of the GUI, quality of the multimodal interaction, etc. The answers were evaluated through a 5-point Likert scale, which ranged from _1--strongly disagree_ to _5--strongly agree_, to reflect how participants agreed/disagreed with the defined statements:  
1. I have a good experience in the use of mobile devices. 2. The interaction with the system does not require any special learning ability. 3. I judge the interaction with the touch interface very satisfying. 4. I judge the interaction with the vocal interface very satisfying. 5. I think that the ability of interacting with the system through the touch interface or through the vocal interface is very useful. 6. The system can be used by non-expert users in the use of mobile devices. 7. The system allows for constantly monitoring the status of clinical activities. 8. The system correctly drives the clinicians in the performance of clinical activities. 9. The doctor may--at any time--access data and information relevant to a specific clinical activity. 10. The system is robust with respect to errors. 11. I think that the use of the system could facilitate the work of a doctor in the execution of its activities. Table 1 summarizes the results of the first user study. From such results, we can infer that the general attitude of the participants towards our system was positive. Results put in light that participants have considered the system as effective in the enactment of CGs, since it was able to concretely orchestrate doctors with executing the clinical activities included in the CG (cf. results of Q8). Furthermore, the system allowed doctors to constantly monitor the status of each clinical activity  
Figure 8: A doctor using the TESTMED system in a ward during the visit of a patient simulator. (cf. results of Q7) and to easily access information relevant to the specific activity under execution (cf. results of Q9). Participants also showed a fair amount of satisfaction with how the system behaves with respect to error handling (cf. results of Q10), learnability (cf. results of Q2), and ease of use for non-expert users (cf. results of Q6). On the negative side, the interaction with the vocal interface was considered as quite unsatisfactory (cf. results of Q4), while high satisfaction was experienced using the touch interface (cf. results of Q3). Nonetheless, the participants agreed that a multimodal interaction involving both touch and vocal interface could be useful to facilitate the work of doctors (cf. results of Q5). It is worth noticing that the questionnaire allowed participants to also add feedback and comments in free text. Using this feature, five out of seven participants explicitly asked us to develop an improved vocal interaction for the system, enabling doctors to dynamically switch the interaction modality (from vocal to touch, or vice versa) when needed.