## 5 Evaluation of the Proposed Measures  
For expert validation of the results of the developed analysis techniques, we interviewed two ICU-doctors and two ICU-nurses (from MUMC+) on the understandability and the usefulness of the linguistic summaries. Below, a summary of our findings is presented as well as a concluding discussion. We used semi-structured interviews with nine questions regarding the protocol compliance measures developed as a research tool. The questions first explored the meaning and importance of protocol compliance, then how understandable some example sentences from the linguistic summaries of their ICU data were, and finally, how useful they think the linguistic summaries would be in their daily practice. The usefulness and understandability (as a component of ease-of-use) are two latent variables that have the most influence on the adoption of a new technology, including tools, techniques, methods and guidelines [77; 78]. The questions are presented in Appendix A. We recorded the interviews, transcribed them, and next, we analyzed them them dramatically. During the interviews, it was acknowledged by all interviewees that protocol compliance is very important: _\"Yes, I personally think that it [compliance to clinical protocols and guidelines in ICU] is very important. For me as well as a newly qualified nurse. This is just this one thing that gives you the guidance how to use certain equipment or how to make some medicine or how to give some medication. I use them a lot.\"_ (Nurse C). **Understandability. ** When being presented with a set of linguistic summaries, all interviewees agreed that they are understandable, although the nurses had some concerns with the current way of working: _\"My first response would be, they [almost all patients have some of their activities compliant] all are understandable. Whether I find them useful, I think that the first two are useful to me, and maybe the third one is less useful in practice, although you could argue that almost all patients have only some of their activities compliant, so that there is a poor compliance to the protocol. They are all understandable, and relevant\"_. (Doctor W) _\"I understand the sentences well. For instance the first one \"many patients have almost all of their activities compliant\" is less strong then most patients. Almost all patients, means that there are almost no patients non-compliant. For me it is understandable.\"_ (Nurse S) _\"Yes, they [some patients have most of their activities compliant] are clear. Of course, we as ICU we, mostly work with numbers, percentages, so we are not used to work with this kind of information.\"_ (Nurse S) _\"Yes, it is comprehensible. [... but] I'm dyslectic. They all look the same.\"_ (Nurse C). \begin{table}
\begin{tabular}{c c c c} \hline \hline  & **Note A** & **Note C** & **All** \\ \hline \multicolumn{4}{c}{Strict compliance} \\ \hline \% of compliant conditions & 53.4\% & 55.5\% & 54.5\% \\ \% of fully compliant checks & 0.12\% (2) & 0.06\% (1) & 0 \\ \% of non-compliant conditions & 2.8\% & 14.9\% & 9.7\% \\ \hline \multicolumn{4}{c}{Fuzzy compliance} \\ \hline Fuzzy compliance (average based) [13] & 54.2\% & 43.9\% & 48.3 \% \\ \hline \multicolumn{4}{c}{OWA-based fuzzy compliance} \\ \hline most (Q\({}_{1}\)) patients have most (Q\({}_{2}\)) of their activities compliant some (Q\({}_{1}\)) patients have most (Q\({}_{2}\)) of their activities compliant some (Q\({}_{1}\)) patients have most (Q\({}_{2}\)) of their activities compliant & 0 & 0 \\ \hline \multicolumn{4}{c}{OWA-based fuzzy compliance with activity importance} \\ \hline most (Q\({}_{1}\)) patients have most (Q\({}_{2}\)) of their important activities compliant some (Q\({}_{1}\)) patients have most (Q\({}_{2}\)) of their important activities compliant & 0 & 0 \\  
**Usefulness. ** When discussing the usefulness of the linguistic summaries, we identified different points of view between the doctors and the nurses.