## 3 Experiments
### Experimental Setting
#### 3.1.2 Planning  
Planning consists in identifying the subjects, preparing the material, specifying how the experiment is going to be performed and formulating the hypothesis. The subjects participating in the experiments were 13 people with different expertise profiles in the field of PROforma modeling, including members of our research group and graduate students, all of them with an IT background. The material used includes both the PROforma models and the exercises. Table 3 lists the PROforma models used. Except for the CHF and COPD ones, which were developed by our research group, the models come from the OpenClinical.net repository [30] and have been developed by independent modellers. Most of the models are based on guidelines from well-recognized international organizations (see _Source guideline_ column in Table 3). Thus, our experiment included guidelines developed by a wide range of medical institutions: the British Thoracic Society (BTC), the Scottish Intercollegiate Guidelines Network (SIGN), the European Society of Cardiology (ESC), the Global Initiative for Chronic Obstructive Lung Disease (GOLD), the American College of Chest Physicians (ACCP), the UK National Institute for Health and Care Excellence (NICE), the New England Medical Center from the US (NEMC), and the Accident Compensation Corporation from New Zealand (ACC). The exercises consisted in questions or tasks, specifically defined for each PROforma model, to be solved by the subjects. The exercises were designed to reflect the analyzability and modifiability characteristics; therefore, they were classified into the _Analysis_ and _Modifiability_ categories (_Type_). Analysis exercises are those dealing with understanding the logical structure and/or the dynamic behavior of the model, and are usually formulated in terms of questions, e.g., about the results of model execution. Modifiability exercises are those where specific model changes are requested based upon a set of requirements. As mentioned before, it was intended that the exercises had different levels of complexity. Accordingly, we agreed that the exercises should fall in the categories _Trivial_, _Average_, and _Substantial_ (_Difficulty_). Once the subjects have solved the exercises, the solutions must be rated. To this end, we initially planned to use the grades Correct (C), Partially Incorrect (PI), in case of minor flaws, or Incorrect (I). As an illustration of the exercises designed, below are given an analyzability exercise and a modifiability one, both related to the top-level plan of Asthma CIG (see Figure 1). The two exercises were classified as trivial (T) in complexity: \begin{table}
\begin{tabular}{l l l l} \hline \hline
**Model** & **Goal** & **Source Guideline** & **Size** \\ \hline Asthma & Assessment and treatment of asthma in adults & BTS/SIGN (UK) & 46 \\ CHF & Diagnosis and treatment of chronic heart failure & ESC (EU) & 89 \\ COPD & Diagnosis, management, and prevention of & GOLD (worldwide) & 57 \\ Cough & Diagnosis and treatment of chronic cough & ACCP (US) & 28 \\ CRcaTrage & Colorectal referral and diagnostic & NICE (UK) & 7 \\ Depression & Management of depression in primary care & NEMC (US) & 18 \\ Dyspepsia & Differential diagnosis of dyspepsia & N/A & 4 \\ Head Injury & Work-up and management of acute head injury & NICE (UK) & 34 \\ IBME\_TB & Screening for tuberculosis & unknown & 14 \\ Statins & Management of patients at elevated risk of & NICE (UK) & 24 \\  & coronary heart disease using statins & \\ STIK & Assessment, investigation and management of & \\  & soft-tissue injury of the knee & ACC (NZ) & 26 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: PROforma models used in the experiment.