# META-GLARE: A Computer-Interpretable Guideline System Shell
## 5 Evaluating Meta-Glare  
Proposing an evaluation of a tool like META-GLARE is a complex matter since it is basically a \"shell\" with which to design and develop CIG systems. To the best of our knowledge, there are no analogous shells in the literature and, consequently, no benchmarks, making it impossible to propose \"direct\" comparisons. Roughly speaking, META-GLARE is a shell supporting software designers/developers in the modification or creation of new CIG systems. As such, it is also quite impossible to propose significant quantitative performance evaluations: designing and developing a new system (with or without META-GLARE) depends on the complexity of the system and the ability of the designers/programmers. As a consequence, concerning \"usability and performance\", we could only consider a few case studies in this paper and discuss the suitability of META-GLARE to cope with them (see Section 6). Indeed, it could be appropriate and interesting to carry out an experimental evaluation of performance in which two homogeneous groups of CIG system designers and developers (one using META-GLARE and the other acting as a control group) have to define a theoretically well-specified new CIG system, in order to perform a quantitative (time required to build the new system) comparative performance evaluation. Such an evaluation would be very costly and time-consuming and is out of the scope of this paper. Indeed, even the evaluation of CIG systems themselves is quite complex and controversial. They are mostly tools _supporting_ experts and knowledge engineers in the acquisition of clinical guidelines and _supporting_ physicians in their execution/application on specific patients. Thus, \"standard\" computer science evaluations (e.g., verification of correctness, analysis of performance) do not seem to apply to CIG systems, since both the correctness of the operations being performed and the time used to carry them out deeply depend on the users being supported. Indeed, we are not aware of any of such evaluations in the CIG literature). In their review, [PERSON] and [PERSON] have proposed a comparative evaluation of different CIG systems, considering eleven parameters [8]. Such an evaluation also considers the GLARE system. Considering the eleven parameters in [8], META-GLARE behaves exactly like GLARE; the evaluation is reported on the following parameters (here we adopt the terminology in [8]):  
1. _existence of a repository of guidelines: Yes_
2. _presence/absence of a tool offering an editor to create and visualize the guidelines: Yes_
3. _formal representation language used for the guidelines: Task Network_
4. _basic elements defined in the guideline representation language: Query, Work, Decision_
5. _if the tool is designed to be deployed as a distributed system: No_
6. _presence of complex coordination elements: Yes_
7. _type of execution engine: Rule-Based_
8. _connection with an electronic medical record: Yes_
9. _ability to integrate the execution engine with an existing clinical management system: Yes_
10. _use of any standard terminology or representation language: XML, ICD-9_
11. _inclusion of security tools to preserve data integrity and authenticate the accesses to the medical data: No_  
In the research area of CIGs, there is a wide consensus that the main parameter along which CIG systems should be evaluated and compared is the _expressiveness_ of the representation formalism. Several individual or comparative evaluations of CIG formalism expressiveness have been proposed in the research literature. For instance, ref. [6] provides a comparative evaluation of Asbru, EON, GLIF, Guide, PROforma, and PRODIGY. Workflowpatterns are the consensus benchmark to evaluate the expressiveness of the formalisms offered by workflow systems (see [17]). Recently, such a benchmark has also been applied to CIG formalism, in the milestone comparative analysis in [23] which assesses the state of the art in the CIG research area.