# Augmented Reality, Mixed Reality, and Hybrid Approach in Healthcare Simulation: A Systematic Review
## 3 Results
### Evaluation of Simulators (FQ9-10)  
To guarantee proper simulator-based training it is important to verify whether improvements in performance on medical simulators translate into improved performance on real patients. In [53], the authors affirm that the evaluation of medical simulators has to satisfy two main criteria that are \"validity\" and \"reliability\". Overall, the \"validity\" is the degree to which a method measures something. The main types of validity are face, content, construct, concurrent, and predictive validity. In the medical simulation context:  
* Face validity refers to simulator realism, and it is assessed by experts by means of questionnaires or surveys;
* Content validity measures the appropriateness and usefulness of the simulator as a training tool, and it is typically assessed by experts with checklists;
* Construct validity determines the ability of simulator to differentiate between expert and novices;
* Concurrent validity indicates the correspondence between trainees' performance tested on a simulator and on a gold standard method or against another, previously validated, simulator;
* Predicitive validity denotes the ability of the simulator to predict future performance in real scenarios [53; 54]. \"Reliability\" refers to the consistency of measurements. However, there are two other less important criteria that are \"fairness\" and \"usability\": the first is an aspect of validity, and it is refers to absence of any bias (test free of bias, lack of favoritism toward test takers, accessibility, and validity of score interpretations), while the second concerns implementation costs, time required, ease of administration, and comprehensibility of the results for the users [53; 54]. To answer question FQ9 (\"Which evaluation method is performed to validate the simulator? (technical, validity, technical + validity)\"), we have defined three evaluation stages: only technical evaluation, only validity evaluation, both validity and technical evaluations. The technical evaluation concerns accuracy, reliability, and efficiency, and only nine studies out of 23 report a technical evaluation [22; 27; 28; 31; 35; 36; 38; 42; 43]. In [35; 38], the authors evaluate the displays in terms of registration accuracy [35; 38] and frames-per-second (FPS) during the visualization of surface and volume data [35]. Measurements of force are performed in [27; 42]. [PERSON] et al. [27] evaluate the physical components of the simulator measuring the average force during transpedicular insertions performed on six vertebral models to perform a comparison with human specimens. [PERSON] et al. [42] instead evaluate whether the developed haptic interface and the mechanical components satisfy the specifications in terms of force requirements. In [23; 28; 36; 43], the authors evaluate the accuracy of simulators that include an AR component. Among them, [PERSON] et al. in [23] evaluate the accuracy of a simulator developed for Microsoft HoloLens. The accuracy was evaluated in terms of the perceived position of AR targets; moreover, the authors evaluate the workload and usability of the HoloLens for their application considering visual and audio perception, interaction, and ergonomics issues. In [31], the authors evaluate the reliability and efficiency of the simulator. The authors evaluate the accuracy of the registration between real and virtual scenarios using the Euclidean distance between the 3D point where the resident hit the equipment in the simulation model and the actual 3D point in the real world, and the robustness, checking if the registration error depended on landmark points. Additionally, the authors demonstrated the efficiency of the system. Validity evaluation: Eleven studies out of 23 report face/content/construct/concurrent/predictive validity tests [22; 25; 29; 30; 32; 33; 34; 37; 39; 40]. Among them, three studies [29; 32; 39] perform a non-detailed preliminary evaluation of simulators, and the results obtained should be confirmed in larger studies.