# HIPPP: Health Information Portal for Patients and Public
## 4 Results and Validation Approach
### Evaluation Pipeline
#### 4.2.1 Time Efficiency Study  
As a key aim of the HIPPP project is to create a system that efficiently evaluates WBHI, we conducted a study to evaluate and compare the evaluation time of the system versus that of a human expert evaluator. Experimental Setup  
In order to gather data for the time taken for a human expert to evaluate a sample of WBHI with respect to the QUEST framework, we first recruited a senior internist to conduct the evaluations. We subsequently trained them in how to apply the QUEST framework to evaluate WBHI. In order to conduct the study, we then provided them with a tool for recording their evaluation scores and timings along with five samples of WBHI (unknown to them) from five Irish news organisations/blogs which contained articles pertaining to the diagnosis, treatment or prevention of colon cancer. The timing began as soon as the evaluator launched the sample of WBHI in their web browser and concluded when they had recorded scores for each of the seven QUEST criteria to give an accurate recording of the actual time spent evaluating a given sample of WBHI. For comparison, we ran the same five WBHI samples provided to the human evaluator through the HIPPP evaluation pipeline using two configurations. The first configuration evaluated the seven QUEST criteria sequentially, one after the other (akin to the procedure a human would apply the QUEST framework). The second configuration was optimised for evaluation time, whereby it evaluates the seven QUEST criteria in parallel, the trade-off being a larger processing/memory footprint. The experiment for evaluating the HIPPP pipeline was conducted on a machine with an Apple M1 Pro 8-core (CPU) and 16 GB DDR5 RAM, without GPU acceleration. In order to mitigate the effect that any performance issues on the machine may have on the results, each configuration of the HIPPP evaluation pipeline was run on the five samples 10 times, with the results being averaged. Experimental Findings  
The time efficiency study yielded the following results:  
Across the five WBHI samples, the human evaluator took a mean time of 229 \(\pm\) 54.81 s, the sequential configuration of the HIPPP pipeline took a mean time 16.83 \(\pm\) 4.8 s and the run-time optimised HIPPP configuration took a mean time 2.76 \(\pm\) 0.76 s, as seen in Figure 12. This equates to a time efficiency improvement of 92.65% (i.e., a factor 13.6) and 98.79% (i.e., a factor 83) for HIPPP and the run-time optimised HIPPP configuration versus that of an expert human evaluator. As a part of this study, we also investigated the effect that the length of an article has on the evaluation time both for a human evaluator and the HIPPP pipeline. The result of this can be seen in Figure 13. We fitted a linear regression line to each of the evaluation methods. In each of the three cases, it appears that there is a linear relationship between the number of tokens and the time taken to evaluate the documents. Intuitively, it would make sense that longer documents take more time for all methods of evaluation; however, it appears that the effect is more significant for human evaluators. Given the steeper line in the context of manual evaluation (red line in Figure 13), it would appear that there is a larger coefficient in the function that defines the relationship between the number of tokens and the evaluation time. As a result, it is possible in the case of very long documents that the performance gap between manual evaluation and HIPPP would be even wider. However, given the small sample size, we are unable to make a definite assertion as to whether this is the case. Figure 12: The mean evaluation time for each of the three evaluation methods (Manual, HIPPP, HIPPP Runtime Optimised), across the five samples of WBHI. An important caveat to the time efficiency study is that it assumes the real-time availability of a human expert evaluator at any time, for immediate evaluation.