Domain coverage and criteria overlap across digital health technology quality assessments: a systematic review
Authors: Frey Anna-Lena , Phillips Ben , Baines Rebecca , McCabe Adam , Elmes Evelyn , Yeardsley-Pierce Emily , Wall Rachel , Parry Jake , Vose Alice , Hewitt Jack , Coburn Justine , Dowdle Curtis , Sollitt Leyla , Leahy Matthew , Hunt Sophie , Andrews Tim , Leigh Simon 
Published: May 2024

Background: Digital health technologies (DHTs) have the potential to improve health outcomes and increase health system efficiency. However, to obtain these benefits, it is necessary to distinguish between high- and low-quality DHTs across domains such as effectiveness, clinical safety, data privacy, and usability. For this purpose, numerous DHT quality assessments have been developed. This review aimed to examine the background characteristics, domain coverage, and criteria overlap of such assessments.Methods: Assessment frameworks were identified through systematic searches of PubMed/MEDLINE, Embase and PsychINFO. Evaluation criteria of eligible assessments were extracted, and each criterion was assigned to one of twelve domains. The percentage of criteria included in each domain was compared across frameworks with different characteristics using Mann-Whitney U tests. Additionally, criteria overlap was examined among health system and government led third-party assessments. Results: The literature search identified 130 assessment frameworks that met eligibility criteria. Higher relative criteria coverage of the clinical safety and/or data privacy domains was observed for assessments that were published after, rather than in or before, 2016 (P privacy= .030), and that were developed with, rather than without, patient (P safety= .006; P privacy= .034) or healthcare professional (P safety= .005; P privacy= .015) input. Among health system and government led third-party assessments, the highest median criteria overlap was observed in the credibility domain at 78.2%, followed by the data privacy domain at 64.0%. The domains of security and technical robustness demonstrated the lowest criteria overlap at 42.1% and 30.7%, respectively.Conclusion: The observed differences in domain coverage based on stakeholder input and publication date highlight the importance of considering different perspectives and current best practices when developing and updating DHT quality assessments. Moreover, while the high criteria overlap found in some domains is encouraging, low overlap in other areas raises concerns regarding duplicated work and potential discrepancies in evaluation results, if a given DHT needs to meet varying standards across different assessments. To address these concerns, it would be beneficial for assessment owners to agree on a set of baseline criteria, which could be supplemented with separately-scored, context-specific criteria as needed. This would streamline evaluation processes and increase trust in assessment outcomes, thereby driving the adoption of high-quality DHTs and ultimately improving patient care quality and safety.Note: This preprint has not undergone peer review or any post-submission improvements or corrections. The Version of Record of this article is published in Health and Technology and is available online at https://doi.org/10.1007/s12553-024-00934-8 . For non-subscribers to the journal, a read-only version of the peer-reviewed paper is available here: https://rdcu.be/d4eJh .