Developing balanced quality indicators for monitoring virtual care in Ambulatory Care Environments: A Modified Delphi Panel Process (Preprint)
Authors: Petrie Samuel , Makanjuola Oluwatoni , Laur Celia , Burke Emerelda , Rios Patricia , Bhattacharyya Onil , Mukerji Geetha 
Published: December 2024

BACKGROUND

While the volumes of virtual visits accelerating in uptake following the COVID-19 pandemic, less is understood regarding the quality of virtual visits. Furthermore, there is limited guidance for decision makers on how best to evaluate the quality of virtual care initiatives. There is a clear need for a comprehensive set of relevant quality indicators for implementation teams and organizations interested in either evaluating or implementing a virtual care initiative.

OBJECTIVE

The objective of this study was to curate a set of ambulatory quality indicators grounded in the quintuple aim quality framework and National Academy of Medicine domains of quality, including sustainability to provide a broad view of quality for virtual care in ambulatory environments.

METHODS

Methods: A three round modified Delphi panel process was conducted. Phase 1 involved curating a list of ambulatory virtual care quality indicators from published and grey literature along with stakeholder derived indicators generated through a pre-Delphi survey; these indicators were mapped and categorized to the Quintuple Aim and National Academy of Medicine (NAM) Quality Domains.. Phase 2 involved rating and ranking of the quality indicators. A 19-member pan-Canadian Delphi panel was convened. The panel included representation from patients, clinicians, policy-makers, academics, and administrators. Panelists rated indicators using the Agency for Healthcare Research and Quality measure attributes on 9-point Likert scale in round 1, followed by a virtual Delphi panel meeting to discuss indicators before round 2 of re-rating, before a final round 3 of ranking assessing importance of indicators within each quality domain and sub-domain. Novel quality indicators of virtual care to address gaps in literature were included through panel discussions, patient group consultations, and a pre-Delphi survey. For indicator advancement during rating exercises, 75% or more of panelist’s responses in the top tertile (between 7-9) with a median composite score of 7 or greater was required.

RESULTS

Results: 140 quality indicators were included and. progressed through the 3 Delphi rounds. There was minimal attrition among Delphi panel members (17/19 participated in all 3 rounds). After round 3, 27 indicators were compiled in the final scorecard. Aligned with the quintuple aim, these indicators include 14 centered on patient experience, 7 regarding provider experience, 2 focused on population health, 2 focused on health equity, and 2 focused on health system costs.

CONCLUSIONS

Conclusions: By creating set of comprehensive set of quality indicators of ambulatory virtual care relevant to stakeholders across the health system, comprehensive evaluations of virtual care programs can be conducted examining all relevant domains of quality. Using the balanced scorecard as a reference, organizations and institutions can compare projects across jurisdictions, and identify priority areas of quality to ensure their virtual care initiative is appropriate for the care being provided.

CLINICALTRIAL

This study was not registered