Development of a Standardized Set of Core Metrics to Assess the Quality of Anesthesia, Perioperative Care, and Acute Pain Management in Canada: Scoping Review and Multidisciplinary Consensus Delphi Study
Conditions: Anesthesia, Consensus Development
Sponsor: University of British Columbia

Development of a standardized set of core metrics to assess the quality of anesthesia, 
perioperative care, and acute pain management in Canada: scoping review and 
multidisciplinary consensus Delphi study 
 
Anesthesia Metrics Delphi (H22-00519), Protocol, v1, 11Apr-2022 
 
Page 1 of 17 
 
Study Protocol 
 
Janny Xuechen Ke MD MSc FRCPC1,2,3, Kathryn Sparrow MD MScHQ FRCPC 4, W. Scott 
Beattie MD PhD FRCPC 5, May-Sann Yee MD MEHP FRCPC6, Louise Sun MD SM 
FRCPC FAHA7, Mindy Smith MD MS9, Edlyn Lim MLis8, Matthias Görges PhD1, 10  
 
1 Department of Anesthesiology, Pharmacology and Therapeutics, University of British 
Columbia, Vancouver, Canada 
2 Department of Anesthesia, St. Paul's Hospital, Providence Health Care, Vancouver, Canada  
3 Department of Anesthesia, Pain Management & Perioperative Medicine, Dalhousie 
University, Halifax, Canada 
4 Department of Anesthesiology, Memorial University of Newfoundland, St. John’s, Canada 
5 Department of Anesthesia, University of Toronto, Toronto, Canada 
6 Department of Anesthesiology, Southlake Regional Health Centre, Newmarket, Ontario, 
Canada 
7 Departments of Anesthesiology and Epidemiology, University of Ottawa 
8 Department of Pediatrics, British Columbia Children’s Hospital 
9 Department of Family Medicine, Michigan State University, East Lansing, MI, USA 
10 Research Institute, British Columbia Children’s Hospital 
 
Trainees: Kang Mu Yoo, School of Medicine, University of British Columbia, Vancouver, 
Canada; Jielin Cai, Faculty of Health Sciences, Simon Fraser University, Burnaby, Canada 
 
Patient partners: Mindy Smith, MD, MS; Kootenay-Boundary Patient Advisory Committee  
 
Steering committee: Janny Ke, Kathryn Sparrow, Louise Sun, W. Scott Beattie, Matthias 
Görges  
 
Partner: Canadian Perioperative Anesthesia Clinical Trials Group 
 
Funding: University of British Columbia Canada’s Digital Technology Supercluster 
expansion research project grant (F19-03058; Perioperative opioid usage quality 
improvement); Michael Smith Health Research BC Scholar Award (M. Görges); Canadian 
Digital Technology Supercluster Consortium (Careteam Technologies Inc, Thrive Health Inc, 
Excelar Technologies, Providence Health Care Ventures Inc, and Xerus Inc.) project (J Ke; 
“Reducing Opioid Use for Pain Management”) 
 
 

Development of a standardized set of core metrics to assess the quality of anesthesia, 
perioperative care, and acute pain management in Canada: scoping review and 
multidisciplinary consensus Delphi study 
 
Anesthesia Metrics Delphi (H22-00519), Protocol, v1, 11Apr-2022 
 
Page 2 of 17 
 
Protocol Abstract  
Background 
Improving anesthesiology care and perioperative outcomes is a growing field encompassing 
quality improvement and quality assurance, research, and continuing medical education. 
Multiple consensus initiatives have standardized perioperative endpoints. However, no 
standard set of metrics exist to evaluate the quality of Canadian anesthesia care delivery.  
Objective 
Develop a consensus core list of metrics which can be used to evaluate the quality of 
anesthesia, perioperative care, and acute pain management for adult inpatient non-cardiac 
surgical patients for use in quality improvement and quality assurance, research, and 
continuing medical education in the Canadian care context. 
Methods 
First, the steering committee will conduct a scoping review of the literature to identify 
candidate metrics, according to Preferred Reporting Items for Systematic Reviews and Meta-
Analyses extension for scoping reviews (PRISMA-ScR) and Joanna Briggs Institute 
methodology for scoping reviews. Metrics will include patient-reported outcome and patient-
reported experience measures, quality and safety indicators, process metrics, and Canadian 
Anesthesiologists’ Society recommended practice standards. The search strategy will include 
English-language articles published or in use within the past seven years (2015/Jan-
2022/March) in 1) MEDLINE, Embase, CINAHL, Web of Science, and Cochrane Database 
of Systematic Reviews, 2) grey literature, including guidelines, and 3) existing evaluation 
metrics used by Canadian departments for quality assurance and improvement. Screening and 
data extraction will be performed by two independent reviewers using Covidence.  
Next, following approval from the Research Ethics Board and with written informed consent 
from participants, a multidisciplinary panel of anesthesiologists, people with lived surgical 
experience (patients, caregivers), surgeons, nurses, internal medicine and family physicians, 
hospital administrators, and researchers will iteratively review the list of candidate metrics 
using a modified Delphi process. Independent voting, using surveys over three rounds, will 
be used to identify the most important metrics to define the quality of anesthesia, 
perioperative care, and acute pain management. Additional cycles may be performed if 
required.  
Impact and dissemination  
This study will establish a core set of Canadian consensus metrics for evaluating anesthesia, 
perioperative care, and acute pain management. 

Development of a standardized set of core metrics to assess the quality of anesthesia, 
perioperative care, and acute pain management in Canada: scoping review and 
multidisciplinary consensus Delphi study 
 
Anesthesia Metrics Delphi (H22-00519), Protocol, v1, 11Apr-2022 
 
Page 3 of 17 
 
Study Protocol 
Introduction  
Improving anesthesiology care and perioperative outcomes is a growing field encompassing 
quality improvement and quality assurance, research, and continuing medical education (1). 
With the scope of anesthesiology practice expanding further into the perioperative domain, a 
greater focus on postoperative recovery and longer-term outcomes is warranted (2). Multiple 
consensus initiatives have standardized perioperative endpoints (3–5). However, no core list 
of metrics exists to evaluate the quality of Canadian anesthesia, perioperative care, and acute 
pain management in the context of a broad range of existing patient-reported outcome and 
experience measures, quality and safety indicators, process metrics, and practice standards.  
With the increasing adoption of Anesthesia Information Management Systems, Electronic 
Health Records, and patient-facing virtual care platforms, we have the opportunity to 
routinely and systematically collect patient-reported and system-collected data to measure 
anesthesiology performance. By enabling personalized feedback and learning health system 
approaches, a standardized set of metrics may provide the foundation for further targeted 
research and interventions to improve anesthesia, perioperative care, and acute pain 
management for Canadians (1). In this study, we will engage a diverse, multidisciplinary 
panel to answer the question: what defines quality in anesthesiology, perioperative medicine, 
and acute pain management in Canada, and how should we quantify it?   
Objectives 
The overall objective is to develop a consensus list of metrics which can be used to evaluate 
the quality of anesthesia, perioperative care, and acute pain management for adults 
undergoing inpatient non-cardiac surgeries that can be used for quality improvement and 
quality assurance, research, and continuing medical education in the Canadian care context.  
The specific metrics will be obtained using the following approach:  
1. Obtain a list of candidate metrics through a scoping literature review. For each metric, 
information will include a) definition(s), b) studies on validity/feasibility and level of 
evidence, c) timing of measurement, and d) population(s) studied 
2. All outcome metrics, as well as process/structure metrics that have limited supporting 
evidence, will go through a multidisciplinary modified Delphi consensus process, 
using surveys, to 1) select and mark the importance for each metric, and 2) select a 
core set of the most important metrics. 
 
 
 

Development of a standardized set of core metrics to assess the quality of anesthesia, 
perioperative care, and acute pain management in Canada: scoping review and 
multidisciplinary consensus Delphi study 
 
Anesthesia Metrics Delphi (H22-00519), Protocol, v1, 11Apr-2022 
 
Page 4 of 17 
 
Methods 
Study Design Overview 
First, the steering committee will conduct a scoping review of the literature to obtain a list of 
metrics that have been validated and/or are in clinical use to evaluate anesthesia, 
perioperative care, and acute pain management, to use as a starting point for the modified 
Delphi process. Then, following approval from the University of British Columbia Research 
Ethics Board and with written informed consent from participants, a multidisciplinary panel 
will identify the most important metrics using a modified Delphi process, in accordance with 
existing literature including the Core Outcome Set-STAndards for Development (The COS-
STAD) recommendations (3,6–11). In the last Delphi cycle, the panel will be asked to select 
the most essential items into a core set of metrics. 
Stage 1. Scoping Review 
We will first conduct a scoping review according to the Preferred Reporting Items for 
Systematic Reviews and Meta-Analyses extension for scoping reviews (PRISMA-ScR) (12) 
and the Joanna Briggs Institute methodology for scoping reviews (13). The purpose of this 
scoping review is to explore the evidence on the use of metrics to assess the quality of 
anesthesia, perioperative care, and acute pain management as reflected in the literature. A 
scoping review has been selected as the methodology of choice for this review because 
standardized metrics in perioperative care are an emerging area of research. The scoping 
review was registered on 2022-03-21 as https://doi.org/10.17605/OSF.IO/8BNVW.  
Search strategy  
The metrics included in our scoping review are patient-reported outcome and patient-reported 
experience measures, patient satisfaction, quality indicators, safety indicators, practice 
guidelines, process metrics, and other existing metrics. An initial search (Appendix 1) of 
MEDLINE (Ovid) was co-developed with guidance by a research librarian (E.L.).  
The search strategy for this review aims to identify both published and unpublished studies. 
A three-step search strategy will be utilized. An initial limited search of MEDLINE (Ovid) 
will be undertaken followed by an analysis of the text words contained in the titles and 
abstracts, and of the indexed terms used to describe the articles. A second full search strategy 
using all identified keywords and indexed terms will then be undertaken on MEDLINE 
(Ovid) and across all other databases. The other databases to be searched will include 
MEDLINE (Ovid), Embase (Ovid), CINAHL (EBSCOhost), Web of Science, and Cochrane 
Database of Systematic Reviews (Ovid). Third, the reference list of all identified reports and 
articles will be searched for additional studies.  
Studies from 2015/Jan-2022/March will be considered for inclusion in this review. This start 
date was chosen because end of 2014 was the end date of the last comprehensive searches in 
this area from previous studies (4,14). Only articles published in English will be included 
because of a lack of resources for translating non-English articles. Also, grey literature 
including guidelines, and existing evaluation metrics used by Canadian departments for 
quality assurance and improvement will be searched.  

Development of a standardized set of core metrics to assess the quality of anesthesia, 
perioperative care, and acute pain management in Canada: scoping review and 
multidisciplinary consensus Delphi study 
 
Anesthesia Metrics Delphi (H22-00519), Protocol, v1, 11Apr-2022 
 
Page 5 of 17 
 
Inclusion criteria:  
• Human adults  
Note that this was conducted using search terms that excluded journal names in the field 
of pediatric patients. First, the filters for “adult” are not the same for all databases that 
will be searched. For instance, the limit for adult on MEDLINE (Ovid) is age 19+ years 
and Embase (Ovid) is 18 to 64 years. Second, we found that some articles were not 
indexed by the age filter, which we did not wish to miss. As such, we will manually filter 
for articles with an adult age group (18+). 
• Must involve surgical patients requiring the presence of an anesthesiologist for anesthetic 
care, specifically for inpatients undergoing non-cardiac surgery.  
Note that cardiac surgery is not included as an exclusion in the search strategy but rather 
will be manually excluded during screening, as we found using search filters excluded 
non-cardiac surgeries.  
• Must include at least one metric, with at least one definition for the metric. We will 
include patient-reported outcome and patient-reported experience measures, Canadian 
guidelines, and Canadian Anesthesiologists’ Society recommended practice standards. 
Since there is an ongoing umbrella review registered on PROSPERO (15), we excluded 
quality indicators from the database search strategy and will use the results directly from 
the review. Since the umbrella review only included systematic reviews and may not 
account for metrics in recent guidelines and clinical use, we will include quality 
indicators from our grey literature search.   
• Published or in clinical use within the past 7 years (2015/Jan to 2022/March), to focus on 
contemporary metrics and because end of 2014 was the end date of the last 
comprehensive search in this area (14).  
Note that only the latest version of guidelines will be included. 
• For published studies only: full-text articles, study types being only experimental and 
epidemiological study designs (including randomized control trials, non-randomized 
control trials, quasi-experimental studies, before and after studies, prospective and 
retrospective cohort studies, case-control studies, and analytical cross-sectional studies), 
scoping/systematic review, and/or meta-analysis. 
Exclusion criteria: 
• For published studies only: we will exclude articles that are case reports, case series, 
letters, opinion pieces, commentaries, or editorials.  
Due to the availability of multiple reviews and consensus guidelines, these article types 
will likely not contribute relevant additional information that is not already covered by 
other included sources. Study protocols are not excluded as they may contain metrics 
being used in contemporary original research.  
• Studies reporting on ambulatory or outpatient surgery (i.e., those who do not require at 
least an overnight stay). 
• Articles published in a language other than English. 

Development of a standardized set of core metrics to assess the quality of anesthesia, 
perioperative care, and acute pain management in Canada: scoping review and 
multidisciplinary consensus Delphi study 
 
Anesthesia Metrics Delphi (H22-00519), Protocol, v1, 11Apr-2022 
 
Page 6 of 17 
 
Grey literature search: 
Since there exist multiple reviews on perioperative process and structure quality indicators to 
leverage, which included grey literature searches themselves, we will not perform grey 
literature search on these metrics. Instead, we will search websites of anesthetic and surgical 
societies in Canada, United States, United Kingdom, and Europe for guidelines and 
standards, and include existing standards from the Multicentre Perioperative Outcomes 
Group (MPOG) (published in 2015/Jan-2022/March as per above).   
Sources of guidelines to be searched will include:  
• CPG Infobase (https://joulecma.ca/cpg/homepage/) 
• ECRI Guidelines Trust (https://guidelines.ecri.org/) 
• Guideline Central (https://www.guidelinecentral.com/guidelines/) 
• Guidelines International Network (https://g-i-n.net/)  
• TRIP database (https://www.tripdatabase.com/ 
 
Existing metrics used in Canada: 
We will contact all Canadian anesthesiology departments as listed in a recent national survey 
(16) through the Canadian Anesthesiologists’ Society (e.g. Association of Canadian 
University Departments of Anesthesiology (ACUDA) and the electronic mailing list to 
department heads) and individually to obtain lists of metrics already being used to measure 
anesthesia, perioperative care, and acute pain management by email, with up to a maximum 
of two follow-up attempts by email. Departments will not be identified in which metrics they 
capture in their institutions.  
Inclusion and exclusion of metrics 
Based on previous literature, there are >200 potential metrics. In order to narrow down to a 
logistically manageable list of candidate metrics (approximately 50 prior to Cycle 1 of 
Delphi), we will use the following objective criteria.  
Inclusion: 
• The Delphi panel is most useful when evidence is limited. If there is a high level of trial 
evidence supporting a process indicator for influencing outcomes (e.g., Grade 1a), the 
metric will be part of the final metric (including core metric) lists without going through 
the Delphi cycles. Participants will be informed about these metrics at the start of the 
questionnaire for each Delphi Cycle.  
• Since research in patient-reported outcome and patient-reported experience measures is 
limited, we will include these metrics regardless of level of evidence. 
• Process metrics must have direct anesthesiologist involvement according to the standard 
of care.  
Exclusion: 
• Metrics specific to a certain surgery (i.e., not generalizable to the average patient 
undergoing inpatient surgery). 

Development of a standardized set of core metrics to assess the quality of anesthesia, 
perioperative care, and acute pain management in Canada: scoping review and 
multidisciplinary consensus Delphi study 
 
Anesthesia Metrics Delphi (H22-00519), Protocol, v1, 11Apr-2022 
 
Page 7 of 17 
 
• Structural indicators that cannot be changed at an individual and/or departmental level 
(e.g., availability of blood bank near a hospital). Structural indicators may be reflected in 
other metrics and can be examined during root cause analysis.  
• The process or structure indicator has strong evidence (Grade 1a) for lack of difference in 
patient or system outcomes.  
Article selection 
All articles found from the database searches will be uploaded into Covidence (Veritas 
Health Information, Melbourne, Australia) and duplicates will be removed. Titles and 
abstracts will be screened by two independent reviewers based on the inclusion/exclusion 
criteria for eligibility. Selected articles from screening will be retrieved in full-text and 
assessed by two independent reviewers against the inclusion/exclusion criteria to select 
studies for data extraction. Disagreements between reviewers will be resolved through 
discussion, and if not resolved, additional reviewer(s) will be involved in the discussion until 
a consensus is achieved. The search, screening, inclusion, and exclusion process will be 
reported in a final scoping review and demonstrated on the PRISMA flow diagram (12). 
Data Extraction 
Two independent reviewers will extract data from each article using a standardized data 
extraction form in Covidence that can be exported to Excel for further analysis (Appendix 2), 
including details about the participants included (cohort characteristics), methodology 
(definition(s) of metrics, validation of metrics, timing of measurement), and any other data 
relevant to the review questions (e.g., feasibility of implementation). Disagreements between 
reviewers will be resolved through discussion, and if not resolved, additional reviewer(s) will 
be involved in the discussion until a consensus is achieved. If required, the steering 
committee will contact authors for missing data, with up to a maximum of two attempts by 
email. If a systematic review is retrieved for a category of metrics, data will be directly 
extracted from the review, and only articles published after the search date of the review 
articles will undergo data extraction since the previous articles were likely already included. 
A draft data extraction form (Appendix 2) will be piloted on two studies and revised if 
needed and reported in the final manuscript.  
Scoping review data analysis and presentation 
The results of the scoping review will be synthesized narratively, in the form of tables and 
text, and supporting data will be directly extracted from the included studies or reports. For 
each metric, we will report definition(s) of metrics, type of metric per Donabedian category 
(structure, processes, and outcomes), validation, timing of measurement, number of patients 
and population studied, feasibility, type(s) of study. The steering committee may also group 
the metrics into specific themes or domains (e.g., patient comfort).  
Metrics may be grouped together amongst similar metrics if their definitions were deemed 
sufficiently similar as agreed per consensus amongst the steering committee. Patient-reported 
measures will be grouped by the outcome or experience of interest with a list of the specific 
measures, rather than the measures as individual items (for example, quality of recovery 
instead of the specific Measuring quality of recovery-15 (QoR-15) scale). 

Development of a standardized set of core metrics to assess the quality of anesthesia, 
perioperative care, and acute pain management in Canada: scoping review and 
multidisciplinary consensus Delphi study 
 
Anesthesia Metrics Delphi (H22-00519), Protocol, v1, 11Apr-2022 
 
Page 8 of 17 
 
The steering committee will work with the patient partner and the patient advisory group to 
ensure that the language describing the metrics is understandable to a general audience with 
at least grade six education, particularly for the process metrics. Within the Delphi survey, for 
each metric there will be a brief description of what it is and why it might be important, and 
expandable links for further explanation and details for each metric (e.g., the items for a 
patient-reported outcome or experience measure), as deemed necessary by the steering 
committee.  
Stage 2. Modified Delphi consensus process  
Study Population 
We aim to recruit 77 participants with the following characteristics (accounting for 30% 
attrition rate (9,17)):  
• 26 anesthesiologists 
• 26 patients and/or their family members/caregivers 
• 9 surgeons, one from each of the following specialties (may include clinician-
researchers):  
o General surgery 
o Neurosurgery  
o Obstetrics and gynecology  
o Orthopedic surgery  
o Otolaryngology  
o Plastic surgery 
o Thoracic surgery  
o Urology  
o Vascular surgery 
• 4 nurses 
• 8 physicians: 2 each from internal medicine, emergency department, family practice, 
family practice anesthetists  
• 4 health administrators or quality improvement leads  
 
Inclusion Criteria: Adult participants aged 18 and above will be included. Patient/family 
participants must have undergone a surgical experience within the past three years, and have 
the capacity to provide informed consent. All other participants must be actively practising in 
Canada in the disciplines they represent. 
Exclusion Criteria: Participants with self-declared conflicts of interest with this project.  
While the study will primarily be conducted in English using BC Children’s Hospital 
Research Institute’s REDCap survey system (18,19), to ensure representation, participants 
will not be excluded based on language or technology limitations. During recruitment, 
patients could indicate accommodations required, including translation and submitting study 
materials through postal mail. The steering committee will review the degree to which 
accommodations would be feasible within the human resources and study budget.  

Development of a standardized set of core metrics to assess the quality of anesthesia, 
perioperative care, and acute pain management in Canada: scoping review and 
multidisciplinary consensus Delphi study 
 
Anesthesia Metrics Delphi (H22-00519), Protocol, v1, 11Apr-2022 
 
Page 9 of 17 
 
Recruitment, consent, panel selection, and withdrawal 
We aim to recruit a geographically and culturally diverse sample of participants that is 
representative of the Canadian population and care settings, following principles of equity, 
diversity, and inclusion (EDI). We will aim to recruit broadly to obtain representation in 
areas including but not limited to: provinces and territories across Canada, gender, 
indigenous communities, visible minority groups, urban vs. rural, academic vs. community, 
anesthesia and surgical subspecialties, and surgical procedures.  
Information about the Delphi study will be distributed through multiple channels (2) 
including Canadian department administrators, Canadian Anesthesiologists' Society mailing 
lists, Perioperative Anesthesia Clinical Trials group, hospital and provincial support for 
patient oriented research (SPOR) organizations, established patient engagement platforms 
(e.g. REACH BC, an established patient research engagement and recruitment platform for 
BC’s health authorities and partner universities), social media (Twitter, Facebook, 
Instagram), patients in participating hospitals, who granted permission to contact for research 
purposes, and targeted contact of pre-identified participants.  
We request waiver of the requirement for a formal informed consent process as the data for 
this minimal risk behavioural study is collected exclusively using surveys. With respect to  
TCPS2 3.7A: a) this research represents a minimal risk to our participants; b) the alteration to 
the consent process is unlikely to affect the welfare of our participants adversely; c) it would 
be impracticable to conduct the research if we adopted a formal consent process (involving 
scheduling phone calls with participants, mailing or emailing consent materials to review 
beforehand, following up with an electronic consent form), especially as we expect our 
participants to be distributed across multiple time zones in Canada; d) the initial survey 
(study information and eligibility check) explains that participants are not required to 
complete it or any subsequent surveys and we propose that this is sufficient to imply consent, 
as it is typical for survey-based research; and e) there will be no plan to debrief participants, 
but at each stage they can email the steering committee to request that their data be 
withdrawn up to the start of the next Delphi round. 
Interested participants will be invited to complete a study information and eligibility check 
survey in REDCap, which will be available via a shared public link. This survey will present 
study information that will allow them to confirm their eligibility, and establish their 
willingness and ability to participate, before submitting their preferred contact email address, 
which will be used to route all subsequent surveys. 
Participant data will be de-identified using study ID throughout the study. The participant 
will be informed that de-identified results from the eligibility survey will be collected to 
report the study inclusion/exclusion flow, and that if they do not want these data collected 
then they should not fill the eligibility survey.  
A participant can withdraw from the study and/or their consent to data sharing for research at 
any time. If this happened prior to the start of the Delphi Round 1, the research team will 
attempt to recruit replacement(s) if possible.   

Development of a standardized set of core metrics to assess the quality of anesthesia, 
perioperative care, and acute pain management in Canada: scoping review and 
multidisciplinary consensus Delphi study 
 
Anesthesia Metrics Delphi (H22-00519), Protocol, v1, 11Apr-2022 
 
Page 10 of 17 
 
Participants who completed all components of the study (demographics survey and Delphi) 
will each receive an electronic gift card of $75 by email as honorarium); those who only 
complete the initial round will receive an electronic gift card of $25.  
Full demographics survey 
After providing electronic informed consent, a full demographics survey will be sent to each 
participant for completion within two weeks. This is based on previous methodology and will 
include age (in decades), sex, gender, race/ethnicity, and geographical category (urban, semi-
urban, rural) of hospital they most frequently interact with for the purposes of surgeries. For 
patients and/or family members only, we will also ask about their highest education level 
(grade school, high school, diploma or undergraduate degree, postgraduate degree), income 
(by Canadian tax brackets), ethnicity, home community location if different from the hospital 
location (urban, semi-urban, rural), occupation, and type(s) of surgery. For health 
professionals, we will also ask about primary practice area, years in practice (<5, 5-9, 10-
19, >20 yrs), subspecialty training, practice setting (academic, community), and involvement 
in any of research, quality assurance/improvement, or teaching continuing medical education.   
Delphi process 
The multidisciplinary panel of participants will iteratively review the list of candidate metrics 
using a modified Delphi process. While specific guidance on consensus thresholds and the 
optimal scales are limited, the design will follow best practices from literature (3,6–11). In 
the last Delphi cycle, the panel will be asked to select the most essential items to include into 
a core set of metrics. 
Independent, anonymous voting using REDCap over three rounds will be used to identify the 
most important metrics to define the quality of anesthetic care. Each round will be 
coordinated through the Department of Anesthesiology, Pharmacology and Therapeutics, 
Faculty of Medicine, UBC. 
Participants will be given at least two weeks of notice about an upcoming round, and will 
have two weeks to complete the REDCap survey for a given round. Reminder emails will be 
sent 7 and 14 days after the survey opens to those who have not yet completed it. If a survey 
is still not completed, up to two additional reminders will be sent once a week, and if still no 
response, the round will be closed. The participant who missed a round will still be able to 
participate in future rounds. There will be at least two weeks between each round to provide 
time for the steering committee to analyze the data and prepare for the next round. Additional 
cycles (beyond the 3rd) may be performed if the objectives have not been achieved.  
Delphi Round 1 
The initial list of metrics and descriptions from the scoping review will be presented. 
Participants will independently score each of the listed metrics for importance on a five-point 
scale (from ‘1=not at all important’ to ‘5=very important’) (8). Participants may also enter 
comments in free-text if they have any specific feedback or suggestions on a given item, 
which will be summarized and shared with participants in Round 2. The mean and median 
scores for each metric will be calculated for overall panel and for subgroups of 1) 

Development of a standardized set of core metrics to assess the quality of anesthesia, 
perioperative care, and acute pain management in Canada: scoping review and 
multidisciplinary consensus Delphi study 
 
Anesthesia Metrics Delphi (H22-00519), Protocol, v1, 11Apr-2022 
 
Page 11 of 17 
 
patient/caregiver and 2) health provider groups, and will also be presented in Round 2. 
Metrics receiving a mean score ≥4 by ≥65% of each subgroup and ≤2 by <20% of the 
subgroups will be retained for voting in Round 2 (8).  
Delphi Round 2 
Participants will score each metric using the same scale as in Round 1, and suggest revisions 
or comments as needed. Metrics receiving a mean score ≥4 by ≥65% of each subgroup and 
≤2 by <20% of the subgroups will be retained for voting and prioritizing in Round 3. 
Delphi Round 3  
Prior to Round 3, the study team will review the list of metrics, and aggregate and/or refine 
their wording as needed. Participants will be presented with a list of metrics with mean and 
standard deviation scores from the Likert scales, and be asked to vote on which metrics 
consist of a core set of problems (i.e., must be addressed by the health system as a priority) 
using a three-point scale of “definitely priority”, “equivocal (either way)”, and “not a 
priority” (10,20). Metrics where >70% participants voted “definite” will be marked in the 
publication as the core metric set. If no consensus is reached, there may be additional rounds 
of rating using a smaller sample of participants.   
 
Statistical analysis 
Analysis will be performed using R (R Foundation for Statistical Computing, Vienna, 
Austria). Cohort characteristics will be reported using descriptive statistics. Each round of 
Delphi will be analyzed as described above, and presented both as an overall group and as 
individual subgroups. 
Sample size calculation  
There is limited guidance regarding sample size for Delphi processes, but, in general, the key 
is to choose a broad representation from key stakeholders with a margin of safety built in for 
possible attrition. Delphi studies often involve a total of 17 members (9) and that adding 
more participants above 23 did not alter the consensus results (21). We have aimed for broad 
representation across all the key stakeholder groups. Thus, we chose n = 20 for each of the 
groups of anesthesiologists and patients. We chose the number of surgeons to represent the 
primary set of noncardiac surgical specialties. We also accounted for an attrition rate of 30% 
per literature (9,17) to ensure that there will be sufficient representation in the last cycle 
despite attrition.   
Impact and future directions 
This study will develop a set of national consensus metrics for evaluating the quality of 
anesthesia, perioperative care, and acute pain management that can be used for quality 
improvement and quality assurance, research, and continuing medical education in the 
Canadian care context. It aligns with the top ten priorities in Canadian anesthesiology 
research (2) by informing the core postoperative outcomes to measure. We will engage the 
Canadian Anesthesiologists’ Society, particularly the Quality and Patient Safety and 
Standards Committees, for knowledge mobilization. The study will be published in an 

Development of a standardized set of core metrics to assess the quality of anesthesia, 
perioperative care, and acute pain management in Canada: scoping review and 
multidisciplinary consensus Delphi study 
 
Anesthesia Metrics Delphi (H22-00519), Protocol, v1, 11Apr-2022 
 
Page 12 of 17 
 
anesthesia journal and presented at relevant conference(s). Future steps include the 
identification of the best measures for the core set of metrics according to the COnsensus-
based Standards for the selection of health Measurement Instruments (COSMIN) guidelines 
(22).  
References  
1.  
Glance LG, Neuman M, Martinez EA, Pauker KY, Dutton RP. Performance 
Measurement at a “Tipping Point.” Anesth Analg. 2011 Apr;112(4):958–66.  
2.  
McKeen DM, Banfield JC, McIsaac DI, McVicar J, McGavin C, Earle MA, et al. Top 
ten priorities for anesthesia and perioperative research: a report from the Canadian 
Anesthesia Research Priority Setting Partnership. Can J Anaesth J Can Anesth. 2020 
Mar 10;  
3.  
Myles PS, Boney O, Botti M, Cyna AM, Gan TJ, Jensen MP, et al. Systematic review 
and consensus definitions for the Standardised Endpoints in Perioperative Medicine 
(StEP) initiative: patient comfort. Br J Anaesth. 2018 Apr;120(4):705–11.  
4.  
Haller G, Bampoe S, Cook T, Fleisher LA, Grocott MPW, Neuman M, et al. Systematic 
review and consensus definitions for the Standardised Endpoints in Perioperative 
Medicine initiative: clinical indicators. Br J Anaesth. 2019 Aug;123(2):228–37.  
5.  
Boney O, Bell M, Bell N, Conquest A, Cumbers M, Drake S, et al. Identifying research 
priorities in anaesthesia and perioperative care: final report of the joint National Institute 
of Academic Anaesthesia/James Lind Alliance Research Priority Setting Partnership. 
BMJ Open. 2015 Dec 1;5(12):e010006.  
6.  
Graham B, Regehr G, Wright JG. Delphi as a method to establish consensus for 
diagnostic criteria. J Clin Epidemiol. 2003 Dec 1;56(12):1150–6.  
7.  
Diamond IR, Grant RC, Feldman BM, Pencharz PB, Ling SC, Moore AM, et al. 
Defining consensus: a systematic review recommends methodologic criteria for 
reporting of Delphi studies. J Clin Epidemiol. 2014 Apr;67(4):401–9.  
8.  
Boney O, Moonesinghe SR, Myles PS, Grocott MPW, Bartoszko J, Beattie WS, et al. 
Core Outcome Measures for Perioperative and Anaesthetic Care (COMPAC): a 
modified Delphi process to develop a core outcome set for trials in perioperative care 
and anaesthesia. Br J Anaesth. 2022 Jan 1;128(1):174–85.  
9.  
Boulkedid R, Abdoul H, Loustau M, Sibony O, Alberti C. Using and reporting the 
Delphi method for selecting healthcare quality indicators: a systematic review. PloS 
One. 2011;6(6):e20476.  
10.  De Meyer D, Kottner J, Beele H, Schmitt J, Lange T, Van Hecke A, et al. Delphi 
procedure in core outcome set development: rating scale and consensus criteria 
determined outcome selection. J Clin Epidemiol. 2019 Jul;111:23–31.  
11.  Kirkham JJ, Davis K, Altman DG, Blazeby JM, Clarke M, Tunis S, et al. Core Outcome 
Set-STAndards for Development: The COS-STAD recommendations. PLOS Med. 2017 
Nov 16;14(11):e1002447.  
12.  Tricco AC, Lillie E, Zarin W, O’Brien KK, Colquhoun H, Levac D, et al. PRISMA 
Extension for Scoping Reviews (PRISMA-ScR): Checklist and Explanation. Ann Intern 
Med. 2018 Oct 2;169(7):467–73.  

Development of a standardized set of core metrics to assess the quality of anesthesia, 
perioperative care, and acute pain management in Canada: scoping review and 
multidisciplinary consensus Delphi study 
 
Anesthesia Metrics Delphi (H22-00519), Protocol, v1, 11Apr-2022 
 
Page 13 of 17 
 
13.  Peters MDJ, Marnie C, Tricco AC, Pollock D, Munn Z, Alexander L, et al. Updated 
methodological guidance for the conduct of scoping reviews. JBI Evid Implement. 2021 
Mar;19(1):3–10.  
14.  Chazapis M, Gilhooly D, Smith AF, Myles PS, Haller G, Grocott MPW, et al. 
Perioperative structure and process quality and safety indicators: a systematic review. 
Br J Anaesth. 2018 Jan 1;120(1):51–66.  
15.  Nguyen F, McIsaac D, Lalu M, Pysyk C, Hamilton G. Evaluation of important quality 
indicators in the perioperative period: an umbrella review. PROSPERO 2020 
CRD42020164691 [Internet]. [cited 2022 Feb 12]. Available from: 
https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=164691 
16.  Kazemi P, Lau F, Simpao AF, Williams RJ, Matava C. The state of adoption of 
anesthesia information management systems in Canadian academic anesthesia 
departments: a survey. Can J Anaesth J Can Anesth [Internet]. 2021 Jan 29; Available 
from: https://rdcu.be/cesb5 
17.  Bamber JH, Lucas DN, Plaat F, Allin B, Knight M, collaborators for the Obstetric 
Anaesthetists’ Association Quality and Outcomes Working Group. The identification of 
key indicators to drive quality improvement in obstetric anaesthesia: results of the 
Obstetric Anaesthetists’ Association/National Perinatal Epidemiology Unit 
collaborative Delphi project. Anaesthesia. 2020 May;75(5):617–25.  
18.  Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG. Research electronic 
data capture (REDCap)--a metadata-driven methodology and workflow process for 
providing translational research informatics support. J Biomed Inform. 2009 
Apr;42(2):377–81.  
19.  Harris PA, Taylor R, Minor BL, Elliott V, Fernandez M, O’Neal L, et al. The REDCap 
consortium: Building an international community of software platform partners. J 
Biomed Inform. 2019 Jul;95:103208.  
20.  Lange T, Kopkow C, Lützner J, Günther K-P, Gravius S, Scharf H-P, et al. Comparison 
of different rating scales for the use in Delphi studies: different scales lead to different 
consensus and show different test-retest reliability. BMC Med Res Methodol. 2020 Feb 
10;20:28.  
21.  Akins RB, Tolson H, Cole BR. Stability of response characteristics of a Delphi panel: 
application of bootstrap data expansion. BMC Med Res Methodol. 2005 Dec 1;5(1):37.  
22.  COnsensus-based Standards for the selection of health Measurement Instruments 
(COSMIN). COSMIN - Improving the selection of outcome measurement instruments 
[Internet]. COSMIN. [cited 2022 Feb 27]. Available from: https://www.cosmin.nl/ 
 
 

Development of a standardized set of core metrics to assess the quality of anesthesia, 
perioperative care, and acute pain management in Canada: scoping review and 
multidisciplinary consensus Delphi study 
 
Anesthesia Metrics Delphi (H22-00519), Protocol, v1, 11Apr-2022 
 
Page 14 of 17 
 
Appendix 1. Scoping Review MEDLINE Search Strategy  
 
Ovid MEDLINE(R) and Epub Ahead of Print, In-Process, In-Data-Review & Other Non-
Indexed Citations, Daily and Versions <1946 to March 16, 2022> 
# 
Query 
Results from 
17 Mar 2022 
1 
exp Anesthesia/ 
200,448 
2 
exp Anesthetics/ 
255,070 
3 
Anesthesiology/ 
32,529 
4 
Anesthesia Recovery Period/ 
5,440 
5 
(an?esthesia* or an?esthetic* or an?esthesiolog*).tw,kf. 
323,470 
6 
or/1-5 
539,493 
7 
exp Perioperative Care/ 
157,136 
8 
Perioperative Medicine/ 
92 
9 
(perioperative adj2 (car* or medicine* or period*)).tw. 
17,106 
10 
(perioperative and (car* or medicine* or period*)).kf. 
2,218 
11 
or/7-10 
170,469 
12 
Patient Comfort/ 
567 
13 
Patient Safety/ 
24,260 
14 
Patient Satisfaction/ 
87,426 
15 
(patient* adj2 (comfort* or safet* or satisfaction*)).tw. 
95,583 
16 
(patient* and (comfort* or safet* or satisfaction*)).kf. 
13,730 
17 
Patient Reported Outcome Measures/ 
11,066 
18 
(patient reported adj4 (outcome measure* or experience 
measure*)).tw. 
8,214 
19 
(patient reported and (outcome measure* or experience measure*)).kf. 
3,051 
20 
Postoperative Complications/ 
389,613 
21 
Intraoperative Complications/ 
33,217 
22 
((peroperative1 or postoperative or intraoperative) adj2 
complication*).tw. 
90,933 
23 
((peroperative or postoperative or intraoperative) and 
complication*).kf. 
10,527 
24 
Outcome Assessment, Health Care/ 
79,846 
25 
((health care or healthcare) adj4 outcome assessment*).tw. 
27 
26 
((health care or healthcare) and outcome assessment*).kf. 
1,119 
27 
or/12-26 
719,962 
28 
6 and 11 and 27 
6,932 
29 
exp animals/ not humans.sh. 
4,973,232 
30 
28 not 29 
6,842 
31 
(child* or adolesc* or pediat* or paediat*).jn. 
281,321 
 
1 This is not a misspelled word, the MeSH term uses this wording. 

Development of a standardized set of core metrics to assess the quality of anesthesia, 
perioperative care, and acute pain management in Canada: scoping review and 
multidisciplinary consensus Delphi study 
 
Anesthesia Metrics Delphi (H22-00519), Protocol, v1, 11Apr-2022 
 
Page 15 of 17 
 
32 
30 not 31 
6,679 
33 
exp case-control studies/ 
1,296,071 
34 
(case$ and control$).tw. 
557,532 
35 
(case$ and series).tw. 
213,282 
36 
case reports.pt. 
2,255,114 
37 
(case$ adj2 report$).tw. 
638,230 
38 
(case$ adj2 stud$).tw. 
282,881 
39 
(comment or editorial or letter).pt. 
2,044,294 
40 
opinion piece.tw. 
447 
41 
or/33-40 
5,990,942 
42 
32 not 41 
4,949 
43 
limit 42 to english language 
3,816 
44 
limit 43 to yr="2015 -Current" 
1,200 
 
 

Development of a standardized set of core metrics to assess the quality of anesthesia, 
perioperative care, and acute pain management in Canada: scoping review and 
multidisciplinary consensus Delphi study 
 
Anesthesia Metrics Delphi (H22-00519), Protocol, v1, 11Apr-2022 
 
Page 16 of 17 
 
Appendix 2. Scoping Review data extraction form  
 
Extractor 1 initial  
 
Extractor 2 initial 
 
Citation details 
Title 
 
Author 
 
Year  
 
Doi link 
 
Publication type  
Multiple choice: systematic review +/- 
meta-analysis, scoping review, other 
review type, randomized control trials, 
non-randomized controlled trials, quasi-
experimental studies, retrospective 
cohort, prospective cohort, before and 
after studies, case-control studies, 
analytical cross-sectional studies, 
guideline, departmental metric, other 
(free text)  
Participants 
Country  
 
Region(s) 
 
Hospital setting  
Academic vs. non-academic 
Urban vs. regional vs. rural 
Population (e.g. disease, age group, etc) 
 
Sample size  
 
Surgery type(s) 
 
Duration of follow-up  
 
Age (mean) 
 
Age (median)  
 
Age (standard deviation) 
 
Age (range) 
 
Sex or gender  
 
American Society of Anesthesiologists 
Physical Status  
 
Length of stay   
 
Other demographics information 
 
Metric(s) information  
Please fill in the below for each complication investigated in the study 
Metric name  
 
Donabedian category  
Structure, process, and outcomes 
- 
Can select multiple  
How was the metric defined? 
 
How was the metric validated in this study? If 
not available, what references were cited in this 
study from previous validations? 
 

Development of a standardized set of core metrics to assess the quality of anesthesia, 
perioperative care, and acute pain management in Canada: scoping review and 
multidisciplinary consensus Delphi study 
 
Anesthesia Metrics Delphi (H22-00519), Protocol, v1, 11Apr-2022 
 
Page 17 of 17 
 
Measurement instrument for the metric (e.g. 
questionnaire name, special equipment, AIMS, 
history, physical exam, imaging, lab test) 
 
Mode of delivery of the measurement 
instrument (e.g. paper questionnaire, emailed; 
online tool, EHR extraction) 
 
Timepoint(s) of measurement 
 
Response rate(s) 
 
Any additional comments or results in the 
study related to feasibility. If not, what 
references were cited in this study from 
previous feasibility studies? 
 
Any additional results related to reliability. If 
not, what references were cited in this study 
from previous publications? 
 
Sampling strategy used for the metric (e.g. 
everyone, random sampling, stratified, etc.) 
 
Any other potentially relevant information