Understanding the Cognition and Decision Making of Community Anesthesiologists in Their Management of End-of-case Neuromuscular Blockade: A Mixed Methods Study
Conditions: Neuromuscular Blockade
Sponsor: Vanderbilt University Medical Center

      MISP Template: Protocol 
1 
Final Version: 8/13/2010 
 
Study Title: 
  
Understanding the cognition and decision making of community 
anesthesiologists in their management of end-of-case neuromuscular 
blockade: A mixed methods study. 
 
NCT 04195178 
 
IRB Approval Date: 
 
June 13, 2023 
 
Institution Name 
 
Vanderbilt University Medical Center (coordinating center and study 
site) 
 
Investigator Contact 
Information: 
- Full address  
- Phone No. 
- Fax  No. 
- e-mail address 
 
Matthew B. Weinger, MD 
Center for Research and Innovation in Systems Safety (CRISS) 
Vanderbilt University Medical Center  
1211 21st Avenue S.  
Medical Arts Building Suite 732 
Nashville, TN 37212 
615-936-6598 
615-936-7373 
matt.weinger@vumc.org 
 
 
 
 
 
 

      MISP Template: Protocol 
2 
Final Version: 8/13/2010 
2.1 Objectives & 
Hypotheses  
 
 
 
 
2.1.1.  Study Objectives 
 
We propose a remotely conducted prospective observational mixed-methods 
study using clinical vignettes and cognitive interviews (CIs) to better 
understand the decision making approaches and preferences of practicing 
community anesthesiologists with regard to the management of non-
depolarizing neuromuscular blockade (NMB) at the end of general endotracheal 
anesthesia cases. The objectives of this initial study are to begin to understand: 
1. When and why do community anesthesiologists choose to reverse NMB? 
2. What are the preferences of community anesthesiologists regarding their 
decisions to reverse NMB, and what are the clinician and clinical case 
factors that influence those decisions?   
3. What are the operational knowledge gaps of anesthesia professionals 
regarding their ability to follow best-practice evidence for the reversal of 
NMB? 
4. Why do anesthesia professionals diverge from best-practice approaches 
and guidelines for the reversal of NMB? 
More generally, the program objective is to refine and validate a novel 
methodology to delineate clinicians’ decision-making strategies and triggers. 
2.1.2.  Study Hypotheses  
 
For this initial study, we hypothesize that: 
1. The clinical vignette-based decision survey method will describe the 
distribution of clinical variables and factors important to the decisions for 
community anesthesiologists’ decision to reverse NMB. 
2. There will be measurable differences in clinician demographics and practice 
attributes between those who closely follow evidence-based expectations 
for reversal of NMB and those who do not. We expect community 
practitioners who do not reverse NMB in accordance with current best 
evidence to be older, to not be fellowship trained or subspecialty certified, 
and to practice in lower acuity settings. 
2.2 Background & 
Rationale, 
Significance of 
Selected Topic & 
Preliminary Data  
 
2.2.1.  Background and Significance 
 
Residual neuromuscular blockade (NMB) following general anesthesia 
continues to be a major problem for post-surgical patients.1-4 Numerous studies 
now demonstrate that 20% to almost 60% (median of ~40%) of post-operative 
patients have residual blockade (i.e., weakness),1,5-7 which can cause respiratory 
insufficiency, aspiration, and pneumonia.1,7-10 Neither clinical signs of motor 
strength (e.g., hand grip, head lift)11,12 nor qualitative twitch monitoring (i.e., 
visual or tactile assessment of train-of-four)13,14 are reliable indicators of 

      MISP Template: Protocol 
3 
Final Version: 8/13/2010 
adequate NMB reversal. While qualitative twitch monitoring is widely used, 
quantitative NMB monitors is a much more reliable method of assessing the 
degree of NMB and determining the need for reversal.3 
 
Until the introduction of suggamadex, reversal of NMB agents (most 
commonly rocuronium or vecuronium) was accomplished with neostigmine or 
another anti-cholinesterase drug. These drugs can be ineffective if dosed 
inappropriately,15 although a recent study by Murphy et al., showed that 
neostigmine administration in the face of near complete recovery of NMB was 
not associated with more weakness, a common misconception.12 While 
suggamadex use appears to significantly reduce the incidence of residual NMB, 
it does not eliminate the problem,1,16,17 especially if quantitative NMB 
monitoring is not used. 18 
 
As a result of ample scientific and clinical evidence, there is now a consensus 
among knowledgeable experts on the best practices for management of NMB.4 
Yet, a substantial proportion of community anesthesiologists do not follow these 
practices. They:  
1) Do not use quantitative NMB monitors (which do have technical challenges 
to their use;19)  
2) Use clinical signs (e.g., hand grip, head lift) and/or qualitative twitch 
monitoring (e.g., presence of 4 twitch with a train of four stimulation);8,13  
3) Fail to reverse or inadequately reverse NMB (e.g., because a sufficient duration 
of time has elapsed since the last dose of NMB);11,20 and 
4) Do not believe that their patients ever experience significant adverse events 
due to residual neuromuscular blockade.20,21 
 
 
We know very little about the underlying reasons and motivation for these 
common clinical behaviors (decisions) that deviate from evidence-based best 
practices. Some have assumed that these practitioners have knowledge gaps (i.e., 
that they do not know or believe the literature evidence). However, many 
physicians fail to practice according to best evidence even when they know that 
evidence.22 
 
Decision Making in Anesthesia. While there is ample medical literature 
about ‘normative’ (analytical) decision-making23,24 as well as clinical decision-
making in chronic care settings,23,25,26 much less is known about decision making 
in acute care situations like anesthesiology. Prior research, primarily in other 
domains like aviation and the military, in ‘naturalistic decision-making’27,28 (also 
called ‘fast thinking’23) are relevant but currently provide limited guidance on 
the role of different contextual factors in specific acute care decisions. We know 
that cognitive biases can adversely affect clinical decisions29,30 but, with some 

      MISP Template: Protocol 
4 
Final Version: 8/13/2010 
exceptions (risk tolerance in emergency room admitting decisions),31,32 little is 
known about how such biases affect most clinical decisions.  
 
Studying the ‘Why’ of Decision Making. Current methods to study why 
clinicians make specific decisions have limitations. Traditional survey methods 
have been unreliable and are prone to various biases that diminish the 
likelihood that individual’s responses will accurately reflect actual practices. 
Survey methods have not been paired with immediate follow-up one-on-one 
cognitive interviews that can get at the underlying reasons behind the recorded 
answers as we now propose to do in the revised methods. Retrospective data 
mining, which depends on routine clinical documentation of the variables of 
interest can tell you what individuals do (within the constraints of 
documentation accuracy and completeness) but not why. Direct observations 
during clinical care may be the most reliable method of discerning what 
clinicians do but are inefficient, expensive, and without cognitive probes or 
interviews will not adequately reveal the rationale for the decisions (e.g., it 
could be the “right” decision but for the wrong reasons). Yet, asking the 
clinician for their decision rationale in the moment can be disruptive and is 
often unwelcome. 
 
Observation of behavior during simulated care is more efficient and cost-
effective but again, without additional methods, does not adequately inform our 
understanding of the cognitive processes underlying those behaviors. In 
contrast, cognitive interviews, such as the Critical Decision Method,33 target the 
underlying cognitive processes associated with prior or hypothetical decisions. 
While perhaps the best method to ascertain the contextual factors influencing 
decisions, cognitive interviews can still be influenced by memory reshaping, 
hindsight bias, and other cognitive biases. In the present study, we propose to 
combine low-fidelity simulations (using clinical vignettes) that force clinicians 
to make targeted decisions followed immediately by focused cognitive 
interviews to discern why they made those specific decisions. 
 
Our research team has conducted studies of decision-making and situation 
assessment, primarily of anesthesiologists during simulated care.34-40 We have 
also done appreciable cognitive research informing the design and evaluation of 
informatics-based decision support.41-43   
 
Since the original proposal, Naguib et al (ref) conducted an innovative survey 
study to assess anesthesiologists’ understanding of existing evidence regarding 
neuromuscular blockade pharmacology and monitoring. They conducted an 
internet-based international survey of practicing anesthesiologists asking 
respondents to answer nine true/false questions related to the use of neuromuscular 
blocking drugs. Participants were also asked to rate their confidence in the accuracy 
of each of their answers on a scale of 50% (pure guess) to 100% (certain of 

      MISP Template: Protocol 
5 
Final Version: 8/13/2010 
answer). 1,629 anesthesiologists completed the survey. The respondents correctly 
answered only 57% of the questions. In contrast, the mean confidence exhibited by 
the respondents was 84%, significantly greater than their accuracy. Of the 1,629 
respondents, 1,496 (92%) were overconfident. While the authors clearly 
demonstrate overconfidence by anesthesiologists’ about their knowledge regarding 
neuromuscular blocking agent pharmacology and monitoring, it is unknown to 
what extent this overconfidence contributes to their previously demonstrated 
deviation from evidence-based guidelines {REF}. Also, their study design did not 
optimally measure actual knowledge about this content domain nor how 
anesthesiologists approach management decisions about neuromuscular blockade, 
monitoring and reversal in actual practice. 
2.2.2 Rationale for Study and Choice of Methods 
 
We are interested in studying why experienced clinicians make certain 
decisions regarding the monitoring and reversal of neuromuscular blockade and, in 
particular, why they make decisions that are contrary to known evidence and 
currently accepted best practices. As described above, the decision to reverse NMB 
at the end of a general anesthetic is a good choice for study because: 1) it has been 
well documented that a substantial percentage of practitioners do not follow best 
practices; 2) the variables are largely circumscribed and well described; and 3) it 
can be framed as a binary decision. 
 
The goals of our original proposal, achieving a more complete understanding of 
anesthesia professionals’ management of neuromuscular blockade, remain 
important to improving patient safety. With the  revised protocol, we will also 
advance general knowledge about the use of cognitive interviews (CIs), particularly 
in combination with surveys, to investigate clinicians’ clinical management 
decisions.   
2.2.3 Rationale for Proposed Change in Study Methods 
 
Because this study was originally designed to be conducted one-on-one in-
person with community anesthesiologists, the onset of COVID-19 in the Spring of 
2020 shut down our research for two years. As COVID-19 receded in the second 
half of 2021, study team members across all sites conducted numerous pilot 
sessions, often done face-to-face, with both experienced and resident physicians. 
We used specially designed case vignettes and different versions of a survey tool 
with the goal of eliciting accurate decision descriptions expressed as curves. In 
parallel, we invested in initial software development to be able to generate 
clinician’s decision curves from participant responses in near real-time so they 
could be available for the subsequent cognitive interviews. However, in these face-
to-face pilot interactions, the anesthesiologist participants appeared consistently to 
report clinical practices and management decisions that they thought would be 
expected of them (i.e., “the oral boards answer”) rather than what they might 
actually do in their clinical work. Further, despite repeated restructuring of the 
clinical vignettes and questions necessary to create decision curves, we determined 
that it was not possible to obtain unbiased curves within a reasonable time-period to 
conduct the study as originally designed.  

      MISP Template: Protocol 
6 
Final Version: 8/13/2010 
 
As described in the (new) last paragraph of the Section 2.2, we have identified 
an alternative method to decision elicitation using a survey tool that includes 
confidence self-assessment for each answer. There is evidence that this approach 
can generate data that may be closer to what community anesthesiologists would 
actually do in clinical practice [4,20, REF]. We propose to adopt this survey 
method with the modification of contextualizing the questions by preceding them 
with clinical vignettes that will provide the foundation for decision exploration 
during the subsequent cognitive interview (see Section #2.3, Study Design). 
 
Further, we believe it is no longer feasible to conduct this study face-to-face and 
now propose to conduct the revised study interactions virtually. The original study 
design was going to recruit participants via ASA MOCA courses. However, 
between modifications in the MOCA requirements and the persistence of the 
COVID pandemic, MOCA course offerings and enrollments are no longer 
sufficient to achieve the study aims. The proposed clinical vignette-based survey 
followed by pragmatic cognitive interviews will be feasible within current 
constraints and allow us to address the original study objectives.  
2.3 Study Design 
 
 
This will be a prospective observational simulation study of 36 experienced 
community anesthesiologists using clinical vignettes and cognitive interviews to 
better understand the decision-making approaches and preferences of practicing 
community anesthesiologists regarding the management of neuromuscular 
blockade (NMB) at the end of general endotracheal anesthesia cases.  
 
 As described in more detail below, an anonymous survey will be distributed 
electronically to a national sample of anesthesiologists responding to a request to 
participate. We will strive to obtain survey responses from at least 150 experienced 
practitioners. All participants will receive the same survey which will include 
essential demographic questions, clinical vignette-based questions about 
neuromuscular blockage management and questions about their confidence in their 
answers. At the end of the survey, participants will be asked to indicate their 
interest in participating in a follow-up cognitive interview (CI). Thirty-six 
participants will then be randomly chosen to participate in the CI which will be 
conducted remotely by an experienced cognitive interviewer. These volunteer 
participants’ responses will be made available to the CIs without revealing 
participant identity (e.g., a randomly generated code number instead of participant 
name will be used and none of the collected demographic data will be available to 
the CIs). Transcripts of the cognitive interviews will be coded by investigators 
blinded to participant identify. 
24 Study 
Procedures 
2.4.1    Study Participants 
 
Study participants will be a national sample of board-certified 
anesthesiologists who practice in community settings. We will recruit until 150 
eligible participants have completed the entire survey and 36 participants have 
agreed to be interviewed. 
 
Inclusion criteria will be community anesthesiologists who are at least 0.5 
FTE clinical and at least 50% of their cases involve general anesthesia. 

      MISP Template: Protocol 
7 
Final Version: 8/13/2010 
 
Exclusion criteria will be those anesthesiologists who are less than two years 
post-residency training, anesthesiologists who are in an academic practice, those 
who spend less than 50% of their work week personally performing or 
supervising intraoperative anesthesia cases, those with practices that do not 
include at least 50% general anesthesia, and anesthesiologists who do not 
routinely make decisions about whether or not to reverse NMB during 
intraoperative anesthesia cases. Thus, for example, an anesthesiologist who has 
exclusively or largely a chronic pain practice would be excluded from this study. 
2.4.2   Recruitment Procedures  
 
We will recruit and consent participants online via various methods and 
through our contacts with subspecialty societies and specific community 
practice groups. An email will be sent to interested participants with a link to 
the survey. The survey will start with an IRB-approved e-consent process. 
Participants will be informed that all data will be anonymous or confidential. 
This will be followed by basic demographic and general clinical practice 
questions (Note that this survey has already been developed). Those participants 
who meet our inclusion criteria (see next paragraph) will be considered ‘eligible’ 
for participation while those who are excluded will be thanked and not 
continue with the study. 
 
Eligible participants and will be presented with the clinical vignette-based 
decision questions. The entire survey process is expected to take less than 30 
minutes. At the end of the survey, eligible participants will be asked if they are 
interested in enrolling in the second phase of the study about which they will 
be informed and then asked to sign an additional consent form. Their contact 
information will be obtained. We will recruit 36 consenting participants for the 
on-line cognitive interviews. 
 
To minimize the risk of bias, we will inform potential participants that this 
is a study of decision making without disclosing the study’s specific objectives or 
hypotheses. Further, we will also conceal the name of the Grantor. At the end of 
each study, we will disclose all of this information to the participant. We have 
experience getting IRB approval to do this in previous studies of clinician 
behavior and decision-making. 
2.4.3   Participant Demographic and Practice Survey 
 
Each participant will complete an already designed and tested web-based 
demographic survey that includes questions about their prior clinical training, prior 
clinical experience, and current clinical practice attributes. This is a modified 
version of the demographic survey we developed and used in the ongoing AHRQ-
funded IMPACTS trial which was based on the completed and published AHRQ-
funded Simulation-Assessment Research Group (SARG) Maintenance of 
Certification in Anesthesia (MOCA) project.38 Our goal will be to collect enough 

      MISP Template: Protocol 
8 
Final Version: 8/13/2010 
information (Table 1) about each participant’s prior training and experience as well 
as about the nature of their current clinical practice (i.e., types of cases, volume, 
solo vs. group practice, etc.) to discern whether there are meaningful differences in 
NMB reversal decision making based on the distribution of these clinician-
descriptive variables (Hypothesis 2).  
 
 Self-selection bias threatens the generalizability of outcomes in all human 
studies, since one cannot force randomly chosen individuals to participate.  
Nonetheless, there are techniques to optimize the heterogeneity of study 
participants. With only 36 study participants in the cognitive interview component 
of the study, we will not be able to stratify on more than a few key variables (e.g., 
small vs. large practice size, hospital vs. ambulatory practice, age cohorts, 
complexity of cases, etc.). As can be seen by these examples, there is likely to be 
appreciable correlation among some of these variables (e.g., those in ambulatory 
practices might be expected to do less complex cases than those working in tertiary 
care facilities). We will be able to explore such potential overlaps in advance using 
demographic data collected previously in the SARG MOCA study38 and from more 
extensive demographic data obtained in the SARG IMPACTS study. From this, we 
will choose the best ‘strata’ on which to recruit participants to achieve better 
heterogeneity. We will use no more than 3 strata since it will be necessary to 
consider all strata combinations.  
Table 1. Participant Variables to be Obtained in the Survey. 
Data Element 
Sub-Element Description 
Expected Categories 
or Data Elements 
Study site 
NA 
1-3 
Geographic 
location 
Pacific - CA, OR, WA, AK, HI 
West - AZ, CO, ID, MT, NM, NV, UT, WY 
Midwest - KS, IA, MN, MO, ND, NE, SD 
West South Central - AR, LA, OK, TX 
East North Central - IL, IN, MI, OH, WI 
East South Central - AL, KY, MS, TN 
South Atlantic - DE, FL, GA, MD, NC, SC, VA, WV 
Middle Atlantic - NY, NJ, PA 
New England - CT, MA, ME, NH, RI, VT 
Participant ID 
 
00000 
Participant 
demographics 
Age 
00.0 Years 
Declared gender 
Male or Female 
Year graduated from residency 
0000 
Years in practice 
00.0 
Participant 
training 
Fellowship training 
Y/N and type 
Additional clinical training 
Y/N and type 
Subspecialty certification 
Y/N and type 
Primary ABA certification date 
00/0000 
Last ABA recertification date 
00/0000 

      MISP Template: Protocol 
9 
Final Version: 8/13/2010 
Date of last exposure to didactics about 
neuromuscular blockade reversal 
00/0000 
Practice 
attributes 
Average work week in hours 
000.0 hrs 
% of week providing anesthesia care 
00.0 % 
% of time doing cases alone (solo or 
non-supervision) 
00.0 % 
% of cases done in a standalone 
surgicenter or office practice? 
00.0 % 
Focus of practice 
Primary types of cases 
performed* 
% of all cases in which NMB used 
00.0 % 
 
Sugammadex use 
Never to Always 
 
Quantitative monitoring use 
Never to Always 
 *  Categories of practice focus are: Acute pain/regional, ambulatory, cardiac, 
chronic pain, critical care, general (colorectal, ENT, orthopedics, plastics, 
urology, etc.), hepatic, obstetrics & gynecology, neurosurgical, pediatric, trauma, 
vascular, and other. Those who do mostly acute pain/regional or chronic pain 
will be excluded from this study. 
 
Survey design and implementation. We have appreciable experience designing 
and successfully administering demographic and related survey tools for anesthesia 
professionals.38,45-48 Further, we have captured many of the data elements in Table 
1 in a prior simulation-based study of board-certified anesthesiologists.38 However, 
during the first phase of this study (i.e., first 6 months), we intend to pilot test and 
refine the survey tool. We present a draft survey during early stage interviews (see 
below) and inquire as to whether any elements are missing or potentially 
misleading. The final survey tool will be deployed via a web portal to maximize 
data capture efficiency. 
2.5.4   Clinical Variables 
 
Based on survey studies,19-21 a number of clinical variables have been 
implicated as affecting anesthesia professionals’ decision as to whether or not to 
reverse NMB. Table 2 indicates the most commonly cited clinical variables. 
Table 2. Preliminary list of clinical variables affecting the reversal decision. 
Clinical variables 
Units, method, or example(s) 
Time since last dose of NMB 
Minutes 
Fade of the TOF 
Visual observation and/or tactile assessment 
Type of NMB drug used 
Rocuronium, vecuronium, atracurium 
Total dose of NMB administered 
Milligrams 
Respiratory parameters 
Tidal volume, respiratory rate 
Breathing pattern 
Irregular, shallow, “looks weak” 
Ability to maintain head lift 
>5 sec or >10 sec 

      MISP Template: Protocol 
10 
Final Version: 8/13/2010 
Perceived muscle strength 
Hand grip, other 
2.4.5   Clinical Vignettes 
 
Each study participant will read and respond to written clinical vignette 
scenarios in which they must answer 1-3 questions about the management of 
patients who have an existing level of neuromuscular blockade. Each answer will 
be followed by a self-assessment of their confidence in their answer. 
 
The vignettes will be relatively brief, realistic, and sufficiently detailed to 
provide appropriate context for the questions (and the subsequent cognitive 
interviews). A sample vignette is: 
“Mr. Jones a 45-year old 85 kg (BMI 30) male smoker, otherwise healthy 
besides chronic low back pain, is undergoing a two-level lumbar laminectomy 
in the prone position under general endotracheal anesthesia. Induction 4 hours 
(240 minutes) ago was with propofol 160 mg, rocuronium 40 mg, lidocaine 100 
mg, and fentanyl 250 mcg. Intubation was easy. Mr. Jones has been maintained 
with isoflurane in air. Blood loss has been 200 ml and adequate crystalloid to 
maintain euvolemia. He has been hemodynamically stable with good 
oxygenation and ventilation. The surgeon is closing. The patient has started 
breathing spontaneously and is on pressure support ventilation with an end-tidal 
CO2 of 34 mmHg. One additional dose of rocuronium 20 mg was administered 
45 minutes ago. A standard qualitative twitch monitor on the left ulnar nerve 
suggests a TOF of 4.”  
 
Vignette design and implementation. Drs. Weinger and Slagle have experience 
doing vignette-based studies.36,50 During Phase 1, we also will design 6 medically 
similar vignettes (A to F) and pilot test them with academic anesthesiologists and 
residents to assure their clinical equivalence with regard to NMB reversal-related 
factors. To design these vignettes, we will start with a more detailed and focused 
literature review. We will then interview experts in NMB, and also conduct brief 
individual interviews of senior residents, faculty, and CRNAs at each of our study 
sites to identify those variables most commonly mentioned as being important to 
their NMB reversal decisions (these interviews will also help us to refine the 
cognitive interview guide and approach). The goal is to design vignettes that create 
equipoise between decision options to maximize sensitivity. We have experience 
using this method to design vignettes.50,51 
During the study period, we will ask participants to focus on what they actually 
do in their usual clinical practice. We will structure the facilitator guide to 
avoid any avoidable bias or priming. 
 
2.4.6   Cognitive Interviews 
 
Within two days of completing the survey, eligible participants who agree to be 
interviewed will be contacted to schedule a cognitive interview.  which will 
preferably take place within the next two weeks. The ~30-minute cognitive 
interview will elicit their thought processes including, goals and judgments and 
their attitudes regarding their decisions. We will use a cognitive task analysis 

      MISP Template: Protocol 
11 
Final Version: 8/13/2010 
(CTA) method that focuses on key decisions,52 a technique we used in prior 
studies53,54 and in the IMPACTS study. We will focus on the factors than appear to 
most influence the decisions of the participants with respect to the clinical vignettes 
and their answers to the questions. This method, based on an interview guide, 
consists of specific cognitive probes such as: “What were the factors you 
considered when you made the decision [not] to reverse this patient with these 
clinical attributes?” “Does this case remind you of a prior real case in which you 
[did or did not] choose to reverse the patient?” “What else might you do to make 
the decision?” Additionally, when the participant mentions a specific attribute that 
might have been a factor in their decision making, this will be explored further. In 
this regard, the interview will allow us to understand what factors are important 
considerations that experienced anesthesiologists use when deciding to use a 
reversal agent. 
 
The interviews will be audiotaped and transcribed (with participants’ identity 
redacted from all transcripts) for analysis (see below). 
 
Study Procedure Pilot Testing. While we have done appreciable pilot testing of 
aspects of the proposed methods, we will still pilot test the entire revised procedure 
with anesthesiologists (residents or attendings). This will allow final refinement of 
the procedure as well as of the facilitator/interviewer guide. Before enrolling the 
first study participant, each already trained cognitive interviewer will pilot test the 
entire study procedure with a senior resident. This interview will be videotaped and 
sent to the Coordinating Center for review and ‘sign off’ by Anders and Weinger 
before allowing that interviewer to participate in study cases. 
2.5 Statistical 
Analysis and 
Sample Size 
Justification  
 
There will be two separate analyses – a quantitative analysis of the survey 
answers across participants and a qualitative analysis of the cognitive 
interviews. The quantitative analyses will be designed and conducted by the 
CRISS biostatistician, Matthew Shotwell, PhD while the qualitative analysis will 
be overseen by Shilo Anders, PhD a human factors engineer. The PI will be 
responsible for oversight of all analysis, interpretation, and dissemination 
activities. In these analyses, we will: 
a. Capture individual participants’ responses to the survey question 
b. Conduct regression analyses to identify participant attributes that associate 
with specific decision patterns to reverse or not reverse the patient.  
c. Use robust qualitative methods (based on grounded theory) to iteratively 
code the interviews to discern the factors that influence the reversal 
decision.   
2.5.1  Coding of survey responses 
Survey responses will be used to construct an overall evidence-based practice 
score for each participant that represents the degree to which participants 
adhere to evidence-based expectations for reversal of NMB in their daily 
practice. The overall score is the count of survey responses that are consistent 

      MISP Template: Protocol 
12 
Final Version: 8/13/2010 
with evidence-based expectations and the respondent indicated greater than or 
equal to 70% confidence in their response. 
2.5.2   Numeric and Graphical Descriptions of the Survey Results 
Participant demographic and clinical practice characteristics (Table 1) will be 
summarized in a tabular fashion. These summaries will be stratified by dividing 
participants into thirds according to the tertiles of the overall evidence-based 
practice score derived from survey responses. Quantitative factors will be 
summarized using the median and interquartile range (IQR), and categorical 
factors will be summarized using percentages. In order to quantify the 
unadjusted associations between participant factors and clinical decision 
variables, tests for homogeneity in participant factors across strata will be 
implemented using the Kruskal-Wallis test or Chi-square test, as appropriate.  
2.5.3   Statistical Analyses  
 
A multivariable regression analysis will be implemented, using a 
proportional odds logistic regression, to evaluate the adjusted associations 
between  participant demographic and clinical practice characteristics versus 
overall evidence-based practice score scores. A relaxed-lasso method will be 
used to select which participant characteristics are used in the regression 
analysis.55 This method selects factors using a lasso method (where the lasso 
penalty is selected using a cross-validation to avoid model overfitting). Clinical 
decision variables are then regressed onto the selected participant factors 
without lasso penalty. The un-penalized regression analysis, and the associated 
Wald-type 95% confidence intervals and tests will form the basis for statistical 
inferences.  
 
In general, the lasso variable selection technique puts all variables on the 
same footing, in the sense that any and all might be excluded from the 
regression model. However, the following characteristics will be  will be not be 
subject to exclusion by the lasso method: routine use of sugammadex in clinical 
practice, sex, and years of practice. 
 
For all regression analyses, the relevant graphical and quantitative regression 
diagnostics will be examined.56 Alternative model structures or data 
transformations will be considered where necessary to achieve acceptable 
diagnostics. The effects of independent variables and covariates will be modeled 
in an additive fashion (i.e., without interactions), and quantitative covariates 
will be modeled using splines to allow for possible nonlinear associations. No 
adjustment will be made to control a familywise type-I error rate.57 All 
statistical tests will be implemented using a fixed type-I error rate of 5%.  
2.5.4   Sample Size  

      MISP Template: Protocol 
13 
Final Version: 8/13/2010 
 
The analyses described above are largely exploratory or qualitative. We use 
the Wilcoxon test, which is closely related to proportional odds logistic 
regression, to provide a gauge of statistical precision. For a type-I error of 5%, 
there is approximately 80% power to detect a concordance probability of 76% 
when there are 17 participants in each of two groups (36 total). The 
concordance probability is the probability that a randomly chosen participant in 
one group will have a greater outcome (i.e., overall evidence-based practice 
score) than a randomly chosen participant in the other group. Thus, the 
proposed sample size is sufficient to identify associations between participant 
characteristics and evidence-based practice scores, if the strength of association 
is similar to that represented by a concordance probability of 76%. In addition, 
these results will also provide much-needed preliminary data for a larger study to 
understand this type of clinical decision-making more fully. 
2.7.5   Qualitative Analysis 
 
Our approach will start with iterative coding and analysis of the narrative 
cognitive interview data, without knowledge of participant identity or 
attributes. We will initially code interview transcripts using a grounded theory 
approach.58,59 Throughout the analysis, coding will remain flexible as different 
factors emerge as relevant to the decision making process. The approach will 
identify all articulated decision factors and themes, and their corresponding 
relationships, because it is grounded in the empirical data and unconstrained by 
prior cognitive theories that may not reflect the current dynamic acute care 
environment.60 We will use a qualitative data analysis tool such as NVivo™61 to 
facilitate coding. 
 
The coding process will involve line-by-line analysis of 6 participant’s 
interview transcripts. Two team members will identify short, coherent excerpts 
(phrases or sentences) and label them with one or more codes (e.g. type of 
factors that participants mentioned). The analysts will then meet and reach 
consensus on an initial code book. We will also capture ways that factors cluster 
together to create meaningful decision model themes for diagnosis or 
treatment.62 We will then code the next 6 transcripts and refine the code book. 
The analysts will then meet with the entire research team to present and review 
their results to obtain feedback which will guide the final analysis. The next 6 
transcripts will be used to assess coding reliability which will be deemed 
acceptable when inter-coder reliability is good (Kappa ≥0.7).63,64 If unacceptable, 
then coding will continue iteratively until reliability is adequate. Coders will go 
back and recode as necessary the original cases. Once all transcripts are coded, 
the comprehensive set of factors and themes will be further explored through 
frequency analysis and examination of concurrent factors or clusters of factors 
around the various decision thresholds. If appropriate, an exploratory regression 

      MISP Template: Protocol 
14 
Final Version: 8/13/2010 
analysis could be performed between decision patterns and participant 
variables. 
2.6 Adverse 
Experience 
Reporting 
Not applicable. 
2.7 References 
Citations 
1. Brull SJ, Kopman AF: Current status of neuromuscular reversal and 
monitoring: Challenges and opportunities. Anesthesiology 2017; 126: 173-190 
2. Brull SJ, Prielipp RC: Neuromuscular monitoring and the cost of antagonism: 
when will we learn? Anaesthesia 2017; 72: 1557-1558 
3. Murphy GS: Neuromuscular monitoring in the perioperative period. Anesth 
Analg 2017 
4. Naguib M, Brull SJ, Kopman AF, Hunter JM, Fulesdi B, Arkes HR, Elstein A, 
Todd MM, Johnson KB: Consensus statement on perioperative use of 
neuromuscular monitoring. Anesth Analg 2017 
5. Fortier L, McKeen D, Turner K, de Médicis É, Warriner B, Jones P, Chaput A, 
Poulio TJ, Galarneau A: The RECITE Study: A Canadian prospective, 
multicenter study of the incidence and severity of residual neuromuscular 
blockade. Anesth Analg 2015; 121: 366-72 
6. Naguib M, Kopman AF, Ensor J: Neuromuscular blockade and postoperative 
residual curarisation: a meta-analysis. Brit J Anaesth 2007; 98: 302-16 
7. Stewart P, Liang S, Li Q, Huang M, Bilgin A, Kim D, Phillips S: The impact of 
residual neuromuscular blockade, oversedation, and hypothermia on adverse 
respiratory events in a postanesthetic care unit: A prospective study of 
prevalence, predictors, and outcomes. Anesth Analg 2016; 123: 859-68 
8. Naguib M, Brull SJ, Johnson KB: Conceptual and technical insights into the 
basis of neuromuscular monitoring. Anaesthesia 2017; 72 Suppl 1: 16-37 
9. Bronsert M, Henderson W, Monk T, Richman J, Nguyen J, Sum-Ping J, 
Mangione M, Higley B, Hammermeister K: Intermediate-acting 
nondepolarizing neuromuscular blocking agents and risk of postoperative 30-
day morbidity and mortality, and long-term survival. Anesth Analg 2017; 124: 
1476-83 
10. McLean D, Diaz-Gil D, Farhan H, Ladha K, Kurth T, Eikermann M: Dose-
dependent association between intermediate-acting neuromuscular-blocking 
agents and postoperative respiratory complications. Anesthesiology 2015; 122: 
1201-13 
11. Brull SJ, Naguib M: How to catch unicorns (and other fairytales). 
Anesthesiology 2017 
12. Murphy GS, Szokol JW, Avram MJ, Greenberg SB, Shear TD, Deshur MA, 
Benson J, Newmark RL, Maher CE: Neostigmine administration after 
spontaneous recovery to a train-of-four ratio of 0.9 to 1.0: A randomized 
controlled trial of the effect on neuromuscular and clinical recovery. 
Anesthesiology 2017 

      MISP Template: Protocol 
15 
Final Version: 8/13/2010 
13. Pedersen T, Viby-Mogensen J, Bang U, Olsen N, Jensen E, Engboek J: Does 
perioperative tactile evaluation of the train-of-four response influence the 
frequency of postoperative residual neuromuscular blockade? Anesthesiology 
1990; 73: 835-9 
14. Viby-Mogensen J, Jensen N, Engbaek J, Ording H, Skovgaard L, Chraemmer-
Jørgensen B: Tactile and visual evaluation of the response to train-of-four nerve 
stimulation. Anesthesiology 1985; 63: 440-3 
15. Naguib M, Dexter F, Brull SJ: Neuromuscular monitoring as the art of 
probability. Anesth Analg 2017; 124: 1400-1402 
16. Batistaki C, Tentes P, Deligiannidi P, Karakosta A, Florou P, Kostopanagiotou 
G: Residual neuromuscular blockade in a real life clinical setting: correlation 
with sugammadex or neostigmine administration. Minerva Anestesiol 2016; 82: 
550-8 
17. Ledowski T, Hillyard S, O'Dea B, Archer R, Vilas-Boas F, Kyle B: 
Introduction of sugammadex as standard reversal agent: Impact on the 
incidence of residual neuromuscular blockade and postoperative patient 
outcome. Indian J Anaesth. 2013; 57: 46-51 
18. Kotake Y, Ochiai R, Suzuki T, Ogawa S, Takagi S, Ozaki M, Nakatsuka I, 
Takeda J: Reversal with sugammadex in the absence of monitoring did not 
preclude residual neuromuscular block. Anesth Analg.; 117: 345-51 
19. Söderström C, Eskildsen K, Gätke M, Staehr-Rye A: Objective neuromuscular 
monitoring of neuromuscular blockade in Denmark: an online-based survey of 
current practice. Acta Anaesthesiol Scand 2017; 61: 619-26 
20. Naguib M, Kopman A, Lien C, Hunter J, Lopez A, Brull S: A survey of current 
management of neuromuscular block in the United States and Europe. Anesth 
Analg 2010; 111: 110-9 
21. Videira R, Vieira J: What rules of thumb do clinicians use to decide whether to 
antagonize nondepolarizing neuromuscular blocking drugs? Anesth Analg 2011; 
113: 1192-6 
22. Chen CL, Lin GA, Bardach NS, Clay TH, Boscardin WJ, Gelb AW, Maze M, 
Gropper MA, Dudley RA: Preoperative Medical Testing in Medicare Patients 
Undergoing Cataract Surgery. N Engl J Med 2015; 372: 1530-38 
23. Kahneman D: Thinking Fast and Slow. New York, NY, Farrar, Straus & 
Giroux, 2011 
24. Croskerry P: The theory and practice of clinical decision-making. Can J 
Anaesth 2005; 52: R1-8 
25. Eddy DM: Clinical Decision Making: From Theory to Practice. Boston, MA, 
Jones and Barlett, 1996 
26. Moore DE, Jr., Green JS, Gallis HA: Achieving desired results and improved 
outcomes: Integrating planning and assessment throughout learning activities. J 
Contin Educ Health Prof 2009; 29: 1-15 
27. Salas E, Klein G: Linking Expertise and Naturalistic Decision Making. 
Mahwah, NJ, Lawrence Erlbaum Assoc, 2001 
28. Klein G, Calderwood R, A C-C: Rapid decision making on the fire ground: The 
original study plus a postscript. J Cogn Engin Decision Mak 2010; 4: 186-209 
29. Croskerry P: The importance of cognitive errors in diagnosis and strategies to 
minimize them. Acad Med 2003; 78: 775-80 

      MISP Template: Protocol 
16 
Final Version: 8/13/2010 
30. Croskerry P, Nimmo G: Better clinical decision making and reducing 
diagnostic error. J R Coll Physicians Edinb 2011; 41: 155-62 
31. Pearson S, Goldman L, Orav E, Guadagnoli E, Garcia T, Johnson P, Lee T: 
Triage decisions for emergency department patients with chest pain: Do 
physicians' risk attitudes make the difference? J Gen Intern Med 1995; 10: 557-
64 
32. Burman RA, Zakariassen E, Hunskaar S: Chest pain out-of-hours – an 
interview study of primary care physicians’ diagnostic approach, tolerance of 
risk and attitudes to hospital admission. BMC Fam Pract 2014; 15: 1127 
33. Klein GA, Calderwood R, MacGregor D: Critical decision method for eliciting 
knowledge. IEEE Trans Sys Man Cyber 1989; 19: 462-476 
34. Howard SK, Gaba DM, Smith BE, Weinger MB, Herndon CN, Keshavacharya 
S, Rosekind MR: Simulation study of rested versus sleep-deprived 
anesthesiologists. Anesthesiology 2003; 98: 1345-1355 
35. Syroid ND, Agutter J, Drews FA, Westenskow DR, Albert RW, Bermudez JC, 
Strayer DL, Prenzel H, Loeb RG, Weinger MB: Development and evaluation of 
a graphical anesthesia drug display. Anesthesiology 2002; 96: 565-75 
36. Connor O, Weinger MB, Cooke NJ, Slagle J: Using psychological scaling 
techniques to assess clinical expertise in anesthesiology, Proc Human Factors 
Ergon Soc, SAGE Publications Sage CA: Los Angeles, CA, 2004, pp 1746-
1750 
37. Weinger MB, Slagle J: Human factors research in anesthesia patient safety: 
Techniques to elucidate factors affecting clinical task performance and decision 
making. J Am Med Inform Assoc 2002; 9: S58-63 
38. Weinger MB, Banerjee A, Burden AR, McIvor WR, Boulet J, Cooper JB, 
Steadman R, Shotwell MS, Slagle JM, DeMaria SJ, Torsher L, Sinz E, Levine 
AI, Rask J, Davis F, Park C, Gaba DM: Simulation-based assessment of the 
management of critical events by board-certified anesthesiologists. 
Anesthesiology 2017; 127: 475-89 
39. Agutter J, Drews F, Syroid N, Westenskow D, Albert R, Strayer D, Bermudez 
J, Weinger MB: Evaluation of a graphic cardiovascular display in a high 
fidelity simulator. Anesth Analg 2003; 97: 1403-13 
40. Anders S, Miller A, Joseph P, Fortenberry T, Woods M, Booker R, Slaughter J, 
Weinger M, France D: Blood product positive patient identification: 
comparative simulation-based usability test of two commercial products. 
Transfusion 2011; 51: 2311-8 
41. Knab JH, Wallace MS, Wagner RL, Tsoukatos J, Weinger MB: The use of a 
computer-based decision support system facilitates primary care physicians' 
management of chronic pain. Anesth Analg 2001; 93: 712-20 
42. Karsh B-T, Weinger MB, Abbott P, Wears RL: Health Information 
Technology: Fallacies and sober realities. JAMIA 2010; 17: 617-623 
43. Anders S, Albert R, Miller A, Weinger M, Doig A, Behrens M, Agutter J: 
Evaluation of an integrated graphical display to promote acute change detection 
in ICU patients. Int J Med Inform. 2012; 81: 842-51 
44. Rosen AK, Gaba DM, Meterko M, Shokeen P, Singer S, Zhao S, Labonte A, 
Falwell A: Recruitment of hospitals for a safety climated study: Facilitators and 
barriers. Jt Comm J Qual Safe 2008; 34: 275-84 

      MISP Template: Protocol 
17 
Final Version: 8/13/2010 
45. Cao CG, Weinger MB, Slagle J, Zhou C, Ou J, Gillin S, Sheh B, Mazzei W: 
Differences in day and night shift clinical performance in anesthesiology. Hum 
Factors 2008; 50: 276-90 
46. Oken A, Rasmussen MD, Slagle JM, Jain S, Kuykendall T, Ordonez N, 
Weinger MB: A facilitated survey instrument captures significantly more 
anesthesia events than does traditional voluntary event reporting. 
Anesthesiology 2007; 107: 909-22 
47. Weinger MB, Herndon OW, Zornow MH, Paulus MP, Gaba DM, Dallen LT: 
An objective methodology for task analysis and workload assessment in 
anesthesia providers. Anesthesiology 1994; 80: 77-92 
48. Weinger MB, Slagle JM, Kuntz AH, Schildcrout JS, Banerjee A, Mercaldo ND, 
Bills JL, Wallston KA, Speroff T, Patterson ES, France DJ: A multimodal 
intervention improves post-anesthesia care unit handovers. Anesth Analg 2015; 
121: 957-71 
49. Weinger MB: The pharmacology of simulation: A conceptual framework to 
inform progress in simulation research. Simul Healthc 2010; 5: 8-15 
50. Wallston K, Slagle J, Speroff T, Nwosu S, Crimin K, Feurer I, Boettcher B, 
Weinger M: Operating room clinicians' ratings of workload: a vignette 
simulation study. J Patient Saf 2014; 10: 95-100 
51. Minnick A, Donaghey B, Slagle J, Weinger M: Operating room team members' 
views of workload, case difficulty, and nonroutine events. J Healthc Qual 
2012; 34: 16-24 
52. Crandall B, Klein G, Hoffman RR: Working Minds: A Practitioner's Guide to 
Cognitive Task Analysis. Cambridge, MA, MIT Press, 2006 
53. Militello LG, Patterson ES, Saleem JJ, Anders S, Asch S: Supporting 
macrocognition in health care: Improving the usefulness of clinical reminders, 
Naturalistic Decision Making and Macrocognition. Edited by Schraagen MC, 
Militello LG, Omerod T, Lipshitz R. Aldershot, UK, Ashgate Publishing 
Limited, 2008 
54. Militello LG, Anders S, Downs SM, Diiulio J, Danielson EC, Hurley RW, 
Harle CA: Using sensemaking to better understand chronic pain management, 
Proceedings of the 13th Conference on Naturalistic Decision Making. Bath, 
England, UK, University of Bath, 2017 
55. Meinshausen N: Relaxed lasso. Comput Stat Data Anal 2007; 52: 374-93 
56. Cohen J, Cohen P, West S, Aiken L: Applied Multiple Regression/Correlation 
Analysis for the Behavioral Sciences, 3rd Edition. Mahway, NJ, Lawrence 
Erlbaum Associates, 2003 
57. Cook RJ, Farewell VT: Multiplicity considerations in the design and analysis of 
clinical trials. J Royal Stat Soc. Series A (Statistics in Society) 1996: 93-110 
58. Glaser B, Strauss A: Grounded theory: The discovery of grounded theory. 
Sociology 1967; 12: 27-49 
59. Strauss A, Corbin J: Grounded theory methodology, Handbook of Qualitative 
Research. Edited by Denzin NK, Lincoln YS. Thousand Oaks, CA, SAGE 
Publications, 1994, pp 273-85 
60. Bernard HR, Ryan GW: Analyzing Qualitative Data: Systematic Approaches. 
Thousand Oaks, CA, SAGE Publications, 2010 

      MISP Template: Protocol 
18 
Final Version: 8/13/2010 
 
61. Saravia-Pinilla M, Daza-Beltrán C, García-Acosta G: Appl Ergon. A 
comprehensive approach to environmental and human factors into 
product/service design and development. A review from an ergoecological 
perspective. Appl Ergonom 2016; 57: 62-71 
62. Ryan GW, Bernard HR: Techniques to identify themes. Field Methods 2003; 
15: 85-109 
63. Altman D: Practical Statistics for Medical Research. London, UK, Chapman & 
Hall 1991 
64. Cohen J: A coefficient of agreement for nominal scales. Educ Psychol Measure 
1960; 20: 37-46