Reducing Potentially Inappropriate Medication Prescribing for Older Patients: Enhancing Quality of Provider Practices for Older Adults in the Emergency Department (EQUIPPED)
Phase: NA
Conditions: Aging
Sponsor: VA Office of Research and Development

 
 
 
  
Reducing Potentially Inappropriate 
Medication Prescribing for Older Patients: 
Enhancing Quality of Provider Practices 
for Older Adults in the Emergency 
Department (EQUIPPED) 
Protocol                                         NCT: 04004936 
JUNE 25, 2021 

EQUIPPED Implementation Evaluation, Protocol, v4, 06-25-2021 
1 of 25 
 
Enhancing Quality of Prescribing Practices for Older Adults Discharged from the 
Emergency Department (EQUIPPED) 
 
VA HSR&D Grant Multiple Principal Investigators 
E. Camille Vaughan, MD, MS (corresponding, Atlanta VAMC) and  
George L. Jackson, PhD, MHA (Durham VAMC) 
Funding agency:  VA Health Services Research & Development (IIR HX00257-01A1) 
 
Atlanta VA Health Care System Protocol – Version 1 Submission Date – August 6, 2018 
  -- Version 2 Submission Date- August 8, 2019 
--Version 3 Submission Date – March 18, 2021 
--Version 4 Submission Date – June 25, 2021 
 
Atlanta VA Health Care System Principal Investigator 
E. Camille Vaughan, MD, MS 
Atlanta Site Director 
Birmingham/Atlanta GRECC 
 
Site Summary Roles 
 
Sites Collecting and Analyzing Identifiable Data 
Atlanta, Durham 
 
Sites with access to CDW data:  
Atlanta, Durham 
 
Additional Sites Receiving Research Dollars 
The Salt Lake City VA will adapt a currently available clinical dashboard for EQUIPPED using 
CDW data.  The dashboard developed for operational purposes.  Dr. Karen Jeans, PhD, CCRN, 
CCIP, and colleagues with the VA Office of Research and Development have indicated that this 
is part of the quality impartment aspects of the EQUIPPED program (please see letter submitted 
with version 1 of this protocol).   
 
Statisticians at the Birmingham VA will analyze deidentified prescribing data (to be certified by 
the Atlanta VA as deidentified) to examine outcomes of the clinical trial.  Additional 
Birmingham staff members will consult on study methods. 
 
ORD Determination that Randomized Sties are Not Engaged in Research 
Dr. Karen Jeans, PhD, CCRN, CCIP, and colleagues with the VA Office of Research and 
Development have indicated that the sites randomized as part of EQUIPPED are not engaged in 
research.  The implementation of EQUIPPED, which was developed for quality improvement 
purposes, by these sites is part of clinical quality impartment.  The sites engaged in research as 
described above are evaluation/studying the process of implement EQUIPPED and the resulting 

EQUIPPED Implementation Evaluation, Protocol, v4, 06-25-2021 
2 of 25 
impact on prescribing. Please see the please see letter from Dr. Karen Jeans submitted with 
version 1 of this protocol 
 
 
Background, Purpose, and Significance 
1.1 Background 
Enhancing Quality of Prescribing Practices for Older Adults Discharged from the Emergency 
Department (EQUIPPED) is a multi-component program to reduce the prescribing of 
potentially inappropriate medications (PIMs) to older adults upon discharge from the 
Emergency Department (ED). It has three core components: 1) provider education, 2) 
Electronic Health Record (EHR)-based clinical decision support (CDS) including pharmacy 
quick order sets to facilitate provider order entry, and 3) provider audit and feedback with 
peer benchmarking. In order to inform a Veterans Affairs (VA) system-wide approach to 
improve prescribing safety for older Veterans, we will conduct a study to determine best 
practices for influencing provider prescribing behavior in order to decrease PIMs prescribed 
for older Veterans at the time of ED discharge. The overall goal of this project is to 
determine which EQUIPPED implementation strategy (active or passive feedback) is most 
effective to reduce prescribing of PIMs for older Veterans discharged from the ED.   
1.2 Purpose/Specific Aims 
This research is being conducted to learn which implementation strategy of EQUIPPED is 
most effective to improve prescribing practices of ED providers toward older Veterans and 
determine the factors influencing implementation of this program to reduce the prescribing of 
PIMs to older adults upon discharge from the ED.  Key questions include: 
- 
Which implementation strategy (passive or active feedback) is most effective to influence 
provider prescribing behavior? 
- 
To what extent is each component of EQUIPPED implemented with fidelity within each 
site and across sites?  
- 
In what way is each component adapted in each site?  
- 
What steps are taken to implement EQUIPPED in each site and what are the similarities 
and differences across sites?  
- 
What facilitates and inhibits implementation?  
- 
How well are EQUIPPED components maintained over time?  
- 
What steps are taken toward sustainability in each site?  
- 
What resources are needed to maintain EQUIPPED over time?  
 
1.3 Preliminary Studies & Significance 
The EQUIPED QI intervention was developed and implemented with funding from the VA 
Office of Geriatrics and Extended Care and Office of Rural Health.  The EQUIPPED QI 
intervention, which provides pilot/preliminary information for the proposed trial, has been 
implemented in 11 VA EDs. Results from 4 of the initial EQUIPPED sites with in-person 
academic detailing demonstrated sustained pre-post improvement (reduction) in PIM prescribing 
rates by nearly 50% at 6 months,1-3 suggesting the possibility of culture change with regard to 

EQUIPPED Implementation Evaluation, Protocol, v4, 06-25-2021 
3 of 25 
provider prescribing behavior. The EQUIPPED QI intervention typically involves in-person 
academic detailing (defined as educational outreach visits to improve clinical decision making4) 
using audit and feedback with peer benchmarking, which is more resource intensive.  
 
The VA already uses both passive feedback (i.e. dashboards to report psychotropic medication 
use in community living center residents) and active feedback (i.e. implementation of a national 
academic detailing pharmacy program); however, there is little guidance on which strategy is 
most effective in the ED. In order to inform the optimal EQUIPPED strategy for improving 
provider prescribing behavior toward older Veterans in ED, we propose a trial comparing 
EQUIPPED with active provider feedback including academic detailing to EQUIPPED with 
passive provider feedback using a dashboard to deliver individual electronic reports.   
 
1.4 EQUIPPED Tools that will be implemented by randomized sites 
 
While each site will be randomized to be with 1) Active Provider Feedback Sites or 2) Passive 
Provider Feedback Sites, the work of implementing the tools will be done as part of clinical care 
quality improvement.  In other words, specific decisions about how to implement the tools and 
with Active Provider Feedback or Passive Provider Feedback versions of EQUIPPED will be 
made by managers, administration, and staff at the facilities.  Likewise, local or VA-wide entities 
may make the decision to rename components of the EQUIPPED program.  Research documents 
may be changed to reflect any related name changes.  Researchers from sites other than the 
randomized sites will evaluate/study the implementation process and results of EQUIPPED at the 
eight randomized sties.  However, the sites, not the researchers, will have quality 
impartment/operational control over the process.  It is possible and expected that the exact 
process of implementing EQUIPPED will vary from site to site based on the clinical, 
organizational, and quality improvement realities of the specific sites. 
 
Dr. Karen Jeans, PhD, CCRN, CCIP, and colleagues with the VA Office of Research and 
Development have indicated that the sites randomized as part of EQUIPPED are not engaged in 
research.  The implementation of EQUIPPED, which was developed for quality improvement 
purposes, by these sites is part of clinical quality impartment.  The sites engaged in research as 
described above are evaluation/studying the process of implement EQUIPPED and the resulting 
impact on prescribing. Please see the please see letter from Dr. Jeans submitted with version 1 of 
this protocol. 
 
1.4.1 The EQUIPPED tools are described in the background of this protocol because the precise 
implementation is being done as QI to enhance usual clinical care at the randomized facilities.  
Information provided in this background section is for reference in reviewing the protocol.  
Information provided in the background section does not represent research procedures. 
 
EQUIPPED with passive provider feedback includes the following components:  1) didactic 
education, 2) computer decision support tools, and 3) provider feedback via monthly individual 
prescribing reports with peer benchmarking provided to ED clinicians.  
 
1) Didactic Education: The component addresses the social cognitive theory (SCT) component 
of behavioral capacity. The EQUIPPED team has developed a presentation which covers the 

EQUIPPED Implementation Evaluation, Protocol, v4, 06-25-2021 
4 of 25 
physiology of aging, the burden of polypharmacy, and PIMs in older adults as defined by the 
2012 and updated for the 2015 American Geriatrics Society Update to the Beers Criteria. 5 This 
presentation can be adapted for shorter presentation times as well. The site champions will offer 
learning sessions to their emergency medicine journal clubs to review the 2015 update to the 
Beers Criteria as published in the Journal of the American Geriatrics Society (may be based on 
future versions if published).5  All educational activities may be monitored for attendance.  
 
2) EHR-based Clinical Decision Support: This component addresses the SCT concept of self-
efficacy by offering providers alternative medications that may be prescribed. Based on previous 
feedback from ED providers, the EQUIPPED team developed templates for outpatient pharmacy 
order sets within CPRS targeting common ED discharge diagnoses among older adults. 
Embedded within the pharmacy quick order sets are dose adjustments for renal impairment, 
point-of-prescribing education regarding medications to avoid, and links to publicly-available, 
evidence-based geriatric content such as a screening tool for depression (Geriatric Depression 
Scale). Providers will be trained by the on-site champion to refer to the quick order sets. When 
providers prescribe medications using the pharmacy quick order sets, the order entry process is 
streamlined with pre-populated fields including: “day’s supply”, “number of pills,” and “method 
for prescription pick-up.”  For our VA QI project, template logic was reviewed by clinical 
pharmacists, ED providers, and the local VA pharmacy and therapeutics committee prior to 
implementation. In addition to expediting the order entry process, the order sets encourage 
providers to choose safer medications.1 The pharmacy order sets simultaneously offer point of 
prescribing education for geriatric conditions and alternatives to PIMs as defined by the Beers 
Criteria. The order sets can be designed to include links at the point of prescribing to additional 
online geriatric educational content. For instance, based on provider feedback, order sets adapted 
by one EQUIPPED site offered links to the World Health Organization analgesic ladder and 
narcotics equivalency tables to simplify pain management.  
 
3) Provider Feedback: As stated above, a key component of EQUIPPED is the provider 
feedback (example in Figure 3) about his or her monthly rate of PIM prescriptions written for 
older adults discharged from the ED. Based upon our experience within the VA, the Beers PIM 
framework was highly useful in the context of the multicomponent EQUIPPED Intervention. 
The usefulness was examined qualitatively among providers participating in the EQUIPPED 
pilot, who indicated that “usefulness” of the order sets was based on both “safety” (i.e., reducing 
risk of adverse events) and increasing “efficiency” of the prescribing process.59 These are 
indications of both enhancing the perceived value of the desired behavior (i.e. SCT concept of 
expectancies) and the ability of the provider to change behavior (i.e. SCT concept of self-
efficacy).  Data related to individual prescribing behavior involves data pulled from VA’s CDW. 
The Atlanta site has successfully piloted pulling ED discharge prescriptions from CDW using a 
Beers lookup SQL tool (developed at the Durham VA), which includes a list of all medications 
considered potentially inappropriate according to the 2015 update to the Beers criteria. For 
provider feedback, we focus on Category 1 medications, which are considered potentially 
inappropriate in all older adults. For the purposes of the ED setting, chronic nonsteroidal anti-
inflammatory drug prescriptions are defined as any dose for 30 days or more. The Beers lookup 
tool facilitates evaluation of prescribing data in order to easily generate monthly provider reports. 
The Atlanta site EQUIPPED team will include a project coordinator who will assist in generating 
the provider reports based on the CDW data pull. Aggregate de-identified prescribing data will 

EQUIPPED Implementation Evaluation, Protocol, v4, 06-25-2021 
5 of 25 
be provided to the statistician based at the Birmingham site of the Birmingham/Atlanta GRECC 
each month for outcome assessment of PIM prescribing. In addition to feedback of individual 
prescribing patterns, providers also receive anonymous peer benchmarking comparing his/her 
individual performance to other ED providers at the same site (i.e., a scorecard of PIM 
prescriptions).6,7 This monthly feedback will come in the form of an emailed link to a clinical 
dashboard showing individual and site level prescribing data for each provider. No patient-
identifiable health information will be included in the provider feedback form; however, site 
champions will have access to individual records if a provider would like to review specific 
cases to determine the appropriateness of a given prescription.  
 
1.4.2 EQUIPPED with active provider feedback includes the same 3 components as those 
included previously with the addition of 1:1 in-person provider feedback using academic 
detailing.  
1) The initial individual feedback session will be provided in-person to all staff providers 
(physician, APRN, PA). Clinical pharmacists or an ED provider may partner with the site 
champion to provide 1:1 feedback depending upon team composition at each site. In-person 
1:1 provider feedback typically requires about 15 minutes and involves the following: a) an 
introduction of the provider’s individual report structure and their baseline data as well as the 
current month’s prescribing pattern; b) review the specific drug classes of PIMs that were 
prescribed during the reporting period; c) comparison of individual prescribing data to the 
cohort of peer providers within local ED; d) review of the top 5 PIM classes to avoid for the 
local ED; e) discussion of evidence-based alternative treatments appropriate for older adults; 
f) a brief demonstration of the EQUIPPED order set appropriate to that provider’s needs 
based on prescribing of PIMs; and g) allowing time for provider question and discussion. 
2) The onsite EQUIPPED champion will be an ongoing local expertise to provide consultation 
on clinical questions resulting from the 1:1 sessions such as consideration of appropriate 
medication alternatives. 
3) To reinforce prescription decisions at the point of the prescription, enhanced medication 
safety alerts/messages will be placed on the screen for prescribing Beers list medications, 
reminding prescribers of potential impacts on older patients.  These “blue line” messages 
were added as part of the EQUIPPED quality improvement work at two facilities, and were 
associated with reductions in PIM prescribing even outside the ED.8  
 
1.4.3 As with many other quality impartment activities, facility leadership will indicate approval 
to implement the program as indicated.  This process and documentation is part of a common 
quality improvement activities that occurs frequently outside of the research setting. 
 
1.5.1 Process for Providing Tools to Sites that will Be Randomized.   
 
The section below describes the assistance that will be provided to sites who are implementing 
the EQUIIPED intervention (Active or Passive) as part of their quality improvement efforts.  
While the processes outlined below will be provided by individuals who are on the research 
team, the process is part of the quality improvement implementation of EQUIPPED.  Members 
of the team have provided similar assistances are part of previous QI development and 
implementation of EQUIPPED.  These processes were reviewed with the VA Office of 
Research and Development, which determent that they did not result in the eight sites 

EQUIPPED Implementation Evaluation, Protocol, v4, 06-25-2021 
6 of 25 
being engaged in research (please see letter submitted with version 1 of this protocol).  
Names are mentioned as examples.  Tasks described below may be done by members of the 
study team with appropriate credentials who are listed on staff study lists and have completed 
required research training.  Additionally, the time lines outlined for the process of 
implementation support provided by the study team may need to be adjusted depending on the 
needs of given facilities.   
 
Stage 1 - Pre-Implementation Preparation (Approximately 6 Months):    
1) Identify and Coach Site Champion. It is critical that all sites have a clinical champion, 
typically an ED physician or geriatrician, who will lead the efforts to promote ongoing 
EQUIPPED education and use of the order set. The site champion will liaison with the 
Atlanta-based implementation team, negotiate time for clinical staff to participate in 
EQUIPPED activities, and help to interface with the local clinical applications coordinator 
and pharmacy service in order to facilitate implementation of the EQUIPPED discharge order 
sets.   
2) Adaptation of the EQUIPPED Order Set: Each site champion will participate in conference 
calls approximately twice a month with the implementation team in Atlanta to guide the 
implementation of EQUIPPED components.  The EQUIPPED discharge medication order 
sets are adaptable depending on local formulary alternatives or based on recommendations 
from the local implementation team. There are 25 EQUIPPED order sets originally 
developed through focus groups with ED providers at the Atlanta VA. These were updated in 
2016 according to the most updated version of the Beers Criteria. Most EQUIPPED sites opt 
to include all of the order sets.  Some sites opt to implement a select number of order sets 
targeting the most commonly prescribed PIMs at the local facility.  This is an example of 
how the EQUIPPED program may be adapted. 
3) Obtain Baseline Data: The Atlanta VA implementation team includes Chris Jasien, who is an 
experienced CDW data analyst.  Ms. Jasien incorporated the Beers lookup tool developed the 
Durham VA to obtain monthly prescribing data from VA’s Corporate Data Warehouse for all 
11 VA EDs currently implementing EQUIPPED. Once the eight implementation sites for the 
HSR&D project are confirmed, Ms. Jasien will obtain baseline prescribing data for the 6 
months prior to implementation to establish initial levels of performance. 
 
Stage 2 – Launch Implementation Models (Approximately 1 Month Prior to Initial 
Measurement of Outcomes):  
The Atlanta team, led by Dr. Vaughan, will finalize a training manual for champions to use to 
deliver provider prescribing education for older Veterans in the ED setting, specifically in their 
local VAMCs. The education materials include didactic information on geriatric prescribing, 
EQUIPPED order set education, AGS pocket cards on Beers drugs, reminder cards, alternative 
prescribing tables, and other tools and training materials needed to train staff at their own 
hospital.  All clinical decision support tools will be accessible in CPRS.  Baseline data will be 
available through a Research project folder within the VA’s VINCI framework on a secure VA 
server. Baseline data from Stage 1 will be shared with all site champions to incorporate in the 
initial provider education session.  Additionally, if a site is randomized to active provider 
feedback, the Atlanta team will coach the site champion (and clinical partner for 1:1 feedback, if 
applicable) in the use of the education materials to train providers in the rationale for using the 
EQUIPPED order sets within the VA EHR. An additional module for the training manual for 

EQUIPPED Implementation Evaluation, Protocol, v4, 06-25-2021 
7 of 25 
active provider feedback sites will review teaching concepts, techniques, and materials for 
providing academic detailing regarding prescribing practices toward older Veterans. Using a 
short 20-minute, PowerPoint presentation appropriate for in-service training, a teleconference 
training session will be conducted to provide a structure for the academic detailing approach 
outlined in section C.3.b. A certification checklist will be used to ensure all champions 
demonstrate competency. 
 
Stage 3 – Continue Provider Feedback and Measurement of Outcomes for 1 Year:  
1) ED providers will receive monthly electronic provider prescribing feedback reports for 
monthly PIM rates, including the number of older Veterans seen in the ED, number of 
prescriptions given, and the number of PIMs (including drug class).     
2) Active Provider Feedback:  In addition to the electronic feedback reports available for the 
remote intervention, the champion (or clinical partner) will provide 1:1 academic detailing 
that includes audit and feedback with peer benchmarking to ED providers and ongoing 
monthly 1:1 feedback to outlier PIM prescribers in the ED at the site.   
 
Research Methods 
 
2.0 Design 
In a parallel cluster randomized trial, we will randomize 8 VA facilities to implement the 
EQUIPPED QI intervention with either passive provider feedback (4 sites) or active provider 
feedback (4 sites). Specifically, it is anticipated that all sites will implement EQUIPPED 
components including: didactic education concerning the Beers Criteria; decision support by 
order sets; and monthly provider prescribing feedback. However, passive provider feedback sites 
will implement monthly provider feedback via an electronic dashboard with audit, feedback and 
peer benchmarking, while active provider feedback sites will implement one-to-one (1:1) in-
person academic detailing from a professional colleague that includes in-person audit, feedback, 
and peer benchmarking and provide on-site expertise. To compare the effectiveness of 
EQUIPPED with in-person academic detailing including proactive feedback (active feedback) 
versus EQUIPPED with passive electronic audit and feedback (passive feedback) intervention, 
we will evaluate the monthly proportion of PIM prescribing as % of individual prescriptions 
(primary outcome) in each arm. It is hypothesized that the decline in PIM rates will be greater in 
the presence of EQUIPPED with active feedback compared to EQUIPPED with passive 
feedback.  
 
After the initial recruitment and randomization, one of the eight sites had to pause order set 
development due to COVID-19 and an off-site detail of one of the core implementation team 
members. After several months of stalled progress with EQUIPPED implementation, the site 
decided to pause EQUIPPED implementation until at least spring 2021. We have recruited an 
additional site to fill the open study spot. We will continue to collect prescribing data from both 
sites and collect implementation evaluation data from the newly recruited facility based on the 
timeline for the award. If the initial facility determines they can move forward in 2021, we 
propose collecting implementation evaluation data as originally planned (would increase number 
of sites to n=9). The data analysis plan for this change is reflected below in section 2. 1. 3. 
 

EQUIPPED Implementation Evaluation, Protocol, v4, 06-25-2021 
8 of 25 
As described in detail in the background section, researchers from sites other than the 
randomized sites will evaluate/study the implementation process and results of EQUIPPED at the 
eight randomized sties.  However, the sites, not the researchers, will have quality 
impartment/operational control over the process.  It is possible and expected that the exact 
process of implementing EQUIPPED will vary from site to site based on the clinical, 
organizational, and quality improvement realities of the specific sites.  Further, local or VA-wide 
entities may make the decision to rename components of the EQUIPPED program.  Research 
documents may be changed to reflect any related name changes.   
 
In order to inform the eventual dissemination strategy, we will also include formative evaluation 
and micro-costing of the two methods of implementing provider feedback as part of EQUIPPED.   
 
We will conduct an in-depth, theory-based formative evaluation of sites, with the goals of 
understanding: 1) factors that may impact organizational readiness to implement EQUIPPED 
(i.e., organizational readiness for change (ORC)) and implementation of the EQUIPPED as 
outlined in the Organizational Theory of Implementation Effectiveness (OTIE); 2) changes in 
these factors over the course of one year; and 3) association between ORC and factors suggested 
by the OTIE to both successful implementation and sustainability over time.  We will use a 
mixed-methods approach to measure ORC and change in related components over time.  The 
ultimate goal is to inform the process of implementing EQUIPPED and be able to interpret 
potential results of the behavior change intervention regardless of primary trial outcome. 
 
The study has three research aims.  The procedures for these research aims are described 
below. 
 
• Aim 1 – Examining the Impact of Passive Provider Feedback vs. Active Provider 
Feedback Through a Randomized Trial 
• Aim 2 – Determination of Factors Affecting Organizational Adoption of EQUIPPED 
• Aim 3 – Micro-Costing the Active and Passive Feedback Versions of the EQUIPPED 
Intervention 
 
2.1 Aim 1 – Examining the Impact of Passive Provider Feedback vs. Active Provider 
Feedback 
 
2.1.2 Primary Implementation Science Trial Outcome – % of Prescriptions that Represent 
Potentially Inappropriate Medications (PIMs) 
 
The primary aim of the study is to compare the effectiveness of the EQUIPPED intervention to 
reduce the proportion of PIM prescriptions using active vs. passive processes for provider 
feedback aimed at improving prescribing toward older Veterans in the ED setting. PIMs will be 
defined according to the Beers criteria, which list medications that should generally be avoided 
in older adults.15,40,41 The Beers criteria have been used extensively in PIM research in the 
outpatient setting, along with data supporting the use of PIM measurement as an overall measure 
for better prescribing quality.4-6 Data on PIMs and all prescriptions will be gathered from the 
CDW from the eight VAMCs 6 months before and 12 months after the intervention period. As 
detailed in the power calculation, the study requires a minimum of 100 prescriptions per month 

EQUIPPED Implementation Evaluation, Protocol, v4, 06-25-2021 
9 of 25 
for older Veterans at each site. It is likely that most EDs will have many more than the required 
100 prescriptions per month during the sampling period. The limit is above 100 because it is 
likely some EDs will have lower overall patient volumes, especially for older Veterans. Using 
our previously developed code (SQL) for the current EQUIPPED QI sites, we will be able to 
collect data needed to determine monthly PIM rates, number of older Veterans (≥65 years of age) 
seen monthly, number of older Veterans who were discharged home, number of prescriptions for 
these discharged Veterans, and the number of PIMs, including specific drugs.  For the sites, we 
will be able to describe the volume of ED visits overall and for older Veterans, the number of 
providers, and the number of overall prescriptions. We will ensure data validity by checking for 
inter-rater agreement on 10% of the data gathered by CDW and chart review at each site by the 
champions. The primary endpoint will be the reduction in PIM risk from baseline to 12-
months after the EQUIPPED Intervention, with a benchmark goal of <5% at each site. We 
have data suggesting that PIM risks vary at sites from 7.4-11.8% at baseline (standard deviation 
of 0.8 to 1.9). 
 
We are applying for waivers of informed consent and HIPAA authorization to allow for 
collecting prescription and related health data originating in the VA electronic health record for 
qualifying patients seen in randomized Emergency Departments. 
 
2.1.3 Aim 1 Data Analysis Plan  
 
The statistical analysis plan described below may be adjusted based on changes in statistical 
science methods and/or to fit the specific mathematical properties of collected data.  
 
The primary efficacy outcome of interest is the % of prescriptions that are PIMS as defined 
according to the Beers criteria prescribed to adults aged 65 and older and discharged from the 
ED.  Poisson regression will be used to evaluate the number of PIMs prescribed for 6 months 
prior to the first EQUIPPED intervention at each site compared to at least 12 months of 
prescribing data following the implementation of EQUIPPED provider feedback at the local site. 
The total number of prescriptions will be used as an offset term to account for differing volumes 
of prescriptions between sites as well as potentially over time. We will not have a single date of 
implementation as the implementation period will begin based on local scheduling.  The 
implementation timeline will vary by site. Poisson regression will be used to evaluate the effect 
of the two methods of provider feedback on prescribing behavior of PIMs by including 
randomization group as a variable in the model. If the Poisson model demonstrates over (or 
under) dispersion other, related, models will be explored, e.g., negative binomial. Analyses will 
be conducted using SAS version 9.4 (SAS Institute, Cary, NC). Our primary aim, on which we 
will base sample size, is the evaluation of the EQUIPPED implementation.  We will have 6 
months of pre- and 12 months of post data including hospital site, provider ID, total 
prescriptions, and PIMs.   
 
We will conduct three analyses for the primary outcome of monthly PIM prescribing to assess 
the impact of both the initially recruited, randomized sites and the newly recruited VA facility’s 
inclusion.  The analyses will be conducted as follows: 
 
1) Intention to treat analysis: Include eight originally randomized sites 

EQUIPPED Implementation Evaluation, Protocol, v4, 06-25-2021 
10 of 25 
2) Per protocol analysis: Include seven originally randomized sites that implemented EQUIPPED 
3) Per protocol analysis: Include seven originally randomized sites that implemented EQUIPPED 
+ the non-randomized 8th site that was added to the Active Provider feedback EQUIPPED group  
 
 
2.1.4 Sample Size and Power Considerations (power calculated prior to trial) and 
Selection/Recruitment of Sites/Subjects 
 
Power estimates are based on Aim 1, hypothesis 1 (i.e., primary outcome of change). The 
primary outcome is at the level of the individual prescribing decision. We plan to engage a total 
of eight VA sites with at least 1,200 eligible prescriptions per year to patients ages 65+ per ED. 
This will provide a total of approximately 4,800 eligible prescriptions during the pre-
implementation period and 9,600 eligible prescriptions during the post time period. It can be 
reasonably assumed that these eligible prescriptions will be equally distributed between the 
randomization groups. Based upon preliminary data, we estimate that approximately 7.1% of 
these total prescriptions at baseline will be Beers PIMS. Given this sample size, we have near 
100% power to detect a change in the risk of PIMs of 5% (i.e., 7.1% to 2.1%). 
 
2.1.5 Assess Impact of EQUIPPED on the Social Cognitive Factors Impact Individual Behavior 
Change 
 
We will ask physicians at participating EDs to complete a brief survey at baseline, 6, and 12 
months to assess key components of the SCT that we expect to be impacted by the intervention.  
This will allow us to determine if these components change and whether they are associated with 
the impact of the intervention on decisions made by individual providers.  Measured components 
will include: 1) behavioral capability (knowledge of Beers List Medications); 2) expectations 
about the importance of addressing PIMs as defined by the Beers List; 3) self -control (believe 
that alternative medications are available); 4) reinforcement/information to monitor goals (use of 
EQUIPPED components at 6 and 12 months), and 5) self-efficacy for making changes. Similar to 
other studies on various provider behaviors and self-efficacy, for example, (e.g., provider panel 
management,9 provider implementation of motivational interviewing in primary care,10 and 
provider intentions to prescribe pre-exposure prophylaxis11), survey questions will be developed 
specifically for this study and according to guidelines provided by Bandura (2006). 12 The 
environment (i.e. system level factors) will assessed through a formative evaluation of the 
implementation process supported by the SCT and particularly guided by the Organizational 
Theory of Implementation Effectiveness (OTIE).  This survey will be fielded after it is approved 
by the IRB. 
 
We will conduct statistical mediation analyses to test whether EQUIPPED changed the 
targeted intermediate variables of behavioral capability, expectations/expectancies, self-control, 
reinforcements, and self-efficacy (an ‘action test’) that are obtained through the survey described 
above; if change in these targeted intermediate variables was associated with a change in the % 
of prescriptions that represent PIMS (a ‘conceptual test’), and if EQUIPPED’s effects on this 
outcome variable were attributable to its effects on the intermediate variables (an ‘indirect 
effects’ or mediation test).13 The mediators will be tested simultaneously in order to determine 
the unique influence of each potential mediator. We plan for analyses will be conducted with 

EQUIPPED Implementation Evaluation, Protocol, v4, 06-25-2021 
11 of 25 
structural equation modeling using Mplus (version 6.11).14  However, other appropriate software 
may be utilized to analyze data from this study. 
 
2.2 Aim 2:  Determination of Factors Affecting Organizational Adoption of EQUIPPED 
 
We will conduct an in-depth, theory-based formative evaluation of sites, with the goals of 
understanding: 1) factors that may impact organizational readiness to implement EQUIPPED 
(i.e., organizational readiness for change (ORC)) and implementation of the EQUIPPED as 
outlined in the Organizational Theory of Implementation Effectiveness (OTIE); 2) changes in 
these factors over the course of one year; and 3) association between ORC and factors suggested 
by the OTIE to both successful implementation and sustainability over time.  We will utilize a 
mixed-methods approach to measure ORC and change in related components over time.  The 
ultimate goal is to inform the process of implementing EQUIPPED and be able to interpret 
potential results of the behavior change intervention regardless of primary trial outcome.  
Further, we will inform the implementation of future provider behavior change programs. Table 
1 below details factors that will be examined using the OTIE through the formative evaluation.  
Definitions are those developed by Bryan Weiner based on his adaptations of work by Klein and 
Sorra.15-18  The numbers in the table referring to data collection processes correspond to section 
2.2.1 below.  Other applicable theories, frameworks, or related concepts may be considered 
during data analysis. 
 
Table 1: OTIE Factors Assessed through Formative EQUIPPED Evaluation 
Factor 
Definition 
Data Collection 
Process 
Pre-and Early Implementation of EQUIPPED – Factors Expected to Impact the 
Readiness of Sites to Implement EQUIPPED 
Change Valence 
Value that organizational members ascribe to a 
proposed change (i.e., its perceived attractiveness). 
2 
Task Demands 
Knowledge about the tasks that need to be 
performed, the resources (human, financial and 
material) that are needed, and the time and effort 
that are needed to implement the intervention. 
2 
Resource 
Availability 
Accessibility of financial, material, or human assets 
that can be used to support initial and ongoing 
innovation use. 
1,2 
Situational Factors Contextual elements that affect the confidence and 
commitment of organizational members to 
implement the intervention. 
2 
Organizational Readiness for Change/To Implement EQUIPPED 
Organizational 
Readiness for 
Change (ORC) 
Extent to which targeted organizational members 
are prepared to make the changes in organizational 
policies and practices that are necessary to put an 
innovation into practice and support its use.  This is 
based on 2 factors: 1) commitment to change and 2) 
change efficacy. 
2 
Process of EQUIPPED Implementation 

EQUIPPED Implementation Evaluation, Protocol, v4, 06-25-2021 
12 of 25 
Implementation 
Policies and 
Practices (IPPs) 
Plans, practices, structures, and strategies that an 
organization employs to put the innovation into 
place to support innovation use 
3,5 
Implementation 
Climate 
Shared perceptions of implementation policies and 
practices in terms of their meaning and significance 
for innovation use 19. 
3,5 
Innovation-Values 
Fit 
Extent to which implementers and users perceive 
that innovation use will foster the fulfillment of 
their values 19-22. Values are concepts or beliefs that 
(a) pertain to desirable end-states or behaviors, (b) 
transcend specific situations, and (c) guide the 
selection and evaluation of behavior and events 23. 
3,5 
Innovation-Task 
Fit 
Extent to which the innovation is compatible with 
task demands, work processes, and organizational 
capabilities. 
3,5 
EQUIPPED Effectiveness and Sustainability 
Implementation 
Effectiveness 
Consistency, quality, and appropriateness of 
innovation use within an organization19,20,24. 
4,5 
Innovation 
Effectiveness 
Benefits an organization realizes from an 
innovation 19 (i.e., did EQUIPPED lead to a 
reduction in PIMs). 
Trial outcome 
Sustainability 
Expected ability to continue to utilize EQUIPPED 
following the study. 
6 
 
 
2.2.1 Formative Evaluation Process:  
 
In summary, the formative evaluation will begin by identifying baseline characteristics (i.e., 
before and/or during the early stages of implementation) of the organization and team that may 
impact implementation.  This will be followed by: 1) assessment of readiness to implement 
EQUIPPED; 2) monitoring of the implementation process; 3) monitoring of implementation 
progress; 4) qualitative interviews addressing implementation factors suggested by 
approximately the 6-8 months following initial implementation of EQUIPPED; and 5) evaluation 
of program sustainability approximately 1 year after the delivery of the delivery of the first 
EQUIPPED report.  
 
Steps in the Formative Evaluation Process 
1. Collection of Baseline Characteristics that may Impact Implementation.  The eight sites 
will be asked to identify all individuals directly involved in the planning and execution of 
implementing EQUIPPED (e.g. clinical champions, ED Chiefs, pharmacists CACs). This is 
termed the core implementation team.  Additionally, the eight sites will be asked to identify 
ED providers and staff that may encounter changed clinical decision-making as a result of 
implementation of the applications. These sites will be asked to complete a baseline site 
information document (1 survey report per site/medical center filed via VA REDCap Survey 
software) that will collect information on core implementation team members and processes 
and size and composition of the medical centers and impacted clinical services, as well as 

EQUIPPED Implementation Evaluation, Protocol, v4, 06-25-2021 
13 of 25 
identify additional ED providers and staff potentially impacted by the implementation.  The 
research team may update lists of names of individuals or information from the site 
information document as needed to conduct the study.    
2. Assessment of ORC.  Organizational readiness for change is the extent to which 
organizational members are prepared as a group to make changes in organizational 
policies/practices that are necessary to implement and support innovation use (change 
commitment) and their perceived ability to do so (change efficacy). As with individuals, 25 
attributes impacting ORC include change valence (perceived value of the innovation) and 
information about perceived task demands, resources available, and situational context (e.g., 
competing demands).18,26 
ORC Survey:  Core implementation team members and ED providers will receive the 
validated Organization Readiness for Implementing Change measure, a 12-item computer-
based survey which examines perceptions of organizational-level change efficacy and 
commitment to newly implemented interventions. Survey responses will objectively examine 
ORC as a two-dimensional construct encompassing change commitment and change efficacy.  
This instrument was developed specifically to measure aspects of the Weiner Theory of 
ORC.26  The survey may include additional context questions and will receive IRB approval 
prior to being fielded.   The survey will be fielded via VA REDCap survey software. 
ORC Semi-Structured Qualitative Phone Interview:  In addition, we will conduct semi-
structured qualitative telephone interviews of the core implementation team and 
approximately 3-5 ED providers at each site to assess ORC and factors that are hypothesized 
to predict ORC (i.e. change valance/value place on the apps and assessment of what it will 
take to implement the EQUIPPED program). Semi-structured interviews will allow us to 
study implementation processes, which tend to be non-linear and context sensitive27,28 and 
will permit us to compare patterns across cases.29  The semi-structured interview guide will 
need to receive IRB approval interviews are conducted. 
3. Quarterly Documentation of Process/Workflow of Implementation.  As part of the quality 
improvement support (i.e., not research activity) for implementing EQUIPPED, the QI team 
will work with each site to establish and summarize the workflow used as part of equipped.  
Components of the workflow process will then be stored in a Microsoft Excel or Word 
spreadsheet or document to be sent to the sites for updates approximately every three months.  
The exact format of the summary, which may be updated and will be retained by the research 
team, will potentially vary based on the quality improvement needs and decisions of each 
participating site. .  
4. Bi-monthly Monitoring of Implementation – Implementation Progress.  We will also 
measure the degree of implementation and barriers and facilitators of implementation process 
through approximately bi-monthly facility-level reports from the eight sites collected through 
a computerized survey using the VA REDCap survey software.   Site progress with 
implementation will be assessed using the Stages of Implementation Completion (SIC).30,31  
The SIC enumerates key pre-implementation, implementation and sustainability milestones. 
Dates by which specific implementation milestones were reached will be identified.  We will 
then examine if the degree of ORC is associated with the rapidity with which sites go through 
implementation steps.  Bi-monthly (every other month) reports will also include assessment of 
barriers and facilitators identified through the ORC measurement process.   
5. Evaluation of the Implementation Process – Qualitative Interviews.  At approximately 6-8 
months following the start of the implementation process at each of the eight sites, we will 

EQUIPPED Implementation Evaluation, Protocol, v4, 06-25-2021 
14 of 25 
conduct semi-structured qualitative telephone interviews among the core implementation team 
at each site to assess OTIE factors suggestive of implementation success.  The goal will be to 
interview the same individuals interviewed at baseline.  The questions composing the 
qualitative interview guide will depend upon the formative evaluation.  As such, the guide 
will be developed after the initial implementation of EQUIPPED and approved by the IRB 
prior to use. 
6. Assessment of Sustainability.  At approximately one-year post-implementation at each site, 
we will assess the sustainability of EQUIPPED.  
Sustainability (PSI) Survey:  The Mancini & Marek Model of Community-based Program 
Sustainability will be used to conceptually guide the evaluation of sustainability.32  Mancini 
& Marek propose that six elements are important to achieve long-term sustainability:  
Leadership competence, effective collaboration, demonstrating program results, strategic 
funding, staff involvement and integration, and program responsivity.  The validated 23-item 
Program Sustainability Index (PSI) computer-based survey measures the 6 sustainability 
elements.32Sustainability Semi-Structured Qualitative Phone Interview:  Each core 
implementation team member and ED provider may be surveyed.   Additionally, using semi-
structured interview methods, we will interview the members of the core implementation 
team at each site.  Precise timing of the interviews may be impacted by individuals’ 
schedules in relation to time of the intervention and previous interviews.  We have previously 
used the combination of PSI and qualitative interviews to evaluate the sustainability of a 
multi-facility VA program.33  The questions composing the qualitative interview guide will 
depend upon the formative evaluation.  As such, the guide will be developed after the initial 
implementation and approved by the IRB prior to use. 
7. Consent Process for Formative Evaluation.  Staff participation in the formative evaluation 
is entirely voluntary.  Some individuals may feel self-conscious or embarrassed if asked 
questions about something with which they are unfamiliar.  Additionally, because research 
participants are VA employees, they may be concerned their responses will become known to 
colleagues or supervisors.  To minimize this risk, we are requesting a waiver of documented 
consent, in part, to avoid providing a link between the research participant and the data 
collected.   
An introductory email may be sent from a known individual at each site informing each 
potentially eligible participant of the study.  The introductory email need not be resent upon 
outreach for subsequent surveys or invitations to participate in interviews with the same 
participants.  
For the surveys, the potential introductory email described above will be followed by a 
scripted email (must be approved by the IRB) sent from study staff containing information 
about the study and informed consent.  Clicking the link to start the survey implies informed 
consent.  
For the qualitative interviews, the potential introductory email described above will be 
followed by a scripted email sent from study staff containing information about the study and 
informed consent.  Study staff will follow-up either by email or phone to schedule a phone 
interview with interested parties.  At the start of the scheduled interview, the person 
conducting the interview will ask if there are any questions about the study and review the 
elements of informed consent.  The study staff member will gather verbal informed consent 
from the participant(s) prior to proceeding with the interview.  Consent documents must be 
approved by the IRB prior to beginning related study procedures. 

EQUIPPED Implementation Evaluation, Protocol, v4, 06-25-2021 
15 of 25 
 
2.2.2 Formative Data Analysis 
 
The analysis plan described below may be adjusted based on changes in related scientific 
methods and/or to fit the specific properties of collected data.  
 
Core to the concept of formative evaluation is continual analysis of results and feedback to 
stakeholders. Like with statistical analysis, the analysis plan laid out below may be adjusted 
based on the statue of the science related to such analysis and the specific features of collected 
data.  Key data sources are qualitative interviews, surveys to assess organizational readiness for 
change, and collection of detail about organizational characteristics and implementation process 
and progress. All qualitative interviews will be transcribed in full. Rapid analysis approaches 
will generate preliminary findings to share among the research team. This effort will involve an 
initial review of factors identified as directly impacting the process of supporting implementation 
and impact on clinical workflow. Rapid analysis will be followed by in-depth content analysis.  
Content analysis to examine the telephone interviews will involve three phases: data coding, 
within-case analysis, and between-case analysis. In the data coding phase, we will use qualitative 
data analysis software (ATLAS.ti or other appropriate analysis software) to code the study data. 
The OTIE will provide a starting list of codes, which we will supplement with emergent codes as 
analysis proceeds. Using a common codebook, two research staff members will conduct a 
preliminary test of codes by independently coding approximately five transcripts (more may be 
independently reviewed if determined to be necessary). Based on the preliminary test, the 
research staff members will sharpen the coding manual’s definitions, decision rules, and 
examples. Research assistants will code the remaining documents.   
 
In the second phase, we will conduct a within-case analysis of each VA using ATLAS.ti to 
generate reports of all text segments for each code. We will assess the degree to which the 
construct emerges in the data (its “strength”), the degree to which the construct positively or 
negatively affects implementation (its “valence”), and the degree to which relationships among 
constructs are consistent with the hypothesized model.  We will assess support for the 
hypothesized relationships by using three criteria proposed by Trochim34 and Miles and 
Huberman.35  First, we will look for the overall covariance of the constructs (e.g., whether VA 
clinics exhibiting strong implementation climate have supportive administration).  Second, we 
will look for explicit attributions or the identification of plausible mechanisms to link the two 
constructs (e.g., participants attribute a strong implementation climate to the deployment of 
appropriate implementation policies and practices).   
 
In the third phase, we will apply the same criteria across the cases to determine if cross-case 
variation in implementation is consistent with the hypothesized relationships in the model. 
Consistent with the organization-level focus of the model, we will aggregate and analyze 
quantitative data on implementation policies and practices (e.g. staffing levels) and other study 
constructs using simple statistics. In addition, we will create within-case and between-case data 
displays that cross-tabulate the quantitative and qualitative data in order to facilitate the use of 
pattern-matching logic.35 
 

EQUIPPED Implementation Evaluation, Protocol, v4, 06-25-2021 
16 of 25 
2.2.3 Linking Measures Within and Across Implementation Processes from Different Data 
Sources:   
 
We will combine quantitative measures of the degree of ORC, factors outlined in the OTIE that 
may influence implementation process, completeness of implementation progress, and 
sustainability to both implementation and intervention effectiveness outcomes. These data will 
be supplemented with qualitative data sources such as semi-structured individual interviews and 
supplemental process information.  Inclusion of the qualitative component will enhance 
understanding of the local context.  Quantitative and qualitative findings will produce mixed 
method findings to inform our understanding of the process of implementing EQUIPPED. 
 
2.3   Aim 3. Micro-Costing the Active and Passive Feedback Versions of the EQUIPPED 
Intervention 
 
Because the earlier version of the EQUIPPED intervention was done as a quality improvement 
project, we do not have detailed information on how much the intervention costs.  For example, 
while the 10 sites that have implemented a QI version of EQUIPPED received an average of 
$50,000 from the Office of Geriatrics and Extended Care, we have no way of knowing details of 
how that money was spent and whether costs were covered.  By micro-costing the intervention 
we will be able to learn details of specific factors that may impact the cost of the intervention to 
the organization. Such costs will impact feasibility of wide and sustained implementation of 
EQUIPPED, if successful. Specific costs will account for 1) labor costs; 2) equipment; and 3) 
overhead costs such as utilities.  The cost of the intervention will be considered from the VA 
health system perspective, hence, we will not consider individual patient-level direct or indirect 
costs.36 Costs will consider both the cost of implementing the program and carrying it out. 
Because many of the tools have been developed for the intervention and will be provided to the 
medical centers, the incremental cost of the program will not include the cost of developing these 
tools. 
 
The micro-costing analysis plan described in this section may be adjusted based on changes in 
related scientific methods and/or to fit the specific properties of collected data. 
 
Micro-costing of the intervention involves direct measurement of the labor inputs required to 
both implement and conduct each intervention task.  Labor cost includes the fixed costs of 
implementation planning (e.g., participation in meetings) and training staff of the intervention 
processes as well as the variable cost of delivering the intervention to each provider.  We have 
developed a log sheet with instructions that can be used by a sample of providers to log their 
activity time for patients in a given project over the course of a one-week period. The training, 
implementation, and intervention-related time of each provider will be multiplied by his/her 
respective wage rate (including fringe benefits), aggregated, and then divided by the number of 
Veterans served at each ED to derive per-Veteran intervention labor cost for a site. We will 
account for equipment needed to deliver the interventions (e.g., computers, patient handouts). 
The VA healthcare system also incurs substantial indirect costs such as administrative costs, 
utilities, etc., that are not specific to a health service. We will calculate a cost multiplier using the 
total indirect and direct cost variables in the VA’s Managerial Cost Accounting System 

EQUIPPED Implementation Evaluation, Protocol, v4, 06-25-2021 
17 of 25 
(formerly Decision Support System) extract file and apply it to the above direct cost estimates to 
derive total (direct + indirect) intervention cost. 
 
3.0 Privacy, Confidentiality, and Information Security 
 
3.1. Lists of Data Reviewed and/or Collected for Screening/Recruitment and Conduction of 
Study: 
 
3.1.1 Patients:  The Personal Health Information that will be obtained, used, and/or shared for 
VA patients in this study includes: 
Identifier(s) 
Source(s) of Health Information 
 Names 
 Medical history & physical exam 
information 
 All geographic subdivisions smaller than a 
State, including street address, city, county, 
precinct, and zip code.  Describe: VA Facility, 
city 
 Photographs, videotapes, audiotapes, or 
digital or other images 
 All elements of dates (except year) for dates 
directly related to an individual, including birth 
date, admission date, discharge date, visit or 
treatment dates, etc.; and all ages over 89,  
Describe:  ED visit dates, patient’s DOB 
 Biologic specimens (e.g., blood, tissue, 
urine, saliva). Describe: 
 Telephone numbers 
 Progress notes 
 Fax numbers 
 Diagnostic / Laboratory test results 
 Electronic mail addresses 
 Operative reports 
 Social Security Numbers 
 Imaging (x-ray, CT, MRI, etc.) 
 Medical record numbers 
 Discharge summaries 
 Health plan beneficiary numbers 
 Survey / Questionnaire responses 
 Account numbers 
 Billing records 
 Certificate and/or license numbers  
 HIV testing or infection records 
 Vehicle identifiers and serial numbers, 
including license plate numbers  
 Sickle cell anemia information 
 Device identifiers and serial numbers 
 Alcoholism or alcohol use information 
 Web Universal Resource Locators (URLs) 
 Drug abuse information 
 Internet Protocol (IP) address numbers 
 Mental health (not psychotherapy) notes 
 Biometric identifiers, including finger & 
voice prints 
 Psychological test results 
 Full-face photographic images and any 
comparable images 
 Genetic testing 
 Any other unique identifying number, 
linked study ID, characteristic, or code, 
describe:        
 Other, describe:        
 
3.1.2 VA Employees:  We are NOT collecting Personal Health Information from VA employees 
in this study.  However, this is the data that will be obtained, used, and/or shared for VA 

EQUIPPED Implementation Evaluation, Protocol, v4, 06-25-2021 
18 of 25 
employees:  names, the employee’s VA facility including city, survey/questionnaire responses, 
audiotaping of interviews, and a study ID created for the VA employee.   
 
3.2 Data and/or Specimen Acquisition 
Data for this study will be collected through (check all that apply): 
  Prospective data and/or specimen collection obtained from participants.  Provide description 
of processes:  The primary outcome related to provider prescribing will collect aggregate data 
relating to provider prescribing patterns for Veterans 65 years and older who are discharged from 
each of the eight EQUIPPED VA ED sites through the VA Corporate Data Warehouse (CDW). 
While some characteristics such as unique Veteran ID will be collected in order to develop the 
provider feedback reports, the research project has no direct or indirect patient contact or 
involvement.  Data collected for the formative evaluation and micro-costing aims do not include 
patient-level information.  Data for the formative evaluation will be obtained via electronic 
surveys and interviews of VA Employees.  Regarding the data collection in support of Aim3, 
authorized study personnel will access the VA’s Managerial Cost Accounting System (formerly 
Decision Support System) extract file and apply it to the direct cost estimates to derive total 
(direct + indirect) intervention cost.   
  Retrospective data collection and/or specimens obtained from medical chart review/data 
access.  Describe how data will be obtained (e.g., fileman, CDW, etc.):  As noted previous in 
Section 1.5.1, Stage 1, subsection 3, the Atlanta implementation team will obtain from CDW 
baseline prescribing data for the 6 months prior to implementation to establish initial levels of 
performance. 
  Retrospective data collection and/or specimens obtained from an IRB-approved data and/or 
specimen repository.  Indicate the repository source including name, VA location, and IRB 
number:       . 
3.3 Level of Data 
The following level(s) of data will be acquired/maintained for this study (check all that apply): 
  Identified (e.g., names, addresses or other identifiers included)  
  Coded (direct and/or all identifiers removed, but study code/ID included)  
  De-Identified (all HIPAA 18 and study ID/code removed): 
  Verified Statistically  
 
OR 
  Verified by Absence or Removal of HIPAA 18 and study ID  
  Limited Data Set 
  Other: Describe:        
 
4.4 Location of Data and/or Specimens, and Data Retention Plan 
 
4.4.1 Data and/or Specimen Location:   
 

EQUIPPED Implementation Evaluation, Protocol, v4, 06-25-2021 
19 of 25 
Electronic Data:  ED provider participant names and contact information will be stored in a 
secure database housed and maintained on the VA Atlanta research server.  Appropriately 
credentialed study staff members form the Atlanta and Durham VAMC teams may access 
identifiable research information stored on the Atlanta server to the extent needed to conduct the 
study.  Further, a copy of data related to the formative evaluation and micro-costing aims of this 
study may be stored on secure servers at the Durham and Atlanta VAMC.  All electronic data 
stored in Durham will be on VA servers behind the VA firewall on the HSR&D drive 
(p:\EQUIPPED).  Identifiable data from the study may be analyzed on either the Atlanta or 
Durham VAMC research servers.  As soon as practical, all official final datasets from all aims 
will be stored on research server at the Atlanta VAMC. 
Prescribing data will be stored in a research-secure project folder within the VA Informatics 
and Computing Interface (VINCI at 
http://vaww.vinci.med.va.gov/vincicentral/VINCIWorkspace.aspx). VINCI is a partnership 
between the VA Office of Information Technology and the Veterans’ Health Administration 
Office of Research and Development.  Researchers and operations staff can use VINCI to access 
data and statistical analysis tools in a virtual working environment through a certified VHA 
network computer using the VA Intranet or Virtual Private Network (VPN). 
As described above, surveys and reports collected as part of the formative evaluation will be 
collected through VA REDCap (Research Electronic Data Capture) stored on the VINCI servers.  
These surveys will not contain individually identifiable patient information.  Data from the 
REDCap surveys may be downloaded to secure research servers at the Atlanta and/or Durham 
VA Medical Centers. Interview data will be collected by recording phone conversations.  While 
most study interviews will be recorded directly to researcher servers in Durham or Atlanta, study 
interview recordings may initially be made on the remotely-located team member’s VA-issued 
and maintained computer and will be moved to the Atlanta server as soon thereafter as is 
practical.   
Per standard VA-wide Health Services Research & Development operating procedures for 
transcribing VA research recordings, approved staff from the VA Salt Lake City (VASLC) will 
transcribe the EQUIPPED audio files. The VASLC has a Professional Transcription Service 
available to VA sites and monitored by their own IRB. The EQUIPPED audio recordings to be 
transcribed by VASLC staff will be labeled by the subject's unique alphanumeric code and saved 
behind the VA Firewall in EQUIPPED study’s secure shared project folder on the Atlanta or 
Durham VAMC researcher servers. The VASLC transcription staff will be given access to a sub-
folder within EQUIPPED study’s secure project folder located on the Durham IRB server (study 
folder labeled EQUIPPED IIR) where audio files are located.  While not anticipated, similar 
arrangements may be made to access data on the Atlanta researcher server if needed.  Approved 
study staff will place a copy of the audio files in this folder for an approved VASLC 
transcriptionist to access for the purposes of transcription. The VASLC transcriptionist will 
transcribe each interview verbatim and save the completed transcript in the sub-folder using the 
same alphanumeric code. No data (audio files, in process transcripts, or completed transcripts) 
will leave the Durham or Atlanta secure research server. As completed transcripts become 
available, approved study staff will move these files from the transcription sub-folder into 
another sub-folder that is only accessible to study staff, where they will be stored and accessed 
for qualitative analyses.   
 

EQUIPPED Implementation Evaluation, Protocol, v4, 06-25-2021 
20 of 25 
Paper records of data:  As referenced in Section 2.3, cost data will be calculated by using a log 
sheet with instructions that can be used by a sample of providers to log their activity time for 
patients in a given project over the course of a one-week period.  This log sheet will be collected 
by the Site Champion and sent via encrypted email to study staff.  Any hard copy files containing 
identifiable information will be kept in a locked file cabinet in a locked office space under the 
care of study research staff.  
 
4.4.2 Data Retention Plan 
 
Research records will be maintained and destroyed according to the National Archives and 
Records Administration, Records Schedule Number:  DAA-0015-2015-0004.  Records 
destruction, when authorized, will be accomplished using the then current requirements for the 
secure disposal of paper and electronic records.  Currently, destruction of research records (see 
DAA-0015-2015-0004, section 7.6 “Research Investigator Files” for materials included in 
research records) is scheduled for 6 years after the cut-off (the cut-off is the completion of the 
research project) and may be retained longer if required by other federal agencies.  Records will 
not be destroyed without pre-notification to the facility records manager. 
 
5.0 Data Access and Data Recipients 
 
Only study staff in Atlanta and Durham involved in recruitment, interviewing, sending surveys, 
and general participant contact as well as those supervising said activities and staff will have 
access to individually identifiable research data.  As previously mentioned, a certified de-
identified data set will be provided by our research team in Atlanta to study team at the 
Birmingham VA for their analysis work. 
All VA research personnel who have access to VHA records are instructed, in accordance 
with VA policy, on the requirements of Federal privacy and information laws and regulations, 
VA regulations and policies, and VHA policy. All study personnel who are VA employees 
working within the VA system have fulfilled all required HIPAA and other VA security and 
privacy policy training requirements and have agreed to follow guidelines pertaining to the 
protection of patient data. All research staff sign VA Rules of Behavior, and all study staff are 
up-to-date with VHA Privacy Policy Training and the VA Office of Cyber and Information 
Security Awareness Training Course. The data security and privacy procedures summarized in 
that course include logging off or locking the computer when walking away from it; no sharing 
of access codes, verify codes or passwords; not allowing anyone else to use the computer under 
one’s password; and disposing of sensitive information using VA-approved methods (e.g., 
shredder bins).  Access to study data will be removed for all study personnel when they are no 
longer part of the research team.  
6.0 Data and/or Specimen Transportation and/or Transmission for all data and/or 
specimens involved in the study: 
 
The coded data containing identifiers will be transmitted between the Durham and Atlanta VA 
study sites using PKI encrypted email.  The data set shared with the study team at the 
Birmingham VA will be de-identified and thus can be sent via unencrypted e-mail or 

EQUIPPED Implementation Evaluation, Protocol, v4, 06-25-2021 
21 of 25 
unencrypted disk (encryption is optional).  Data will not be removed from the VA protected 
environment.  
 
7.0 Risk Mitigation Strategies: 
 
For the data collection involving gathering data from patient medical records using CDW, the 
risk of a privacy or data security incident is minimal as there is no subject contact and data is 
collected in aggregate.  Only study staff have access to the research data.  As described above in 
Section 1.4.1, subsection for “Provider Feedback”, the Site Champion will have access to 
prescribing data for operational purposes.  The Site Champion will be trained as to appropriate 
use of this information and patient privacy protections. 
 For the survey and qualitative interviews of VA employees, there is a small risk of losing 
anonymity with the use of demographic information.  Additionally, some individuals may feel 
self-conscious or embarrassed if asked questions about something with which they are 
unfamiliar, and they may be concerned that their responses will become known to colleagues and 
supervisors.  All research-related interaction with VA employees will be conducted by study 
staff located at a different VA facility.  Survey responses and recordings/transcripts of interviews 
will be stored on secure servers to which only study staff have access.  
All study staff, including the PI, will participate in the monitoring of participant data. The 
proposed protocol contains minimal risk so no adverse events are expected. Some participants 
may not complete scheduled interviews or surveys.  This is expected and will not be considered 
an AE. However, should any adverse event occur, the study staff member who discovered the 
adverse event would immediately alert the Principal Investigator, who will have the 
responsibility of informing the IRB of record for the study. 
8.0 Suspected Loss of VA Information 
Should any incident such as theft or loss of data, unauthorized access of sensitive data or non-
compliance with security controls occur it will be immediately reported according to VA policy. 
All incidents regarding information security/privacy incidents will be reported to the ISO and PO 
within 1 hour of acknowledgement of issue and done so using the VHADUR Research Events 
Report e-mail group (VHADURResearchEventReport@va.gov). 
 
9.0 Reporting of Results 
Reporting of results, such as in scientific papers and presentations, will never identify individual 
subjects.  Data will be presented in aggregate and individual-level data will not be published. 
10.0 Future Use of Data 
No future use of data is currently planned. 
11.0 Use of Mail Merge Technology 
Not applicable. 
12.0 Use of Non-Standard Software 

EQUIPPED Implementation Evaluation, Protocol, v4, 06-25-2021 
22 of 25 
We do NOT intend to use any new specialized software (i.e. software that’s not already approved 
OR installed) in this study. 
13.0 Use of Cloud Computing Services 
Cloud computing services will NOT be used in this study. 
 
 
 
 

EQUIPPED Implementation Evaluation, Protocol, v4, 06-25-2021 
23 of 25 
References 
1. 
Stevens MB, Hastings SN, Powers J, et al. Enhancing the Quality of Prescribing Practices for Older 
Veterans Discharged from the Emergency Department: preliminary results from EQUIPPED, a 
novel multicomponent interdisciplinary quality improvement initiative. J Am Geriatr Soc. May 
2015;63:1025-1029. 
2. 
Moss JM, Bryan 3rd WE, Wilkerson LM, et al. Impact of clinical pharmacy specialists on the 
design and implementation of a quality improvement initiative to decrease inappropriate 
medications in a Veterans Affairs emergency department. J Manag Care Spec Pharm. 
2016;22:74-80. 
3. 
Stevens M, Hastings SN, Markland AD, et al. Enhancing Quality of Provider Practices for Older 
Adults in the Emergency Department (EQUiPPED). Journal of the American Geriatrics Society. 
2017;65(7):1609-1614. 
4. 
O'Brien MA, Rogers S, Jamtvedt G, et al. Educational outreach visits: effects on professional 
practice and health care outcomes. Cochrane Systemat Rev. 2007;4. 
5. 
By the American Geriatrics Society Beers Criteria Update Expert P. American Geriatrics Society 
2015 Updated Beers Criteria for Potentially Inappropriate Medication Use in Older Adults. J Am 
Geriatr Soc. 2015;63(11):2227-2246. 
6. 
Ivers N, Jamtvedt G, Flottorp S, et al. Audit and feedback: effects on professional practice and 
healthcare outcomes. Cochrane Database Syst Rev. 2012;6:Cd000259. 
7. 
Kiefe CI, Allison JJ, Williams OD, Person SD, Weaver MT, Weissman NW. Improving quality 
improvement using achievable benchmarks for physician feedback: a randomized controlled 
trial. JAMA. Jun 13 2001;285(22):2871-2879. 
8. 
Vanderman AJ, Moss JM, Bryan WE, Sloane R, Jackson GL, Hastings SN. Evaluating the Impact of 
Medication Safety Alerts on Prescribing of Potentially Inappropriate Medications for Older 
Veterans in an Ambulatory Care Setting. Journal of Pharmacy Practice. December 23, 2015 2015. 
9. 
Strauss SM, Jensen AE, Bennett K, Skursky N, Sherman SE, Schwartz MD. Clinicians’ panel 
management self-efficacy to support their patients’ smoking cessation and hypertension control 
needs. Translational Behavioral Medicine. 09/16 2015;5(1):68-76. 
10. 
Midboe AM, Cucciare MA, Trafton JA, Ketroser N, Chardos JF. Implementing motivational 
interviewing in primary care: the role of provider characteristics. Translational behavioral 
medicine. 10/28 2011;1(4):588-594. 
11. 
Sachdev DD, Stojanovski K, Liu AY, Buchbinder SP, Macalino GE. Intentions to Prescribe 
Preexposure Prophylaxis Are Associated With Self-efficacy and Normative Beliefs. Clinical 
Infectious Diseases. 2014;58(12):1786-1787. 
12. 
Bandura A. Guide for constructing self-efficacy scales. Vol 5. Greenwich, CT: Information Age 
Publishing; 2006. 
13. 
Cerin E, MacKinnon DP. A commentary on current practice in mediating variable analyses in 
behavioral nutrition and physical activity. Public Health Nutr. 2009;12(8):1182-1188. 
14. 
Muthen LK, Muthen BO. Mplus User's Guide. 6th ed. Los Angeles, CA: Author; 2011. 
15. 
Klein K, Sorra J. The challenge of innovation implementation. Academy of. Management Review. 
1996;4:1055-1080. 
16. 
Klein K, Conn A, Sorra J. Implementing computerized technology: A Organizational Analysis. . 
Journal of Applied Psychology.86(NO.5):811-824. 
17. 
Weiner BJ, Lewis MA, Linnan LA. Using organization theory to understand the determinants of 
effective implementation of worksite health promotion programs. Health Educ Res. Apr 
2009;24(2):292-305. 
18. 
Weiner BJ. A theory of organizational readiness for change. Implement Sci. 2009;4:67. 

EQUIPPED Implementation Evaluation, Protocol, v4, 06-25-2021 
24 of 25 
19. 
Klein KJ, Sorra JS. The Challenge of Implementation. Academy of Management Review. 
1996;21(4):1055-1080. 
20. 
Holahan PJ, Aronson ZH, Jurkat MP, Schoorman FD. Implementing Computer Technology: A 
Multi-Organizational Test of Klein and Sorra's Model. Journal of Engineering and Technology 
Management. 2004;21:31-50. 
21. 
Leonard-Barton D. Implementation as Mutual Adaptation of Technology and Organization. 
Research Policy. 1988;17:251-267. 
22. 
Leonard-Barton D. Implementation Characteristics of Organizational Innovations. 
Communication Research. 1988;15:603-631. 
23. 
Schwartz SH, Bilsky W. Toward a Theory of the Universal Content and Structure of Values - 
Extensions and Cross-Cultural Replications. Journal of Personality and Social Psychology. May 
1990;58(5):878-891. 
24. 
Klein KJ, Conn AB, Sorra JS. Implementing Computerized Technology. Journal of Applied 
Psychology. 2001;86(5):811-824. 
25. 
Prochaska JO, Redding CA, Evers KE. The Transtheoretical Model and Stages of Change. In: Glanz 
K, Rimer BK, Lewis FM, eds. Health Behavior and Health Education, Theory, Research, and 
Practice. 3rd ed. San Francisco, CA: Jossey-Bass; 2002:99-120. 
26. 
Shea CM, Jacobs SR, Esserman DA, Bruce K, Weiner BJ. Organizational readiness for 
implementing change: a psychometric assessment of a new measure. Implement Sci. 2014;9:7. 
27. 
Ferlie E. The nonspread of innovations: the mediating role of professionals. Acad Manage J. 
2005;48:117-134. 
28. 
Van De Ven A, Polley D. The Innovation Journey. New York: Oxford University Press. 1999. 
29. 
Polit D, Beck C. Nursing research: principles and methods. Philadelphia: Lippincott Williams & 
Wilkins; 2004. 
30. 
Chamberlain P, Brown CH, Saldana L. Observational measure of implementation progress in 
community based settings: the Stages of Implementation Completion (SIC). Implement Sci. 
2011;6:116. 
31. 
Saldana L, Chamberlain P, Wang W, Hendricks Brown C. Predicting program start-up using the 
stages of implementation measure. Adm Policy Ment Health. Nov 2012;39(6):419-425. 
32. 
Mancini JA, Marek LI. Sustaining community-based programs for families: conceptualization and 
measurement. Family Relations. 2004;53:339-347. 
33. 
Fortune-Britt AG, Nieuwsma JA, Gierisch JM, et al. Evaluating the implementation and 
sustainability of a program for enhancing veterans' intimate relationships. Mil Med. Jun 
2015;180(6):676-683. 
34. 
Trochim WMK. The Research Methods Knowledge Base, First Edition. Cincinnati, OH: Atomic Dog 
Publishing; 2001. 
35. 
Miles MB, Huberman AM. Qualitative Data Analysis: An Expanded Sourcebook, Second Edition. 
Thousand Oaks, CA: Sage Publications; 1994. 
36. 
Sanders GD, Neumann PJ, Basu A, et al. Recommendations for conduct, methodological 
practices, and reporting of cost-effectiveness analyses: Second panel on cost-effectiveness in 
health and medicine. JAMA. 2016;316(10):1093-1103.