Evaluation of Implementation of a National Point-of-Care Ultrasound Training Program
Conditions: Other Acute Illnesses Presenting to the Hospital, Heart Failure, Pneumonia, Deep Venous Thrombosis, Cellulitis, Abdominal Aortic Aneurysm, Acute Cholecystitis, Peritoneal Free Fluid, Hydronephrosis, Pleural Effusion, Urinary Retention
Sponsor: VA Office of Research and Development

Department of Veterans Affairs
Quality Enhancement Research Initiative (QUERI)
Evaluation of Implementation of a National Point-of-Care Ultrasound Training 
Program
Partnered Evaluation Initiative
(NCT03296280)
September 9, 2016
Principal Investigator
Nilam Jayant Soni, MD, MSc
VA San Antonio
nilam.soni@va.gov

1. Specific Aims
 
Point-of-care ultrasonography (POCUS) is a focused, goal-directed bedside ultrasound examination to 
guide evaluation and management of patients. POCUS allows clinicians to rapidly detect conditions and 
reduces the number of diagnostic imaging tests, particularly computed tomography (CT) scans.1 POCUS can 
also be used to guide procedures and has been shown to reduce procedural complications and improve 
procedural success rates.2 Thus, POCUS improves patient care by expediting diagnosis, especially of life-
threatening conditions; reducing radiation exposure from unnecessary diagnostic imaging tests; and minimizing 
risks of complications from invasive bedside procedures. By reducing the number of imaging tests and time to 
diagnosis, POCUS reduces healthcare costs.3    
 
Despite its potential advantages, increased availability, and supporting evidence, POCUS has not been 
incorporated into routine clinical practice. For instance, despite both strong evidence from randomized trials 
and recommendations from national patient safety organizations over the past 20 years, ultrasound guidance 
for central line insertion has not been universally adopted, and adoption rates may be worse in the VA 
healthcare system.4-7 Data from a national survey conducted by our group revealed that only 49% of VA 
providers use ultrasound guidance to insert central lines compared to 72-88% of non-VA providers, and VA 
medical coding data revealed only 21% were inserted with ultrasound guidance.8  Barriers to POCUS use 
include limited access to ultrasound machines, lack of training, and provider resistance to change.9, 10  
The VHA Specialty Care Centers of Innovation (SCCOI) and Simulation Learning, Education and 
Research Network (SimLEARN) are partnering to develop a national POCUS training program to teach VA 
providers basic POCUS applications. Starting in October 2016, hands-on POCUS training courses will be held 
at the new SimLEARN National Simulation Center in Orlando.  Training of providers has traditionally consisted 
of a combination of didactics and supervised scanning on live models to learn image acquisition and 
interpretation skills. 
 
The goal of our partnered evaluation is to assess the effect of the VA’s national POCUS training 
program on local implementation of POCUS use. First, we will evaluate the effectiveness of the 2.5-day 
POCUS Training Course on provider POCUS skill acquisition and skill retention.  Second, we will assess the 
frequency of POCUS use by providers before and 6 months after participation in the POCUS Training Course. 
Third, we will survey providers and facilities to identify provider and facility barriers to POCUS use.  This 
evaluation plan will allow us to accomplish the following objectives: 
Objective 1.  Evaluate provider skill acquisition and retention, and frequency of POCUS use 
after participation in the POCUS Training Course. Pre- and post-course testing will be used to assess 
acquisition of knowledge and technical skills to perform POCUS exams.  Post-course testing for knowledge 
and skill retention, and frequency of use, will be performed 6 months after the training course.  We postulate 
that immediate post-course knowledge and skills test scores will improve by ≥25% or to a minimum passing 
score of >75% for the majority of providers.  After returning to their facilities, providers that use POCUS 
frequently (>3 times/week) will have better skill retention (reduction in 6-month post-course test scores <10%) 
after 6 months vs. providers that use POCUS infrequently (<3 times/week) who will have worse skill retention 
(reduction in 6-month post-course test scores >10%). 
Objective 2.  Determine the effect of the POCUS Training Course and implementation facilitation 
on facility-level frequency of POCUS use.  Facilities with providers that infrequently use POCUS will be 
eligible to participate in the POCUS Training Course.  However, not all facilities will be able to be 
accommodated in the 1st year of the training program, requiring a facility waiting list. Facilities with providers 
participating in the POCUS Training Course will be compared to wait-listed facilities with regard to frequency of 
POCUS use. Using the Brief Provider POCUS Survey, frequency of POCUS use by providers in both 
participating and wait-listed facilities will be compared.  Additionally, coding data will be reviewed to assess 
frequency of procedures performed with and without imaging guidance and procedural complication rates 
comparing the two groups of facilities. We postulate that facilities with trained providers will have a higher 
frequency of POCUS use to guide bedside procedures (>10%) and relatively lower procedural complication 
rates compared to wait-listed facilities over the same time interval. 
Objective 3.  Determine provider and facility-level barriers and facilitators to POCUS use. 
Provider and facility-level barriers will be assessed using 3 tools: Provider POCUS Survey (trained providers), 
Brief Provider POCUS Survey (wait-listed and participating facilities), and Facility POCUS Survey (all facilities). 
Differences in barriers reported and their relationship to frequency of POCUS use will be compared.  We 
postulate the frequency of POCUS use by providers will be inversely proportional to the number of provider- 
and facility-level barriers reported.  Further, providers at facilities where barriers to POCUS are relatively low 
will have greater skill retention after 6 months. 
Specific Aims
Page 2
Contact PD/PI: Soni, Nilam Jayant

2. Research Plan 
Background:   
Medical error is estimated to be the third leading cause of death in the United States.  
Implementation gaps exists for integration of existing technologies to intercept and record medical errors, 
rescue patients when errors occur, and reduce the frequency of preventable errors.9  Point-of-care 
ultrasonography is one such technology that can reduce the frequency of preventable errors but has been slow 
to integrate into patient care.    
Point-of-care ultrasonography improves patient care.  The advent of compact, portable ultrasound 
devices over the past 25 years has given rise to point-of-care ultrasonography (POCUS) – real-time use of 
ultrasound by clinicians at the bedside to guide the evaluation and management of patients.  POCUS can be 
used to guide invasive bedside procedures, or perform goal-directed diagnostic examinations. Use of POCUS 
in clinical care has been shown to expedite diagnostic work-ups and reduce the number of required tests, 
especially computed tomography (CT) scans that are costly and harmful through cumulative radiation 
exposure; reduce the risk of procedure-related complications, especially for central line placement; and reduce 
the risk of failed procedures.1, 2, 10-13 
 
POCUS is underutilized by providers in current clinical practice.  POCUS has changed the 
standard of care for certain conditions, but gaps in adoption are evident.  For example, national safety 
organizations, including the Agency for Healthcare Research and Quality (AHRQ), National Institute for Health 
and Care Excellence (NICE), and Centers for Disease Control and Prevention (CDC), have recommended the 
use of ultrasound guidance to insert central lines since 2001.5, 14, 15 However, approximately half of providers 
do not use POCUS to guide insertion of central lines.4, 8 Data from a recent national survey conducted by our 
group revealed that only 49% of VA providers use ultrasound guidance to insert central lines compared to 72-
88% of non-VA providers, and these numbers likely overestimate the actual frequency of use.16   
Many veterans would benefit from integration of POCUS in VA healthcare.  Based on 2015 
inpatient ICD-9 CM coding data for VA hospitals, a substantial number of veterans would benefit from the 
procedural and diagnostic applications of POCUS. Below are the numbers of Veterans that could potentially be 
impacted by POCUS, although we anticipate the actual numbers to be greater due to underreporting and 
under-coding using ICD-9 codes, especially for procedural complications.  We believe ICD-10 may differentiate 
procedures performed with or without imaging guidance, reducing under-coding compared to ICD-9.       
Diagnostic Applications 
Procedural Applications & Complications 
Heart failure 
110,263 
Central line 
placement 
35,448 
Arterial injury 
4 
Shock 
37,551 
Pneumothorax 
355 
Hypotension 
33,208 
Thoracentesis 
7,423 
Pneumothorax 
295 
Joint effusion 
27,026 
Paracentesis 
7,130 
Hemoperitoneum 
29 
Urinary retention 
20,929 
Bowel injury 
167 
Pneumonia 
13,794 
Lumbar 
puncture 
2,477 
Subarachnoid 
bleeding 
8 
Pleural effusion 
15,603 
Subdural bleeding 
15 
Peritoneal free fluid 
10,997 
Arthrocentesis 
1,904 
---- 
--- 
Limited provider training is a barrier to POCUS use in clinical care.  Slow integration of POCUS in 
clinical care has been attributed to the limited number of trained providers.7, 8 Peer institutions, national patient 
safety organizations, and educational accreditation bodies are increasingly pressuring healthcare systems to 
incorporate POCUS training into clinical practice.  The Accreditation Council for Graduate Medical Education 
(ACGME) now requires ultrasound training for at least nine different medical specialties.17  However, most 
practicing providers, including faculty supervising trainees in ACGME-accredited residencies and fellowships, 
are unfamiliar with POCUS use because they have not been trained. Despite the ACGME requirements, only 
25% of accredited internal medicine residency programs reported having a formal POCUS curriculum in 
2013.18  The provider training gap is likely to persist until large-scale POCUS training programs are developed 
and deployed in healthcare systems, such as the National POCUS Training Program for the VHA.19 
The VHA’s Simulation Learning, Education and Research Network (SimLEARN) and Specialty 
Care Centers of Innovation (SCCOI) are collaborating to develop a National POCUS Training Program.  
Frontline VA providers have voiced the need for the VHA to offer POCUS training, and SimLEARN has been 
tasked to bridge this gap. SimLEARN’s primary goal is to develop an effective training program to teach basic 
POCUS applications, and SCCOI’s primary goal is to support SimLEARN’s efforts and facilitate dissemination 
of POCUS training nationally. 
Research Plan
Page 3
Contact PD/PI: Soni, Nilam Jayant

The goals of our proposal are consistent with the goals of SimLEARN and SCCOI.  Our 
evaluation plan will measure the outcomes of the initial implementation of the National POCUS Training 
Program.  First, we will assess the change in provider skill acquisition after participation in the POCUS Training 
Course, and assess skill retention 6 months post-course. Second, we will compare the difference in frequency 
of POCUS use and procedural complication rates between facilities that participated in POCUS training and 
wait-listed control facilities using data reported by providers and system-level coding data.  Lastly, by defining 
the provider and facility-level barriers to POCUS use, our findings will guide our operating partners’ efforts to 
implement POCUS use nationally in the VA healthcare system.     
Implementation of a national POCUS training program fulfills several VA national priorities in 
the VA’s Blueprint for Excellence.  First, Strategy 2 under Theme 1 in the Blueprint for Excellence is to 
“deliver high-quality, Veteran-centered care that compares favorably to the best of private sector in measured 
outcomes, value, access, and patient experience.”20 Successful implementation of POCUS use in the VHA is 
an opportunity for the VHA to provide high-quality, patient-centered care that is equal to or better than the care 
provided in the private sector.  POCUS use can flourish quickly in the VHA because the patient benefits and 
healthcare system incentives are well aligned.  Second, our project also fulfills Strategy 5 under Theme 2 to 
“foster an environment of continuous learning, responsible risk-taking, and personal accountability.”20  By 
encouraging learning of new skills among providers, this project supports the “continuous development of 
clinical, technical and professional skills to assure the highest levels of competency” among those providing 
direct patient care.  Third, this project meets the requirements of Theme 3 to “advance healthcare innovation 
for veterans and the country.”  System-wide implementation of POCUS use in a large integrated healthcare 
system like VHA has not been previously described.  Findings from our project will serve as a national model 
for system-wide implementation of POCUS.  Finally, integration of POCUS use in the VHA meets the Institute 
for Healthcare Improvement’s Triple Aim of “improving the patient experience of care (including quality and 
satisfaction); improving the health of populations; and reducing the per capita cost of health care,” because 
POCUS improves provider and system efficiency, reduces the number of imaging tests, and reduces patient 
risk of harm from radiation exposure or invasive bedside procedures.  
Our evaluation plan is based on the i-PARIHS 
framework for implementation of innovations (Figure 1) 
as modified by Harvey and Kitson.21 This framework was 
chosen because it guides us in the assessment of the 
multiplicity of factors (barriers and facilitators) that can 
impact the success of POCUS implementation. The 
PARIHS framework recognizes that successful 
implementation is the result of the interrelationship of 
evidence, context, and process. The i-PARIHS framework, 
in contrast, recognizes the role of facilitation in the 
implementation process as well as the role of the 
innovation recipient as they act within the local 
organizational and system context.21  The revised 
framework, acknowledges the role of individual and group 
agency in not only defining how an innovation is 
implemented but also how knowledge is diffused given 
both local and broader organizational contexts. The 
complexity of these interactions at the local (micro) and 
system (macro) levels is spanned by the active facilitation 
of the innovation in each facility which enables innovation spread.21   
The factors for predicting successful implementation of the national POCUS training program 
are based on the i-PARIHS Framework.  The external context to develop the national POCUS training 
program is driven by patient safety organizations, accreditation organizations, and clinical practice guidelines 
recommending adoption of POCUS use based on current evidence as well as third party payers denying 
reimbursement of potentially preventable procedural complications.  The inner organizational context of the 
VHA system includes SimLEARN, SCCOI, and the National Emergency Medicine Field Advisory Group (EM 
FAG).  In addition, providers advocating for the VHA to provide POCUS training are part of this inner context 
because the initiative to develop a VHA POCUS training program was driven by requests from frontline 
providers. The inner local facility context includes assessment of the local barriers and facilitators to POCUS 
implementation that will be performed using the national Facility POCUS Survey completed by Chiefs of Staff 
 
              Figure 1.  i-PARIHS Framework
Research Plan
Page 4
Contact PD/PI: Soni, Nilam Jayant

and Provider POCUS Survey completed by VA providers.   The recipients are the frontline providers that desire 
acquisition of POCUS skills, and the facilitation strategy is the 2.5-day POCUS Training Course, monthly 
conference calls, and SharePoint® website. The assessment tools for the recipients and facilitation strategy 
include the Pre- and Post-course Knowledge and Skills Tests, Provider POCUS survey, and Course 
Evaluation.  Specific questions will assess the provider’s frequency of use and comfort level for POCUS 
applications.21   
Methods (Evaluation Plan): 
Study Design.  A prospective, observational study with wait-listed control facilities will be conducted to: 
1) evaluate the effectiveness of the POCUS Training Course on provider skill acquisition and retention, 2) 
assess the frequency of POCUS use after the POCUS Training Course, and 3) identify provider and facility-
level barriers to POCUS use. Providers from facilities that participated in the POCUS Training Course will be 
compared to wait-listed control facilities with regard to frequency of POCUS use and procedural complications.
Project Preparation.  In April of 2016, a national Facility POCUS Survey was sent to all Chiefs of Staff 
(COS) in the VA healthcare system. Also, the core faculty of the national POCUS training program will gather to 
pilot test the POCUS Training Course curriculum, Pre-/post-course Knowledge and Skills Tests, and the 
Provider POCUS Survey during four pilot courses at the SimLEARN National Simulation Center in April, June, 
July, and August of 2016.  Feedback from these pilot courses will be incorporated to fine tune the course and 
assessment tools.  Logistics will be finalized in preparation for full launch of the POCUS Training Course in 
October of 2016 (FY17) with 2 sessions per month during most months. 
Population and Sampling Plan. Provider Recruitment:  Approximately 200 participants from 15 
different VA medical centers will participate in the 2.5-day POCUS Training Course at the National Simulation 
Center in Orlando.  Fifteen VA medical centers will be selected based on data gathered from the VA Facility 
Survey: 1) COS financial support to pay for travel to Orlando, 2) COS support for release from clinical duties, 3) 
Facilities with providers that infrequently use POCUS, and 4) Availability of ≥2 portable ultrasound machines at 
the facility.  Facilities (n=22 per preliminary data) reporting no POCUS use will be excluded from this first 
training cohort due to high likelihood of providers not having access to a portable ultrasound machine, a barrier 
that has to be surmounted before training ensues.  Approximately 10 providers per course will participate in the 
POCUS Training Course from October 2016 to September 2017. Provider Sites:  Facilities participating in the 
POCUS Training Course will constitute the “intervention” sites while the wait-listed “control” facilities will be 
similar VA facilities based on eligibility criteria above.  After closure of the Facility POCUS Survey in June of 
2016, intervention and waitlist facilities will be selected.  Provider Specialties:  Novice physician providers 
without prior ultrasound training or experience primarily from emergency medicine, internal medicine/hospital 
medicine, and critical care medicine will be invited to participate in the 2.5-day POCUS Training Course. 
Completion of all surveys and skills assessments will be a course requirement disseminated by SimLEARN in 
the invitation letter and emphasized at the start of the POCUS Training Courses.
           POCUS Training Course.  The 2.5-day course will combine focused didactics and supervised hands-
on ultrasound scanning practice with live and simulation models.  A standard provider to core 
Factor 
Who/What 
Assessment Tool 
External Context 

Patient safety organizations

Accreditation organizations

Clinical practice guidelines

Payers
None 
Inner Context: 
Veterans Health 
Administration 

SimLEARN

Specialty Care Centers of Innovation

National Emergency Medicine Field
Advisory Group

Providers interested in POCUS

Monitor VHA policies and requirements
for POCUS use by team members that
are operating partner leaders or members
of the national Emergency Medicine Field
Advisory Group
Inner Context: 
Local VHA 
facility 

Local barriers and facilitators to
implementation of POCUS

Providers interested in POCUS

Facility POCUS survey

Provider POCUS survey
Recipients 

Providers participating in POCUS
Training Course

Pre-/post-course Knowledge & Skills
Tests
Facilitation 
Strategy 

POCUS Training Course

Monthly conference calls

SharePoint® website

Provider POCUS Survey

Course evaluation
Research Plan
Page 5
Contact PD/PI: Soni, Nilam Jayant

faculty ratio of 3-4:1 will be maintained.  Each course will have a maximum of 10 providers and minimum of 3 
faculty.    
Data Sources and Assessment Tools. Here is a summary of the data sources and assessment tools. 
Assessment Tools Source 
Data Elements 
Timing 
VA Facility POCUS 
Survey 
Chiefs of Staff/ 
Service Chiefs 
1. Provider and facility-level barriers
Pre-course and 18 
months  
POCUS SkillsTest 
Course 
Participants 
1. Image acquisition skills
2. Image interpretation skills
Pre-course, immediately 
and 6 months post-course 
POCUS Knowledge 
Test 
Course 
Participants 
1. POCUS knowledge
Pre-course, immediately 
and 6 months post-course 
Provider POCUS 
Survey 
Course 
Participants 
1. Frequency of POCUS use
2. Self-efficacy
3. Procedural complications
4. Provider and facility-level barriers
Pre-course, 6 months and 
18 months post-course 
Brief Provider 
POCUS Survey 
Providers at both 
participating & 
wait-listed 
facilities 
1. Frequency of POCUS use
2. Self-efficacy
3. Procedural complications
Pre-course and 18 
months 
Coding Data 
Review 
VHA Corporate 
Data Warehouse 
1. Frequency of POCUS use for
procedure guidance
2. Procedural complications
Pre-course, 18 months 
Course Evaluation 
Course 
Participants 
Feedback on training course: 
1. Lectures
2. Hands-on training sessions
3. Faculty
Immediately post-course 
Monthly 
Conference Calls 
All Course 
Participants and 
Faculty 
Narrative summary of: 
1. Provider and facility-level barriers
2. POCUS training course feedback
Every 1 month 
SharePoint® 
website 
All Course 
Participants and 
Faculty 
Narrative summary of: 
1. Provider and facility-level barriers
Open access for all 
course participants and 
faculty 
Pre-course Assessments.  VA Facility POCUS Survey: This survey was disseminated in April 2016 to 
COS’s by the HAIG (Healthcare Analysis and Information Group) to gather baseline data on the use of 
POCUS, assess provider training needs, and identify potential barriers/facilitators to POCUS use at individual 
facilities. The survey will be repeated at 18 months.  Pre-course Provider POCUS Survey:  Prior to starting the 
POCUS Training Course, providers will complete the Pre-course Provider POCUS Survey using REDCap™ 
survey software.  The Pre-course Provider POCUS Survey will be available only to registered providers.  The 
survey should be completed before arriving at the National Simulation Center, but staff will ensure completion 
by all providers before starting the course. This survey will assess the pre-course frequency of POCUS use, 
known procedural complications, provider self-efficacy (i.e., confidence in POCUS), and provider and facility-
level barriers to POCUS use. Provider demographic data will also be gathered 
(gender, age, primary specialty, practice setting, percentage of time providing direct clinical care).  The 
provider questions were developed by POCUS subject matter experts and education specialists, and self-
efficacy scales are based on published scales.22 Pre-course Knowledge and Skills Tests (Appendices 8 and 9): 
To assess baseline knowledge and skills, providers will take an electronic Pre-course Knowledge Test and 
hands-on Pre-course Skills Test using standardized checklists. The tests are based on similar tools used by 
national professional society POCUS training courses and input from our multidisciplinary group of core faculty. 
Clinical knowledge relevant POCUS applications, image acquisition, and image interpretation skills will be 
assessed. The Skills Test will be administered by the same core faculty to ensure consistency in grading. 
Course Evaluation:  A standard course evaluation survey administered by Employee Education System (EES) 
will gather feedback on the course structure, facilities, and faculty.  Brief Provider POCUS Survey:  This survey 
will be administered by the HAIG (Healthcare Analysis and Information Group) to gather facility-level 
comparison data from physicians specializing in emergency medicine, critical care medicine, and internal 
medicine/hospital medicine that are not participating in the POCUS Training 
Research Plan
Page 6
Contact PD/PI: Soni, Nilam Jayant

Course, but practicing at both participating and wait-listed facilities. This survey will assess providers’ 
frequency of POCUS use, known procedural complications, provider self-efficacy (i.e., confidence in POCUS), 
and barriers to POCUS use.  Coding Data Review:  Using Coding Data from the VHA Corporate Data 
Warehouse, we will assess frequency of procedures performed with and without imaging guidance and rates of 
specific procedural complications. The rationale to focus on procedures is 1) POCUS diagnostic applications 
are not currently coded, 2) Procedural coding is relatively more specific, and 3) Reduction on procedural 
complications has the strongest evidence base. 
Post-course Follow-up.  Monthly Conference Calls:  After participation in the POCUS training course, 
providers will be invited to participate in monthly conference calls led by one of the core faculty.  The 
conference calls will be unstructured discussions to address technical questions, gather feedback on provider 
and facility barriers encountered to POCUS use in clinical practice, and discuss potential short-term and long-
term solutions to overcome barriers.  Conference calls will be recorded and transcribed.  Faculty will also keep 
notes.  These qualitative data will be reviewed and presented in the final report to inform our operating partners 
and the VHA about provider experiences that will be important for system-wide 
implementation.  SharePoint® website:  A website will be established on the VA intranet for the POCUS 
Training Program by SimLEARN to facilitate dialogue between faculty and providers, and amongst providers.  
Documents from the SharePoint® website will be archived and synthesized for inclusion in the final project 
report. 
Post-course Assessments.  VA Facility POCUS Survey :  This survey will be repeated 18-months into 
the grant period (April 2018) to assess for changes in POCUS use, provider training needs, and barriers/
facilitators to POCUS at individual facilities.  Post-course Provider POCUS Survey:  The Post-course Provider 
POCUS Survey will contain additional questions about facilitation (monthly conference calls, SharePoint 
website) compared to the pre-course Provider POCUS survey.  The post-course survey will be distributed 
electronically to providers at the time of the 6-month post-course Skills and Knowledge Test and at 18 months.  
Post-course Knowledge and Skills Test (Appendices 8 and 9):  An immediate Post-course Knowledge and 
Skills Test to assess skill acquisition and a 6-month Post-course Knowledge and Skills Tests to assess skill 
retention will be conducted.  The same Knowledge and Skills Test templates will be used pre- and post-course.  
However, the 6-month Post-course Skills Test will be administered remotely using REACTs™ simulation 
services, a program designed specifically for remote ultrasound training. REACTs™ will allow the faculty to see 
the ultrasound probe position on the body by camera while simultaneously seeing the ultrasound images on the 
screen.  Paid volunteers that may be administrative personnel or non-participating providers will be used for the 
6-month Post-course Skills Test.  Each patient volunteer will receive a $50 incentive each time they volunteer 
for a Post-course Skills Test.  If REACTs™ is not available, an alternative strategy will be to use Microsoft 
Lync® to videoconference.  Brief Provider POCUS Survey:  The HAIG (Healthcare Analysis and Information 
Group) will be repeat administration of the Brief Provider POCUS Survey to providers at both participating and 
wait-listed facilities after 18 months to asses change in frequency of POCUS use, known procedural 
complications, provider self-efficacy (i.e., confidence in POCUS), and barriers to POCUS.  Coding Data 
Review:  Coding data from facilities of providers that participated in the POCUS Training Course and wait-listed 
facilities will be reviewed at 18 months post-course.
Data Collection Procedures:  The Facility POCUS Survey and Brief Provider POCUS Survey have been 
developed by the HAIG (Healthcare Analysis and Information Group) who will administer the survey and collect 
survey responses.  The Pre-course and Post-course Provider POCUS Surveys and POCUS Knowledge and 
Skills Tests will be administered and collected using REDCap™ survey software available through VA servers.  
The Course Evaluation will be administered by the Employee Education System using the Talent Management 
System (TMS) at the conclusion of the POCUS Training Course before providers depart the National 
Simulation Center. Coding data from the VHA Corporate Data Warehouse will be collected and analyzed with 
SAS.  Field notes collected during the monthly conference calls led by core faculty and from the SharePoint® 
website will be analyzed qualitatively.  
Data Analysis: 
Objective 1.  Evaluate provider skill acquisition and retention, and frequency of POCUS use 
after participation in the POCUS Training Course. The statistical approach used for this specific aim is 
hierarchical repeated measures (mixed effects) analysis.  The repeated learner assessments collected for this 
aim have a natural hierarchical structure with level 1 being the repeated assessment, level 2 being the 
individual learner, and level 3 being the facility/organization with which the individual learner is associated.  
Research Plan
Page 7
Contact PD/PI: Soni, Nilam Jayant

Level 1 will capture within subject variation while level 2 captures the between subjects variability, and level 3 
captures the between organizations variability.  Our analyses will enable us to identify key determinants 
associated with our outcomes and our estimated mixed effects models will have the following general form: 
〖E(Y(i,j,k))=f -1 (α+X(i,j) β + Z(j) γ + μ(j) + W(k)η + ε(i,j,k)) 
Regression models will be estimated using the PROC MIXED procedure in SAS 9.4. We will conduct 
diagnostic tests to ensure appropriate model specifications and we will also estimate intraclass correlation 
coefficients relating the percent of variation in the assessment score explained by individual learner 
characteristics (demographics) and by organizational-level factors. 
Objective 2.  Determine the effect of the POCUS Training Course and implementation facilitation 
on facility-level frequency of POCUS use. The frequency of POCUS use by each provider and by each 
facility would ideally be measured quantitatively using encounter or coding data.  Capture of POCUS usage is 
an active area working groups are addressing.  In the regression model for this objective, the outcome variable 
will be the difference in self-reported frequency of use by providers from the Brief POCUS survey administered 
by the HAIG at 0 and 18 months. Predictor variables will include the vector of organization-level barriers and 
facilitators, whether the facility participated in POCUS training or was wait-listed, and a variable that 
categorizes the facility’s frequency of POCUS use at the start of the study (e.g., 0-25%, 26-50%, etc.).   
Objective 3.  Determine provider and facility-level barriers and facilitators to POCUS use.    
Descriptive Analysis:  The Facility POCUS Survey will be analyzed for the following: number of responding 
VHA facilities that use POCUS; reasons why facilities are not using POCUS; for those facilities using POCUS 
the following: how POCUS is specifically being used (both diagnostic and procedural applications), what new 
POCUS applications facilities would like to incorporate in the future, the clinical areas that are using POCUS, 
the specialties of the providers performing POCUS including whether trainees use it and who trains them; do 
local policies or practice guidelines exist for POCUS, is there a formal credentialing process for POCUS and/or 
a competency evaluation; is a quality assurance process for POCUS in place; how is workflow documented 
and captured; and how are providers trained in POCUS.  These data will be used to guide the implementation 
strategies to be used in the monthly conference calls and on the SharePoint website. The barriers to POCUS 
use in the participating and wait-listed facilities will be captured in the Brief Provider POCUS Survey and VHA 
Facility POCUS Survey that will both be repeated at 18 months.  Categorical variables will be compared using 
chi-squared test and continuous variables using t-test or Mann-Whitney if data are not normally distributed.   
Relevant variables will be used in the models described in Objectives 1 and 2 for the facility level.  Qualitative 
Analysis:  In addition to the descriptive analysis of the Brief Provider POCUS Survey and the VHA Facility 
POCUS Survey, the core faculty will interface with providers during monthly conference calls to create field 
notes to record the barriers/facilitators to POCUS implementation. These activities will also be recorded to 
facilitate verification of field notes. Dr. Soni will maintain a database of the barriers and facilitators.  He will 
group them first by known themes and add to the list of themes as new barriers and facilitators are revealed.  
The investigative team will group these barriers that should be addressed at a national policy level vs. those 
that should be addressed at a local level using active facilitation.  This qualitative description of barriers and 
facilitators will be part of the final report to SimLEARN and SCCOI, 
Variable Definition 
Project Data Elements 
Y(i,j,k) 
learner assessment score for learner i 
within organization j, at time k 
Post-course and 6-month Knowledge and Skills Test 
scores, Self-efficacy, Self-reported frequency of 
POCUS use 
X(i,j) 
vector of learner characteristics 
Provider demographics (age, post-graduate year, 
specialty), Pre-course knowledge/skills/self-efficacy 
Z(j) 
vector of organization-level variables 
Barriers/facilitators identified from Facility POCUS 
Survey, Provider POCUS Survey 
μ(j) 
organization-level random intercept 
N/A 
W(k) 
time element 
W = 0 post-course, W=1 at 6-month assessment 
ε(i,j,k) 
error term 
N/A 
f(∙) 
functional form for f(∙) will be determined 
by the data generating process of the 
learner assessment score 
N/A 
Research Plan
Page 8
Contact PD/PI: Soni, Nilam Jayant

but will also be relayed in real-time as new barriers are identified.  Solutions to overcoming barriers described 
by facilities will be coded as well.   
Project Deliverables.  We will provide a narrative summary of our project’s findings to our operating 
partners, SimLEARN and SCCOI, and make the report available to all VHA facilities.  The final report will be a 
narrative summary that will include four sections: 1. A report of effectiveness of training on provider skill 
acquisition and retention.  This report will guide SimLEARN’s ongoing training courses, and provide data for 
SCCOI to plan its facilitation strategy for system-wide POCUS implementation through its regional training 
centers; 2. A report of frequency of POCUS use by providers before and after participation in the POCUS 
Training Course, as well as any relationships revealed between frequency of use, skills retention, and/or 
barriers/facilitators to POCUS use.  This report will provide insight to the overall effectiveness of the POCUS 
Training Course based on an intermediate outcome, frequency of POCUS use; 3.  A report of barriers and 
facilitators to POCUS implementation per providers and facilities. This report will also synthesize all available 
data about potential or proven solutions that were shown to be effective during the project period, and that may 
be used to overcome local and system-level barriers to implementation. This data will primarily serve SCCOI in 
its efforts to develop solutions to barriers and facilitate implementation of POCUS use nationally.  
Potential Obstacles and Solutions.  One potential obstacle is use of REACTs™ simulation services 
which requires internet access to perform post-course testing remotely.  Although REACTs™ meets HIPAA 
compliance and other patient privacy requirements, it is not currently an approved software in the VHA.  We 
have applied for approval more than 5 months prior to project launch.  However, if we are unable to utilize 
REACTs™ simulation services for remote post-course skills testing, then Microsoft Lync® that is available 
throughout the VA system is an alternate program. Although survey response rates in the U.S. are steadily 
declining,23, 24 our team has conducted 8 major surveys in the past three years and found this strategy 
produces response rates of approximately 27-30% among patients, and we believe that our results will be 
higher among this group of providers given their individual investment in POCUS training.  To minimize 
provider attrition, our research associate will directly contact providers not responding to email 
communications, and if a provider does not respond to the research associate’s requests, the provider’s 
supervisor will be contacted.  During the POCUS Training Course orientation, the expectation for post-course 
follow-up will be emphasized.    
Research Team and Relevant Experience: 
We have selected a multidisciplinary team of experts in collaboration with our operational partners, 
SimLEARN and SCCOI, to do this evaluation, and our team brings diverse skill sets in administration, 
evaluation, and clinical education to the project.   
The primary evaluation team for this project will be led by Nilam Soni, MD, Mary Bollinger, PhD, MPH, 
Jacqueline Pugh, MD, Jacqueline Pugh, MD, Erin Finley, PhD, MPH, Brian Lucas, MD, MSc, Michael Mader, 
MS, and Erika Bowen, MA.  All team members, except Brian Lucas, are located at the South Texas Veterans 
Health Care Systems (STVHCS) in San Antonio.  Nilam Soni, MD, is the principal investigator for this project, 
and he has been collaborating with SimLEARN and SCCOI to develop the VHA’s National POCUS Training 
Program.  Dr. Soni is an internationally recognized physician educator in POCUS and will serve as the Course 
Director and Assistant Director of Point-of-care Ultrasound Training for SimLEARN.  Mary Bollinger, PhD, MPH 
will serve as a co-investigator for this project.  Dr. Bollinger is an expert in creating geographic and spatial 
models, and has led multiple needs assessments and program evaluations for the VHA.  Jacqueline Pugh, MD 
will serve as a co-investigator and senior mentor for this project.  She is an implementation scientist and the 
Associate Chief of Staff for Research at the STVHCS.   
The evaluation team for this project will be led by Mary Bollinger, PhD, MPH, Jacqueline Pugh, MD, 
Erin Finley, PhD, MPH, Brian Lucas, MD, MSc, Michael Mader, MS, and Erika Bowen, MA.  Dr. Finley is a 
medical anthropologist and health services researcher at the STVHCS who will serve as a co-investigator. Her 
research has focused on mental health concerns among U.S. Veterans, and implementation of evidence-
based practices in VHA healthcare settings.  Dr. Lucas is a research scholar at the White River Junction VA 
Hospital who will serve as a co-investigator for this project.  He has led previous POCUS research projects to 
locally implement POCUS training and assess provider competency, as well as measurement of impact of 
POCUS use on clinical outcomes.   Michael Mader is a statistician with extensive experience in health services 
research data analysis.  Erika Bowen will serve as a research associate for this project, and her research 
interest is assessment of impact of quality improvement programs on learners and healthcare systems. 
The clinical education team will be led by Nilam Soni, MD, Christopher Schott, MD, MS, Jeremy Boyd, 
Research Plan
Page 9
Contact PD/PI: Soni, Nilam Jayant