Summary Report

Reporting for the Adoption Strategies for 2D Barcode Project 

Prepared for

Immunization Services Division

National Center for Immunization and Respiratory Diseases

Centers for Disease Control and Prevention

Prepared by

Deloitte Consulting LLP

September 15, 2015

Summary Report – September 2015
Introduction: Overview of 2D Barcoding and 2DA Pilot

On September 30, 2013, Deloitte Consulting LLP (Deloitte) was contracted by the Centers for Disease 

Control and Prevention (CDC) Immunization Services Division (ISD) to conduct the Adoption Strategies for 2D 

Barcodes Project. This contract built on the work of a previous pilot (Implementation Pilot for Two-Dimensional (2D) 

Vaccine Barcode Utilization) in which Deloitte evaluated the introduction and initial use of 2D barcode scanning 

to record vaccine information at a number of facilities across the United States (U.S). There were relatively 

few 2D barcoded vaccines available at the time of this initial pilot and therefore opportunities to scan 2D 

barcodes to record vaccine data were quite limited. Many 2D barcoded vaccines that were not available at the 

time of the original pilot, including several that are frequently administered outside of a traditional provider 

setting, have since come to market This development provided an opportunity to further understand the 

effects of introducing 2D barcode scanning on recording vaccine data through the current Adoption Strategies 

for 2D Barcodes Project.

Complete and accurate recording of the manufacturing lot number, expiration date, and product identifier of 

an administered vaccine is important for accurate and rapid identification of vaccine products or lots during a 

vaccine recall or disease pandemic event. The small font often used on human readable labels can be 

challenging for health care practitioners to read and interpret and may lead to errors and inefficiencies when

vaccine data are manually entered into electronic systems. Improved data quality can help the public health 

community to determine if vaccine-related adverse events are associated with a particular vaccine lot, which 

could indicate a safety problem with that lot. Improved data quality can also assist health care practitioners in 

identifying patients who have received a vaccine from a recalled lot. Additionally, improved data quality can 

assist health care practitioners in verifying that patients receive the correct vaccine at the right time. Barcodes 

that use 2D symbology may lead to improved data quality. These barcodes have a higher data capacity than 

linear barcodes, allowing the storage of more data elements. A 2D barcode can hold the product 

identification (ID), lot number, and expiration date, while a linear barcode can only hold the product 

identifier. A 2D barcode, unlike a linear barcode, is small enough to place on a vaccine unit-of-use. Further, 

barcode scanning technology can automate the process of recording data about a vaccine lot number, 

expiration date, and product identifier, and as a result, has the potential to improve efficiency and reduce 

errors associated with the manual entry of this information. 

The project focused on three primary topics of interest, which are described here and explored further in this 

report. The topics are as follows:

1. Time Savings

2. Data Quality and Compliance

3. User Experience 

In this document, in order to look at Time Savings, we focus on two primary areas: 1) quantifying the amount 

of time it takes to record vaccine information by scanning 2D barcoded vaccines compared to entering non2D barcoded vaccines through another method, and 2) understanding participant perceptions about time 

savings when using 2D barcode scanning to record vaccine information.

For Data Quality and Compliance, we address a series of evaluation questions around the associations 

between 2D barcode scanning, electronic medical record (EMR) data quality and user (staff member 

administering the vaccine) compliance with 2D barcode scanning. We initially examined the potential impact 

Summary Report – September 2015
of 2D barcode scanning on data quality, defined by the correctness (whether the data element provided was 

both complete and accurate) of lot numbers and expiration dates entered in vaccine administration records. 

Total correctness means that the data elements (lot number, expiration date) were in the EMR extract and 

accurate. A goal of this analysis was to determine correctness of recorded data by comparing lot number and 

expiration data from EMR records with those same elements from reference data files from the 

manufacturers, the Vaccines for Children (VFC) program, the Vaccine Adverse Event Reporting System 

(VAERS), as well as inventory vaccine shipping files from the participating site. To better understand the 

relationship between 2D barcoding and data quality, we performed descriptive and inferential statistics. Based 

on these analyses, we offer recommendations for continued improvement in EMR data quality through the 

use of 2D barcode scanners. Additionally, this report explores the extent to which individuals complied with 

2D barcode scanning protocols. Compliance refers to whether a 2D barcoded vaccine capable of being 

scanned by the facility was actually scanned by the user. We employed both quantitative and qualitative 

analyses on a subset of data that includes a direct indicator of scanning (scan log files of vaccine 

administrations scanned at specific facilities). The analysis included a comparison of facilities observed 

compliance from EMR data to self-reported compliance from answers provided in two online surveys of

users and leaders within facilities. Additionally, we extended our analysis to investigate how compliance and 

perceptions about 2D barcode scanning affect overall data quality and ultimately influence 2D barcode 

scanning adoption by the broader immunization community. 

Finally, to understand User Experience, we examine the implementation of 2D barcode scanning and the 

experience of leaders and users at participating facilities within the current project. This report focuses on: 1)

the experience of the end user during the process of adopting 2D barcodes to record vaccine information, 

and 2) the potential for sustaining 2D barcode scanning after the current project ends. We provide 

information in this report about how participating facilities implemented 2D barcode scanning; user 

experiences while implementing 2D barcode scanning; the types of challenges implementers encountered and 

any approaches they employed to remedy these challenges; and the interest of project participants in 

continuing to scan 2D barcodes to record vaccine information once the project ends. 

Methods: Data Sources, Evaluation Questions and Analyses

Facilities that expressed interest in, but had not enrolled in, a previous CDC 2D vaccine barcode evaluation 

were contacted for recruitment. The Deloitte team conducted additional targeted recruitment of other types 

of facilities such as community vaccinators and pharmacies. Recruiting and enrolling practices was completed 

in multiple stages. Many of the sites were interested in participating in the project and contacted the project 

team to be enrolled. The project team tracked interested sites and once a recruitment survey was available, the 

team sent the survey and completion information to each interested site. Follow-up calls were made to sites 

that did not initially complete the recruitment survey. The project team sought engagement from practices 

that differed across multiple characteristics and expanded the diversity of facility types beyond prior 

barcoding projects. Facilities that fell into these categories were targeted by the project team. Effort to engage 

and enroll these facilities included initial contact and several follow-up calls by the project team

Ultimately, the Deloitte team included facilities if they (1) were likely to use vaccines with 2D barcodes 

scheduled for distribution during the project period, (2) volunteered to scan 2D barcoded vaccines 

administered into their EMRs, (3) agreed to report de-identified EMR vaccine administration data for the 

evaluation, (4) used an EMR to capture vaccine administration data that could be configured to input data 

using a 2D barcode scanner, and (5) agreed to use technology that allowed us to identify if the lot number 

Summary Report – September 2015
and expiration date for a given vaccine administration was entered into the system with a 2D barcode 

scanner. For each participating facility, we provided, installed, and configured corded, handheld image 

scanners with USB interface (i.e., 2D barcode scanners) to scan 2D barcoded vaccines to enter lot number 

and expiration date into EMRs. We also conducted in-person staff training on use of scanners. This 

evaluation was deemed to be public health practice and did not require IRB review.

From the enrolled facilities, the initial aim was to recruit 20 facilities for participation in the Work Flow 

Analysis (WFA). These facilities were purposively selected, with the following three primary criteria of 

interest: (1) Mix of immunizer type, (2) Level of interest expressed by the immunizer, and (3) Volume of 2D 

barcoded vaccines present in the facility. Our team used two sources of data to select an initial set of

candidate facilities to enroll in the WFA as well as back-up facilities: (1) Information from regular status calls 

performed with facility points of contact and (2) Responses from the Recruitment Survey. Recruitment and 

scheduling of WFA practices was fluid in nature with the aim of: (1) Obtaining 20 facilities, (2) Conducting 

the WFA visit on dates when the team member was most likely to maximize observations of actual rather 

than mock scans, and (3) Acquiring a mix of facility types –private practices, public practices, and retail 

pharmacies. Feasibility constraints did exist for performing the WFA which limited our ability to fulfill these 

aims. Specifically, all visits needed to be performed prior to February 2015 to allow for adequate time to 

clean, analyze, and interpret this data for inclusion in the final evaluation reports and other communications 

materials. 

Data Sources

The team used eight sources to capture information about the processes of 2D barcode scanning 

implementation, including the data entry process steps for 2D and non-2D barcoded vaccines, and the 

challenges experienced and approaches individuals within participating facilities used to overcome these 

challenges. In addition, these data sources included information on the characteristics of participating 

facilities, user perceptions of 2D barcode scanning, opportunities for improvement, the likelihood of 

sustainability of 2D barcode scanning in participating facilities, data quality and compliance from EMR

records, and time measurements capturing the length of time for facility staff to record data about vaccines 

administered by scanning 2D barcodes or through manual data entry into the EMR. We provide a summary 

of the types of data collected, the timing of collection, and a brief overview of the analysis for each type of 

data in Table 1. 

Table 1. Data Sources Used to Develop Findings

Data Source Data Collected Method of Data Collection Timing of Collection

1. EMR Data 

Files 

Vaccine name, Manufacturer, 

Expiration Date, EMR Timestamp, 

Vaccination Date, CPT Code, CVX

Code, NDC, Lot Number, Funding 

Source and Vaccine Type (i.e.: Flu)

De-identified vaccine EMR records 

from all non-2D and 2D barcoded 

vaccines was submitted in a delimited 

format by facilities.

• Three phases of 

collection: Base (July 

2013-July 2014), Learn 

(August 2014-

September 2014), and 

Mature (October 2014-

January 2015)

Summary Report – September 2015
Data Source Data Collected Method of Data Collection Timing of Collection

2. Scan Log Data Time Stamp, Scanner Serial Number, 

Barcode Data, Machine Name

Facilities employed one of three 

different procedures to generate an 

indicator of scanning: (1) A native scan 

log (when EMR systems had a built-in 

capability to record whether a vaccine 

record was scanned that could be 

extracted), (2) A scan log installed by 

our team for the purposes of this 

project (captured in a Notepad file), or 

(3) Assurance of a procedure in which 

scanning of 2D barcodes was the only 

option for data entry.

• Throughout project 

observation period 

(August 2014-January 

2015)

3. Reference 

Data Files 

(Manufacturer 

lot lists, VFC 

Program files, 

VAERS files, 

Participant site 

Inventory and 

Shipping files)

Vaccine Name, Lot Number, NDC, 

Manufacturer, Expirations Date, 2D 

Barcode Flag

We received manufacturer files and 

VFC records electronically and 

considered the primary source of 

information. Shipping manifests and 

inventory records included some data 

that were written or transcribed thus

considered secondary. 

• Multiple points 

throughout project

period. Complete files 

by end of project 

(January 2015)

4. Leader 

Experience 

Survey (Leader 

Survey)

Leader feedback on current 

project/2D barcode scanning; 

challenges experienced and solutions; 

ease of integration into workflow/ 

satisfaction; perceived 

accuracy/efficiency of data with 2D 

barcode scanning; sustainability of 

scanning; respondent characteristics

One leader per facilities was identified 

during onsite installation. Sites 

provided contact information. 

Personalized links to the online survey 

were sent to leaders via email. 

• Near end of project 

(January/February 

2015)

5. User 

Experience 

Survey (User 

Survey)

User feedback on current project/2D 

barcode scanning; challenges 

experienced and solutions; ease of 

integration into workflow/ 

satisfaction; perceived 

accuracy/efficiency of data with 2D 

barcode scanning; sustainability of 

scanning; respondent characteristics

Facilities provided contact information 

for two to five users. The user survey 

was administered online using 

Qualtrics. Personalized links were sent 

to users via email.

• Near end of project 

(January/February 

2015)

6. Workflow 

Analysis 

(WFA)

Observations, interviews, time 

measurements of vaccine 

administration process with and 

without 2D barcode scanning; 

challenges experienced and suggestions 

for improvements; practitioner 

characteristics

Team members conducted two-day site 

visits at select facilities and collected 

data through physically observing the 

vaccine administration and recording 

process, conducting in-person staff 

interviews, and timing staff (e.g., 

nurses, nurse / office managers, and 

medical assistants) carrying out specific 

activities around recording data about 

vaccines administered or inventoried. 

• 2 day site visits 

conducted during the 

observation period

(October 2014 – 

February 2015)

Summary Report – September 2015
Data Source Data Collected Method of Data Collection Timing of Collection

7. Site 

Characteristics 

File 

Demographic information on 

participating facilities (including size 

and specialty); technology landscape 

(computer operating system/EMR

system used); vaccines administered

This file was developed during the

recruitment phase and updated 

throughout the project. Our team 

members entered data into this file 

primarily from the Recruitment Survey 

in which candidates projected estimated 

vaccine usage, provided organizational 

information, and reported data specific

to their vaccine administration 

processes. Additional data were 

collected during correspondence with 

participating facility candidates 

(email/telephone), and with data 

obtained through ongoing contacts 

with all facilities

• During recruitment of 

project facilities; 

updated as relevant at 

check-points during 

project and any time 

change identified 

8. Activity Logs Recruitment and Training logs with 

configuration details, initial survey 

responses, demographic information, 

point of contact

The project team tracked interested 

sites. Our team sent a recruitment 

survey and completion information to 

each interested site. Follow-up calls 

were made to sites that did not initially 

complete the recruitment survey. 

• During the recruitment 

process and on-site 

installation visits.

Our team employed a mixed methods approach to answer evaluation questions within each topic, using both 

quantitative and qualitative data collection and analysis techniques. Multiple sources of data were analyzed to 

address the evaluation questions, with a primary reliance on different data sources for each topic. 

Following the previously approved analysis plan, our first step in the analysis process was to examine the 

quality, completeness, and distribution of data we obtained. We calculated descriptive statistics for all 

quantitative data, analyzed qualitative responses to open-ended survey questions, and identified salient points 

from the WFA interviews and observations. Visual comparison of data by number or percentage provided 

information on the level of similarity or difference in responses; however, these comparisons were not 

intended as indicative of statistically significant or meaningful differences unless noted. When we performed 

statistical tests the specific test statistics and associated p-values are noted in text. When we describe patterns 

identified in data analyses in the absence of a test statistic, we did not conduct a specific statistical test. Due to 

skip patterning in the surveys, not all participants received every question; therefore, data presented are for 

responses received to each question. We denote the number of individuals responding to these questions 

within the figure, table, or associated text for reference. Table 2 contains a summary of the analyses our team 

performed and the data sources we used for these analyses to answer each of the evaluation questions. 

Table 2. Analysis Conducted and Data Sources Used to Answer Each Evaluation Question

Evaluation Question Data Sources Analyses Conducted

Time Savings

Summary Report – September 2015
Evaluation Question Data Sources Analyses Conducted

1. Does the use of 2D barcodes to record data 

about vaccine administrations change the 

amount of time it takes to record vaccine 

information? 

• WFA- time 

measurement/ 

observation/interview

• Site Characteristics File

• Descriptive statistics

• Statistical tests

• Qualitative analysis 

2. To what extent does the broader community 

perceive changes in efficiency related to 

recording vaccine administration data? 

• Leader Experience 

Survey

• User Experience Survey

• WFA- observation/ 

interview 

• Site Characteristics File

• Descriptive statistics

• Qualitative analysis 

3. To what extent do perceptions of 2D barcode 

scanning efficiencies for recording data about 

vaccines administered align with 

observations?

• Leader Experience 

Survey

• User Experience Survey

• WFA- time 

measurement/ 

observation/ interview

• Descriptive statistics

• Qualitative analysis

4. What are project participants’ perceptions 

about changes in efficiency related to 

recording data about vaccines into inventory? 

• Leader Experience 

Survey

• User Experience Survey

• Descriptive statistics

• Qualitative analysis

Data Quality and Compliance 

1. To what extent does vaccine administration 

data quality change with the introduction of 

2D barcode scanning?

• EMR Data

• Reference Data

• Descriptive Statistics

• T-Test

2. How does data quality differ between vaccine 

administration records that were scanned and 

those that were not?

• EMR data

• Reference data

• Scan Log Data

• Descriptive Statistics

• T-Test

• Multi-Level Logistic 

Regression

3. Is there variation among facilities in 

compliance with scanning 2D barcoded 

vaccines?

• EMR data

• Reference Data

• Scan Log Data

• Descriptive Statistics

• Multi-level Logistic 

Regression

4. What types of errors in data entry occurred? • EMR data

• WFA Observation

• Leader Experience 

Survey

• User Experience Survey

• Descriptive Statistics

• Qualitative Analysis

5. What are practitioners’ perceptions related to 

data quality and compliance • WFA Observation

• Leader Experience 

Survey

• User Experience Survey

• Descriptive Statistics 

• Qualitative Analysis

User Experience

Summary Report – September 2015
Evaluation Question Data Sources Analyses Conducted

1. What is the experience of the end user when 

adopting 2D barcodes to record vaccine data? • Leader Experience 

Survey

• User Experience Survey

• WFA- interviews/ 

observations

• Site Characteristics File

• Descriptive statistics

• Qualitative analysis 

2. What is the potential for sustaining 2D 

barcode scanning after the pilot project? • Leader Experience 

Survey

• User Experience Survey

• Site Characteristics File

• Descriptive statistics

• Qualitative analysis 

Findings: Participant Characteristics

Respondent and Facility Characteristics

Ninety-four facilities initially enrolled in the project. Eighty-seven of these facilities (93%) remained enrolled 

for the entire duration of the project. These facilities were located in 21 different states, most frequently 

Louisiana (n=15) and Texas (n=14). As shown in Table 3, these facilities were most commonly self-identified 

privately funded vaccinators (i.e., facilities that did not receive the majority of their funding from federal or 

state sources, e.g. Federally Qualified Health Centers) specialized in pediatrics (n=30, 35%), pharmacy (n=13, 

15%), or community health care (n=11, 13%). They most frequently had between one and five physicians 

(n=35, 40%) and used EPIC, Athenanet or EPRN for their EMR system. Less than half of all facilities had a 

direct indicator of barcode scanning, which we refer to as facilities with “scan log” capabilities and describe 

below in further detail. 

Seven facilities did not remain enrolled for the duration of the project. These facilities did not complete the 

project for a variety of reasons, including but not limited to: 

• An inability to provide the necessary EMR extracts for the data analysis

• Complete non-compliance with 2D barcode scanning

• EMR changes/upgrades that were not compatible with scanning configuration

Table 3 includes a description of the characteristics of facilities included in the Leader and User Surveys, 

WFA, Scan log, and all facilities. 

Table 3. Characteristics of Facilities Represented by Leader and User Surveys, Workflow Analysis, 

and Scan Log Data Compared to All Participating Facilities

Characteristic

Leader 

Survey 

(N=67)

n (%)

User 

Survey 

(N=116)

n (%)

WFA 

Facilities

(N=22)

n (%)

Scan Log 

Facilities 

(N=32)

n (%)

All 

Facilities

(N=87)

n (%)

Facility type

Privately funded vaccinators 47 (70) 85 (73) 15 (68) 19 (59) 54 (62)

Summary Report – September 2015
Characteristic

Leader 

Survey 

(N=67)

n (%)

User 

Survey 

(N=116)

n (%)

WFA 

Facilities

(N=22)

n (%)

Scan Log 

Facilities 

(N=32)

n (%)

All 

Facilities

(N=87)

n (%)

Publicly funded vaccinators1 13 (19) 18 (16) 4 (18) 8 (25) 15 (17)

Retail pharmacy chain 5 (8) 9 (8) 3 (14) 0 (0) 10 (12)

Independent pharmacy 2 (3) 4 (3) 0 (0) 0 (0) 3 (3)

Mass Vaccinator2 0 (0) 0 (0) 0 (0) 5 (16) 5 (6)

Facility specialty

Private Practice – Pediatrics 25 (37) 48 (41) 9 (41) 13 (41) 30 (35)

Community Health Care 9 (13) 19 (16) 1 (5) 0 (0) 11 (13)

Hospital 8 (12) 10 (9) 4 (18) 1 (3) 8 (9)

Pharmacy – Retail Chain/ Independent 7 (11) 13 (11) 3 (14) 0 (0) 13 (15)

Federally Qualified Health Center 

(FQHC) 6 (9) 12 (11) 2 (9) 3 (9) 6 (7)

Remaining specialties- Health Dept., 

Family medicine, Mass vaccinator 12 (18) 14 (12) 3 (13) 15 (47) 19 (22)

Number of physicians

0 11 (16) 13 (11) 3 (14) 5 (15) 21 (24)

1 to 5 29 (43) 56 (48) 8 (36) 26 (76) 35 (40)

6 to 14 11 (16) 19 (16) 4 (18) 2 (6) 14 (16)

15 or more 16 (24) 28 (24) 7 (32) 1 (3) 17 (20)

EMR Vendor

EPIC 28 (42) 40 (35) 8 (36) 8 (25) 29 (33)

Athenanet 14 (21) 29 (25) 2 (9) 16 (50) 16 (18)

EPRN 5 (8) 9 (8) 3 (14) 0 (0) 10 (12)

Allscripts Pro EHR 4 (6) 4 (3) 4 (18) 0 (0) 6 (7)

Greenway 4 (6) 5 (4) 2 (9) 0 (0) 4 (5)

eClinical Works 2 (3) 2 (2) 0 (0) 0 (0) 3 (3)

Insight 2 (3) 4 (3) 2 (9) 1 (3) 2 (2)

PrimeSuite 2 (3) 7 (6) 0 (0) 0 (0) 2 (2)

Other systems 6 (9) 16 (14) 1 (5) 7 (22) 15 (17)

Scan Log

Native (EMR functionality) 14 (21) 29 (25) 2 (9) 17 (53) 21 (24)

Installed for the purpose of this project 11 (16) 11 (9) 0 (0) 10 (31) 12 (14)

None 42 (63) 76 (66) 20 (91) 5 (16)3 54 (62)

Scan Log Facilities

A direct indicator of barcode scanning was available in 32 of the 87 facilities (37%), referred to as scan log 

facilities. These facilities differed from the other facilities enrolled in this project in a few notable ways. The 

scan log facilities were concentrated among private practices and mass vaccinators; no pharmacies had scan 

log capabilities (Table 3). Due to the internal configuration capabilities, scan logs were concentrated among a 

smaller subset of EMR systems than other facilities in the project. Athenanet was the dominant EMR in the 

scan log sample due to the native ability to record scanning. Similar to the distribution of EMRs across all 

participating facilities, EPIC was a common EMR system among the facilities with a scan log. Facilities with a 

1 Facilities that receive most of their funding from federal or state sources (e.g., public health departments, Federaly Qualified Health 

Centers, Military site) 2 Any facilities that dropped out of the current project and the mass vaccinator (VaxCare) were not included in the pool eligible for 

user experience data collection. VaxCare chose to opt out of the survey pool after being part of the instrument pre-testing process, 

based on staffing and vaccine administration processes that varied greatly from other facilities included in the project. 

3 The Mass Vaccinator was only able to enter data through 2D barcode scanning.

Summary Report – September 2015

 
scan log were most frequently small facilities, with 76% (n=26) of the scan log facilities having between 1 and 

5 physicians. Approximately 300,000 records came from facilities with a direct indicator of scanning (scan log 

facilities).

Survey and WFA Participants

Most frequently, one leader and between one and three users per participating facility submitted responses to 

the experience surveys. Sixty-seven of 81 leaders invited to participate submitted a complete survey4, resulting 

in a response rate of 83% for the Leader Survey. Of the 210 users who received a survey, 116 submitted a 

complete survey, resulting in 55% User Survey response rate. We received at least one response from users in 

63 of the 81 eligible facilities (78%). 

Twenty-two facilities participating in the current project received on-site visits and participated in additional 

data collection (interviews, time measurements, and observation) through the WFA. Of the 22 facilities 

participating in the WFA, 18 Leader Surveys (from 18 WFA facilities) and 28 User Surveys (from 14 WFA 

facilities) were received. 

The characteristics of facilities represented by Leader and User Surveys, as well as the WFA, did not differ 

meaningfully from the characteristics of all facilities participating in the project (N=87) (Table 3). The 

facilities represented by survey respondents and WFA participation, as well as the project facilities overall, 

most frequently were private practice pediatric offices with between one and five physicians. These facilities 

most frequently reported using EPIC or Athenanet for their EMR system. A majority (63%) of facilities 

participating in the project did not have a mechanism in place for flagging when staff scanned in a vaccine 

data element (i.e., a scan log), either through their EMR functionality or installed by our team for the 

purposes of this project. Participating facilities typically had between two and six 2D barcode scanners 

installed to record vaccines administered, with a range of 1 to 38 scanners across all facilities (the mode was 

three scanners for all facilities).5

One aim of administering a Leader Survey and User Survey was to capture the most informed point of view 

within facilities on specific topics. For example, our expectation was that leaders could provide overarching 

information about the systems used within facilities, workflow of staff, etc. The self-reported role of leaders 

aligned with this expectation, with the majority of leaders noting that they directly supervise individuals who 

record data about vaccines administered (n=44, 65%) and inventory data (n=34, 51%), and that they establish 

policies related to vaccine data collection (n=43, 64%). We surveyed users, on the other hand, to learn about 

the specific experience of using 2D barcode scanners to record vaccine data. The self-reported roles of users 

aligned with this expectation, with most noting that they have personally scanned 2D barcoded vaccines 

administered (n=88, 86%) or into inventory (n=33, 53%). 

Characteristics of respondents to both surveys and the WFA participants in facilities contributing time 

measurement data (n=16) were often fairly similar to each other with respect to job title, years of experience 

recording vaccines administered, and being more likely to have previous experience with barcode scanning in 

non-medical settings (compared to medical settings). Survey respondents frequently self-identified as nurses 

(users n=53, 46%; leaders n=15, 29%), with reported job experience ranging from less than one year to over 

4 A complete survey required participants to reach the end of the survey and submit their survey responses. 5 Additional details about scanners installed at participating project facilities are provided in the Technical Documentation Report.

Summary Report – September 2015

 
30 years of experience. More than half of User Survey respondents reported having experience prior to the 

project with barcode scanning in a non-medical setting (n=77, 66%), with fewer having previous barcode 

scanning experience in a medical setting (n=33, 28%). Nearly all users indicated either a “high” (n=67, 58%) 

or “moderate” (n=41, 35%) level of comfort with barcode scanning technology (Table 4). 

Table 4. Frequency distribution of respondent characteristics from Leader and User Surveys and 

WFA Facilities6

Respondent Characteristics

Leader survey 

(N=67)

n (%)

User survey 

(N=116)

n (%)

WFA Participants¶

(N=71) 

n (%)

Number of years at organization

Up to 1 year 3 (5) 5 (4) 21 (30)

2-3 years 23 (34) 40 (35) 16 (23)

4-8 years 11 (16) 31 (26) 18 (25)

9+ years (range was 9 – 35 years) 30 (45) 40 (35) 16 (23)

Number of years recording vaccines administered

Up to 1 year N/A§ 21 (21)* 13 (18)

2-3 years N/A 31 (30)* 15 (21)

4-8 years N/A 25 (25)* 19 (27)

9+ years (range was 9-32 years) N/A 25 (25)* 24 (34)

Title 

Nurse 15 (29) ‡ 53 (46) 48 (68)

Medical Assistant/Aide 0 (0) 32 (28) 16 (23)

Pharmacist 9 (17) 14 (12) 0 (0)

Pharmacist Assistant 0 (0) 3 (3) 0 (0)

Administrator, Admin Assistant 20 (38) 3 (3) 0 (0)

Physician 2 (4) 4 (3) 0 (0)

Nurse Practitioner 6 (12) 2 (2) 6(8)

Physician Assistant 0 (0) 1 (1) 0 (0)

Other† 0 (0) 4 (3) 1 (1)

Experience with barcode scanning technology in 

medical setting

Yes N/A 33 (28) 33 (46)

No N/A 83 (72) 38 (54)

Experience with recording 2D barcoded vaccine data 

in non-medical setting

Yes N/A 77 (66) 45 (63)

No N/A 39 (34) 26 (37)

Training received on the use of 2D scanners

From a barcoding project team member N/A 59 (51) 30 (42)

From a colleague N/A 43 (37) 42 (59)

Other§ N/A 5 (4) 2 (3)

No training received N/A 13 (11) 2 (3)

Level of Comfort with Barcode Scanning Technology

High N/A 67 (58) N/A

Moderate N/A 41 (35) N/A

Low N/A 8 (7) N/A

*Due to skip patterns in the User Survey, 102 respondents received this question; percentage provided is based on this denominator.

† Examples of “Other” responses provided for title include: Nurse Supervisor, Clerical, Clinical Manager and Retired Nurse.

6 Table 4 includes practitioner characteristics for the 16 WFA facilities that are included in the time analysis.

Summary Report – September 2015

 
‡“Title” was asked as open-ended question on Leader Survey (with n=52 responses received, used as denominator when calculating 

percentage in this category) and closed-ended question on User Survey, therefore response options do not align fully.

§ Examples of “Other” responses for training received included: already knowing how to use the scanner; another nurse providing 

training; brief training provided from an unknown source; and training provided prior to this project.

¶ Includes practitioner characteristics from the 16 WFA facilities that provided time measurement data.

Time Savings

The first topic explored focused on assessment of time savings with the introduction of 2D barcode scanning 

at participating facilities through exploration of four evaluation questions. Broadly, these questions focus on

quantifying the amount of time it takes to record vaccine information when scanning 2D barcoded vaccines 

compared to entering non-2D barcoded vaccines through another method and understanding participant 

perceptions about time savings when using 2D barcode scanning to record vaccine information. 

The findings provide evidence of measured time savings and positive perceptions of time savings from 

project participants with the use of 2D barcode scanning to record vaccines administered. However, further 

discussions are needed among members of the vaccine community to determine whether the time savings 

identified in this project—an average of 3.4 seconds saved with the use of 2D barcode scanning, not 

controlling for other factors – represents a meaningful difference. The combination of time measurements, 

survey, and write-in responses by both leaders and users at participating facilities, as well as feedback 

provided through WFA observations and interviews provide a more extensive look at the concept of time 

savings and efficiencies with the use of 2D barcode scanning.

Findings by each of the four evaluation questions are presented below. 

Question 1: Does the use of 2D barcodes to record data about vaccine administrations change the 

amount of time it takes to record vaccine information? 

The WFA technician from our team collected 1,026 time measurements from 16 of the 22 WFA facilities 

visited. Six (of the 22) facilities that participated in the WFA were excluded from the time measurement 

portion of data collection because of one of the following reasons: 1) the facility was selected as a “pilot” for 

the WFA to pre-test the proposed methodology and data collection procedures, 2) the facility had no or a 

very small volume of vaccines administered during the WFA visit, or 3) the facility had not yet reached a 

steady state with their new 2D barcode scanning data entry process and faced issues with staff not using the 

scanners consistently (i.e., not knowing how to scan properly or scanners not configured properly for 2D 

barcode scanning). These time measures were almost evenly divided between 2D barcoded vaccines for 

which data were scanned into the system (n=495, 48.2%) and vaccines that were not 2D barcoded and 

therefore entered using the facility’s traditional method (n=531, 51.8%). On average, 31 (SD= 7) instances of 

data entry using 2D barcode scanning were observed per facility. 2D barcode scanning observations ranged 

from 18 to 46 per facility. An average of 33 (SD= 9) instances of data entry using the traditional method of 

data entry were observed per facility. Traditional data entry methods ranged from 14 to 54 per facility. The 

average time to record vaccine administration data using a 2D barcode scanner was 6.86 seconds (N=495, 

SD=8.14). The average time to record vaccine administration data for non-2D barcoded vaccines was 10.30 

seconds (N=531, SD=8.07). Scanning 2D barcodes was, on average, 3.44 seconds faster than recording data 

from vaccines without 2D barcodes using the traditional method. This difference was statistically significant 

at an alpha level of 0.01 (t(1,024)=30.91, p=0.001).

Summary Report – September 2015
There are several other factors beyond the method used to enter data that can affect the time it takes staff to 

record vaccine data, such as previous experience staff have recording data about vaccines administered or the 

type of EMR used in the facility. To understand whether a difference in the mean amount of time to record 

data about vaccines administered still exists when these other factors are accounted for, our team employed a 

hierarchical linear regression model. This model included a random intercept for nurse and facility to account 

for the non-independence in facility context and differences in staff data entry process as well as variables to 

account for the prior experience of staff in scanning barcodes to record data (inside and outside of medical 

setting) and number of years recording vaccine administration data.7 Even after accounting for potential 

confounding factors, time to enter vaccine administration data was significantly less when using a 2D barcode 

scanner than traditional data entry methods (χ2=40.92, p<.001). On average, while controlling for potential 

confounders, 2D barcode scanning was associated with a 0.71 second decrease in time to record vaccine 

administration data.

Beyond the aggregate comparison in time to record vaccine data between the two types of data entry, our 

team conducted several univariate analyses to ascertain variations that may emerge within the context of the 

scanning. First, we examined whether the magnitude or direction of the difference seen between using 2D 

barcode scanning to record vaccine data and traditional methods of data entry varies by facility. Variation was 

found in both the magnitude and direction of the mean difference in time to record vaccine data using 2D 

barcode scanning compared to the traditional method. Twelve of the 16 facilities experienced a decrease in 

average time to record vaccine administration using 2D barcode scanners for 2D barcoded vaccines. In eight 

of these 12 instances, the difference detected was statistically significant (α=0.05). These reductions in time 

ranged from 1.3 seconds to 11.22 seconds (with a median of 5.1 seconds). In four of the facilities it took 

longer, on average, for staff to scan 2D barcoded vaccines than to enter data from non-2D barcoded vaccine 

using traditional methods. Increases ranged from .39 seconds to 9.26 seconds (with a median of 1.37

seconds). In only one instance, was the difference in mean values statistically significant (α=0.05).

Further analyses investigated facility-specific and staff-specific factors with the potential to influence the time 

it takes staff to record data about vaccines administered, beyond the type of recording method used (e.g., 2D 

barcode scanning or traditional method). Factors included in these analyses included the vaccine type, 

manufacturer, EMR type, and specific characteristics of the individuals recording the data. Through these 

analyses, substantial variation was found in the time to record vaccine administration data across both facility 

and staff-specific factors were also identified. Prior scanning experience outside a medical setting was 

associated with a statistically significant effect on time to record vaccine administration data (χ2=203.71, 

p=.001). Counterintuitively, prior scanning experience outside a medical setting was related to longer data entry 

time. Years at the facility was also associated with a statistically significantly effect on time to record vaccine 

administration data (χ2=69.00, p=.001). Practitioners with a longer tenure at the facility took longer to enter 

vaccine administration data.

From the WFA it was observed that the time required to record data for vaccine administration by scanning 

the 2D barcode at a given facility was influenced by a variety of factors and cannot be attributed to a single 

variable. However, it became evident that facilities in which staff took longer to scan encountered one or 

more of the following challenges: 1) inconsistent response from the scanner; 2) faint or smeared barcodes; 3) 

limited volume of 2D barcoded vaccines; 4) lot number and/or expiration date incorrectly entered with 2D 

7 Several model specifications were tested and compared using a Likelihood Ratio Test. Results presented here represent the final 

model selected based on these tests, model parsimony, and theoretical assumptions.

Summary Report – September 2015

 
barcode scanning; 5) EMR software issues; or 6) lack of 2D barcode training. These challenges typically 

introduced process inefficiencies such as requiring several scan attempts before getting to a successful one,

reverting to manual data entry, or switching back and forth between manual and 2D barcode scanned data 

entry.

Question 2: To what extent do project participants perceive changes in time savings related to 

recording vaccine administration data? 

Survey respondents were asked about their perceptions related to time savings with the introduction of 2D 

barcode scanning to record vaccines administered. One question was included in the Leader Survey and three 

questions in the User Survey to assess various aspects of time savings and changes in time to record vaccines 

administered with use of 2D barcode scanning. 

Leaders and users were asked to indicate their level of agreement or disagreement with the statement “2D 

barcode scanning saves (staff/me) time when recording data on vaccines administered to patients.” 

Respondents were presented with a five-point Likert scale for response options ranging from “Strongly 

Disagree” to “Strongly Agree.” As can be seen in Figure 1 below, perceptions of time savings differed 

somewhat by the perspective of leaders and users. 

Figure 1. Leader and User Survey Responses about Time Savings with 2D Barcode Scanning When 

Recording Vaccines Administered

Users were also asked to indicate their level of agreement or disagreement with the following two statements: 

1) “I was able to record data quickly using 2D barcode scanning” and 2) “Based on time only, I would prefer 

to record data about vaccines administered to patients using 2D barcode scanning.” The same five-point 

Likert scale of “Strongly Disagree” to “Strongly Agree” was used for these questions. Responses to both 

questions were similar and positive; users most often agreed or strongly agreed that they could record data 

quickly with 2D barcode scanning and expressed a preference for 2D barcode scanning based on time alone

(Figure 2). 

5 

19

38

19 19

10

22

18

34

16

0

5

10

15

20

25

30

35

40

45

Strongly

Disagree

Disagree Neither Agree

nor Disagree

Agree Strongly Agree

Respondents (%)

Leader (n=42) User (n=88)

Summary Report – September 2015
Figure 2. Additional User Survey Responses about Time Savings with 2D Barcode Scanning When 

Recording Vaccines Administered

10

17 17

43

13

10

14

19

47

10

0

5

10

15

20

25

30

35

40

45

50

Strongly Disagree Disagree Neither Agree

nor Disagree

Agree Strongly Agree

User Respondents (%)

Able to Record Data Quickly (User, n=88) Based on Time Only, Prefer 2D (User, n=88)

Several factors have the potential to affect the perceptions leaders and users have with respect to any time

savings conferred through the scanning of 2D barcodes to record vaccines administered. To identify patterns 

suggestive of factors that may be associated with these perceptions, we calculated cross-tabulations using data 

from the User Survey compared with facility characteristics, user background characteristics, patterns of use, 

and user experiences scanning 2D barcoded vaccines. Visual comparison of numbers and percentages were 

used to draw conclusions from these cross-tabulations. 

We examined five factors related to characteristics of the facilities within which users work (number of 

physicians in the facility, the facility specialty, the type of facility, previous methods used at the facility to 

record data about vaccines administered to patients, and the approximate percentage of vaccines at the facility 

that were 2D barcoded as of the survey response date). Three of these factors demonstrated patterns that 

indicate a potential relationship with user perceptions of time savings, including facility specialty, facility type 

and data entry methods previously used. With respect to facility specialty, a larger percentage of users who 

work in Federally Qualified Health Centers (FQHCs) (n=8, 73%) noted that they agree scanning 2D barcodes 

saves them time compared with users who work in private pediatric practices (n=17, 39%). For facility types, 

almost 20% more users working in facilities classified as publicly funded vaccinators (n=10, 63%) indicated 

that 2D barcode scanning saved them time compared with users in privately funded vaccinator facilities 

(n=30, 45%). With respect to previous data entry methods used within the facility, users who reported using 

both drop down lists and manual entry to record vaccine data immediately prior to this project were equally 

divided; 47% (n=17) responded that they disagreed that 2D barcode scanning saved them time when 

recording data about vaccines administered, while 44% (n=16) felt that 2D barcode scanning saved them 

time. A different pattern emerged for users who noted that they manually typed data into the 

EMR/Immunization Information Systems (IIS) immediately prior to the project—more than half of these 

users (n=22, 56%) agreed that 2D barcode scanning saved them time whereas only 18% (n=7) disagreed. 

Summary Report – September 2015
The background and experience of users with respect to barcode scanning, and recording vaccine data in 

general, has the potential to affect their perceptions of the time savings associated with 2D barcode scanning. 

Five user background characteristics were compared with perceived time savings (number of years recording 

vaccines administered, previous experience with barcode scanning in a medical setting and outside of a 

medical setting, level of comfort with scanning technology, and 2D barcode scanner training received). Of 

these user characteristics, three patterns of interest were identified— number of years of experience, previous 

experience with barcode scanning in a medical setting, and users with a high level of comfort with scanning 

technology. Forty-five percent (n=10) more users with four to eight years of experience recording data about 

vaccines administered agreed (n=15, 68%) rather than disagreed (n= 5, 23%) that 2D barcode scanning saved 

them time. Responses provided by individuals with up to one year of experience or nine or more years of 

experience demonstrated a similar direction in responses (more in agreement than in disagreement regarding 

time savings); however, not with the same magnitude. Of users who previously used barcode scanning to 

record medical information (n=25), 32% more agreed (n= 13, 52%) than disagreed (n=5, 20%) that 2D 

barcode scanning saved them time recording data on vaccines administered to patients. Similar patterns were 

not seen among users who had not previously used barcode scanning in medical settings (n=63). Twentyseven percent (n= 14) more users who reported a high level of comfort with barcode scanning technology 

agreed (n= 30, 57%) rather than disagreed (n=16, 30%) that 2D barcode scanning saved them time. Similar 

patterns were not seen among respondents who reported a moderate level of comfort with barcode scanning 

technology.

The frequency with which users had the opportunity to scan 2D barcoded vaccines and the frequency with 

which they actually scanned 2D barcoded vaccines could be associated with their perceptions about whether 

2D barcode scanning saves them time when recording data about vaccines administered. Four factors related 

to patterns of use were compared with user perceptions of time savings (when 2D barcoded vaccines are 

scanned, when non-2D barcoded vaccines are entered, frequency of scanning 2D barcoded vaccines, and 

percentage of 2D barcoded vaccines scanned versus entered through another method). Patterns emerge 

related to two factors – percentage of vaccines currently 2D barcoded at the facility and frequency of 

scanning 2D barcoded vaccines. Users who indicated that some to most (26%-99%) of vaccines in their 

facility have 2D barcodes more frequently responded that they agreed (rather than disagreed) that 2D barcode 

scanning saves them time. The opposite pattern, however, was seen when very few vaccines within a facility 

(1-25%) were 2D barcoded—in this case, a higher frequency of respondents disagreed (n=12, 71%) than 

agreed (n=2, 12%) that 2D barcoding saves them time. The percentage of users who agreed and disagreed, 

for those who reported scanning 2D barcodes to record vaccines administered up to six times per week was 

same (n=16, 36%). This differs from the pattern seen among users who scan 2D barcodes 10 or more times 

per week—with more than half (n=27, 59%) agreeing that 2D barcode scanning saved them time and 30% 

(n=14) disagreeing. Users who reported using a scanner to record very few (1-25%) of the 2D barcoded 

vaccines most frequently disagreed with the statement that 2D barcode scanning saved them time when 

recording data about vaccines administered (n=12, 71%).

Difficulty or ease with scanning 2D barcodes on vaccine products may relate to users’ perceptions of whether 

2D barcode scanning saves time when recording vaccine administration data. Twelve factors related to user 

experience with 2D barcode scanning and perceptions of time savings were compared. These included factors 

assessing the difficulties users experienced with 2D barcode scanning, their general experience with 2D 

barcode scanning, and users’ perceptions of the benefits of 2D barcode scanning. Consistent patterns 

emerged for the majority of factors examined under each of these headings. 

Summary Report – September 2015
With respect to users’ responses regarding their level of agreement or disagreement with time savings in 

relation to potential difficulties with scanning—higher percentages of respondents who agreed that they 

found it easy to scan vaccine vials, syringes, and labels that they peeled off of a vaccine vial or syringe agreed 

(n=24, 77% for vials; n=26, 72% for syringes; n=17, 74% for peeled-off labels) rather than disagreed that 2D 

barcode scanning saves them time. An opposite pattern was seen in responses from individuals who did not 

find it easy to scan vials or syringes –72% (n=18) of those who had difficulty scanning vials and 68% (n=13) 

of those who had problems scanning syringes did not feel that 2D barcode scanning saves them time. Users 

who did not experience inconsistencies in 2D barcode scanning (requiring them to scan more than once) 

more frequently noted that they agree that 2D barcode scanning saved them time (n=15, 75%) compared 

with users who did experience 2D barcode scanner inconsistencies (n=26, 42%). 

With respect to the users’ general experience with using 2D barcode scanning—users who disagreed that 

benefits of scanning were worth the change in process, that scanning was easy to integrate into normal work 

procedures, or that it was easy to use 2D barcode scanning to record data all most frequently indicated that 

they disagreed that 2D barcode scanning saved them time (n=14, 93% for benefits; n=17, 81% for integration 

into work procedures; n=12, 75% ease of use). An opposite pattern emerged for respondents who agreed 

with these statements, with most noting that they agree 2D barcode scanning saves them time (n=41, 75% for 

benefits; n=37, 77% for integration into work procedures; n=36, 76% ease of use). 

For users’ perceptions about other potential benefits of 2D barcode scanning to record vaccine data (e.g., 

improvements in data accuracy) and their general perspective and satisfaction regarding the use of this 

technology – users who disagreed that 2D barcode scanning leads to more accurate vaccine data than other 

approaches their facility uses, disagreed that they prefer recording data using 2D barcode scanning over other 

approaches they have used, or disagreed that they were satisfied overall with the use of 2D barcode scanning 

frequently indicated that they disagreed that 2D barcode scanning saves them time (n=9, 82% accuracy; n=16, 

89% data entry preference; n=21, 88% overall satisfaction). An opposite pattern was observed for those who 

agreed with these statements – with most of these respondents agreeing that 2D barcode scanning saves them 

time (n=41, 63% accuracy; n=37, 76% data entry preference; n=34, 83% overall satisfaction). 

Challenges encountered during scanning were identified from responses to open-ended questions in both 

surveys and through WFA visits. As might be expected, these challenges generally reduced data entry 

efficiency and time savings associated with barcode scanning. Overall, users were slowed down during the 

data entry process if they had to attempt multiple scans before achieving a successful one, abandon the 

scanner and revert to a manual entry process, or alternate between manual and scanned entries. The causes of 

these problems were difficult to ascertain and could include numerous variables, including the scanner, 

barcodes, EMR system or configuration, individual, or other components involved in the scanning process, 

the process change, the availability of scanners in the facility, or the number of 2D barcoded vaccines. 

Respondents of the Leader and User Surveys provided written feedback about benefits impacting data entry 

that are realized or anticipated with the use of 2D barcode scanners to record vaccine data. The main benefits 

impacting efficiency that were identified included: time savings with 2D barcode scanning; convenience and 

ability for less manual entry with 2D barcode scanning; and additional data elements captured in a single scan. 

Additionally, many respondents indicated conditional or potential benefits that are yet to be fully realized. 

Further detail about each of these benefits and their impact on the data entry process is provided below. 

Summary Report – September 2015
Question 3: To what extent do perceptions of 2D barcode scanning time savings for recording data 

about vaccines administered align with observations?

To address this evaluation question we compared data from time measurements to record vaccines 

administered with the following survey questions “2D barcode scanning saves staff time when recording data 

on vaccines administered to patients”(Leader Survey) and “2D barcode scanning saves me time when 

recording data on vaccines administered to patients”(User Survey). Both survey questions were presented to 

respondents with a five-point Likert scale ranging from “Strongly Disagree” to “Strongly Agree.” 

Comparison of time measurement observation data and survey data show a mix of alignment and 

inconsistencies between leaders and users, even within the same facility. Of the 16 facilities where time 

measurements were taken: 

• three facilities only had data for the timed measurements (therefore comparison with leader or user 

perceptions was not possible); 

• four facilities showed consistent findings between measured time savings with the use of 2D barcode 

scanning and perceptions of time savings by leaders and/or users; and

• nine instances of inconsistencies were found, including 1) a mis-match between time measurement 

data and perception of survey respondents and 2) disagreement among survey respondents. 

An example of inconsistency in both directions was seen when timed measurements indicated a statistically 

significant difference in time savings with 2D barcode scanning (compared to their traditional entry method), 

yet the leader respondent disagreed that 2D barcode scanning saved time and the two user respondents either 

agreed or were neutral as to 2D barcode time savings. 

For the four WFA facilities with faster timed measurement using their traditional method (not 2D barcode 

scanning), two had leaders and/or users indicating the same pattern (disagreement with 2D time savings), 

whereas the other two facilities only had respondents perceiving no time difference or indicating that 2D 

scanning saved them time. 

Data gathered through the WFA indicate that inconsistencies between observations or time measurements 

and surveys or interviews may potentially be attributed to the overall process workflow characteristics. 

Perceptions about efficiency and time savings may be driven by peripheral process or scanning challenges and 

not be directly driven by the time required for data entry through 2D barcode scanning. A scenario observed

during a WFA visit showed that despite being one of the fastest in terms of the average time required for 2D 

barcode scanning (4.94 seconds), it entailed several process steps, including multiple data entry steps to the 

state registry, a paper chart and the EMR, as well as connecting and disconnecting the scanners to the laptop 

for every batch data entry. The Leader Survey input was aligned with some of the comments received through 

the on-site staff interviews indicating that they did not see 2D scanning as a time savings improvement. It is 

possible that although data entry does appear to be quicker when timed, the improvement of approximately 

five seconds may not be recognized as meaningfully different by those engaged in the process. 

Question 4: What are project participants’ perceptions about changes in time savings related to 

recording data about vaccines into inventory?

Survey respondents were asked about their perceptions related to time savings with the introduction of 2D 

barcode scanning to record vaccine inventory. One question was included in the Leader Survey and three 

Summary Report – September 2015
questions in the User Survey to assess various aspects of changes in time to record vaccine inventory with use 

of 2D barcode scanning. 

Leaders and users were asked to indicate their level of agreement or disagreement with the statement “2D 

barcode scanning saves (staff/me) time when recording vaccine inventory.” Respondents were presented with 

a five-point Likert scale of response options ranging from “Strongly Disagree” to “Strongly Agree.” 

Assessment of time savings when recording vaccine inventory was most frequently neutral for both leaders 

(n=6, 35%) and users (n=9, 27%) and evenly distributed between agreement (leaders n=4, 24%; users n=8, 

24%) and disagreement (leaders n=4, 24%; users n=8, 24%). More users “strongly disagreed” with this 

statement, compared with leaders (n=5, 15% and n=1, 6%, respectively). However, there were a small 

number of respondents to this statement and therefore the significance of these data is unclear.

Users were further asked to indicate their level of agreement or disagreement with two additional statements 

“I was able to record vaccine inventory quickly using 2D barcode scanning” and “Based on time only, I 

would prefer to record vaccine inventory using 2D barcode scanning.” The same five-point Likert scale of 

“Strongly Disagree” to “Strongly Agree” was also used for these questions. Users were mixed as to whether 

they agreed/strongly agreed (n=12, 36%) or disagreed/strongly disagreed (n=14, 42%) that 2D barcode 

scanning allowed rapid inventory recording. For the preference of recording vaccine inventory using 2D 

barcode scanning, based on time alone, 48% (n=16) indicated agreement/strong agreement, compared to 

27% (n=9) indicating disagreement/strong disagreement.

Data Quality and Compliance

The second topic of interest focused on five evaluation questions that investigate the associations between 

2D barcode scanning, data quality, and compliance with 2D barcode scanning. Both measured data quality 

and compliance were included, as well as comparison with self-reported survey data. 

Findings indicate that 2D barcode scanning improved the data quality (total correctness) of both lot number 

and expiration date. Data quality improved by 11% (for total correctness) when a 2D barcode was scanned 

versus when it was entered via traditional methods (i.e., manual entry, drop down menus). However, although

use of 2D barcode scanning led to improvements in data quality, there were still errors introduced in the data 

entry process. One common error was introduced into the lot number field, where intentional edits were

made by staff to indicate whether the vaccine was publicly purchased. Analysis of facilities with scan logs 

revealed that there was variation between actual scanning behavior and self-reported behavior from survey 

respondents at those same facilities. While survey respondents reported relatively high rates of scanning (most 

frequently more than 50% scanning compliance), measured compliance for those same facilities revealed 

relatively low rates of scanning (most frequently less than 25% scanning compliance).

Below we present a summary of findings from the five overarching evaluation questions of focus in this 

topic.

Evaluation Question 1: To what extent does vaccine administration lot number and expiration date 

correctness change with the introduction of 2D barcode scanning? 

To answer this evaluation question the Deloitte team examined data quality for all of the EMR vaccination 

records submitted by facilities during the project period. This included records for both 2D and non-2D 

Summary Report – September 2015
barcoded vaccines administered both pre- and post- 2D barcode scanner installation. Pre-installation 

correctness was 83.7% correct for lot number and 72.7% for expiration date. This indicates that the facilities 

participating in the project had good correctness prior to scanner installation. Despite the relatively good data 

quality pre-installation, facilities observed improvements in the data quality of both lot number and expiration 

date post-installation of 2D barcode scanners. From pre- to post-scanner installation, the lot number 

correctness improved by 8%, from 84% to 92%, and expiration date total correctness improved by almost 

11%, from 73% to 84%. The observed improvements in correctness from pre- to post-installation for both 

lot number and expiration data were statistically significant.

We also explored variation in data quality by facility type. Variation in facility specialties may influence data 

quality due to different specialties having unique data entry processes, reporting requirements, and standards 

around vaccination data. Total correctness for lot number total correctness was consistently above 90% for 

all specialties, while expiration date correctness ranged from 83% to 98%. Facilities that self-identified as 

publicly funded, were the facilities with the highest percent total correctness for both lot number and 

expiration date

Evaluation Question 2: How does the data quality differ between vaccine administration records that 

were scanned and those that were not? 

To assess whether data quality improvements are related specifically to the use of 2D barcode scanning the 

project team compared data quality between vaccines that were scanned and those that were not scanned, 

from a subset of facilities with a direct indicator of scanning. The average lot number total correctness of 

scanned vaccination records was 95.3%. The average lot number total correctness of vaccination 

administration records entered via traditional methods was 94.3%. This improvement was statistically 

significant and corresponds to an additional 1,200 accurate vaccine records out of approximately 125,000 

records in the EMR dataset. 

In addition, total correctness improved for expiration date for vaccines that were scanned. The average 

expiration date total correctness increased by 11% between records that staff scanned versus entered using 

traditional methods, a statistically significant difference. The average expiration date correctness of scanned 

vaccination records was 96%. 

This project also explored how data quality varied between vaccine records that were scanned and those that 

were not scanned across various types of facilities. Lot number total correctness was higher when lot number 

was scanned in public health departments and primary care facilities. Lot number total correctness was not 

improved when vaccines were scanned in Federally Qualified Health Centers, hospital based clinics and 

pediatric facilities. Expiration date total correctness improved when vaccines were scanned in hospital based 

clinics. 

To account for any undue influence from non-2D barcoded vaccines on data quality results, the project team 

examined the difference between vaccines scanned and entered via traditional methods among just those 

vaccines with 2D barcodes. The average lot number correctness was higher (97.3%) for 2D barcoded 

vaccines that were not scanned into the EMR compared to those that staff scanned into the EMR (96.9%). 

When the data are limited to just those vaccines with 2D barcodes there is not a statistically significant 

difference in lot number correctness between vaccines scanned versus those not scanned; there are a number 

Summary Report – September 2015
of factors that could be influencing this, including the use of the 2D barcode indicator to subset the entire 

population. This will be explored further in future analyses. 

A different pattern emerges when analyzing expiration date. Records scanned into the EMR exhibited higher 

levels of average expiration date total correctness for 2D barcoded vaccines scanned into the system (95.8%) 

compared to those that were not (91.7%). Expiration date correctness demonstrates a statistically significant 

difference of four percentage points—with 2D barcoded vaccines scanned into the system demonstrating a 

higher level of total correctness in expiration date than those not scanned into the system. 

While improvements in vaccine administration data quality appear to be a benefit of 2D scanning, there could 

be confounding factors that influence the relationship between data quality and 2D barcode scanning. For 

example, facility type may influence data quality independent of 2D barcode scanning. Other potential 

confounding factors include individual facility data quality, manufacturer, scan log type, vaccine type and 

temporal effects. To evaluate the effect of 2D barcode scanning on data quality, the project team constructed 

two multilevel logistic regression models –one for lot number correctness and one for expiration date 

correctness. Results indicate that after controlling for potential confounding factors, 2D barcode scanning 

was associated with improved lot number and expiration date total correctness. 

Evaluation Question 3: What factors explain compliance with scanning 2D barcoded vaccines?

Results of previous Evaluation Questions suggest that 2D barcode scanning improves data quality. A very 

important factor in whether these benefits will be fully realized depends on the extent to which practitioners 

use the 2D barcode scanners. The direct indicator of scanning (i.e., from scan logs) available in this project 

allowed us to assess compliance of 2D barcode scanning within facilities and to better understand factors that 

may affect this compliance. 

To address this question, the project team first considered whether variation in compliance exists by facility 

and then by the type of vaccines scanned. Such variation, both among facilities and vaccine type, is important 

to understand so that barriers to scanner use can be addressed. For example, high volume facilities may scan 

less than lower volume sites because integrating a new technology into an already strained workload may be 

challenging. Among vaccines, variation in scanning compliance may highlight issues such as faded or difficultto-read barcodes, which can result in less scanning of these types of vaccines. 

Compliance varied considerably among facilities and vaccine types. Compliance by facility ranged from 3% to 

100%. Facilities with the highest compliance were mass vaccinators that do not have an alternative method 

for data entry aside from the scanner. Compliance by vaccine type ranged 0-100% with a median of 21.3 

percent. Note that the extreme ends of the compliance spectrum are vaccines that appear infrequently in the 

EMR data. Among those vaccines that appear often in the EMR data, vaccines with the highest compliance 

included Engerix®-B, Menactra®, Adacel®, Tenivac® and Boostrix®. Those vaccines with the lowest

compliance included Daptacel®, Fluarix® and Pentacel®. To explore additional factors that could influence 

compliance, the project team constructed a multi-level logistic regression model. The analysis demonstrated a 

significant decrease in compliance in September, suggesting that scanning may decrease during busier times of 

the year such as back-to-school and flu season, which increase staff workload. Vaccine type was tested based 

on user feedback during the WFA observations and user experience surveys. Model results indicated 

significant variation in compliance by vaccine manufacturer and vaccine type. When staff administer more 

than one vaccine during a single patient visit (as is often the case for patients under three years old), some 

Summary Report – September 2015
practitioners may find it easier to enter all the data about these vaccines into the EMR using one method of 

data entry. Model results support this assertion, with lower scanning compliance when 2D barcoded vaccines 

are given in conjunction with non-2D barcoded vaccines.

This finding reinforces the need for all vaccine products to be 2D barcoded. Qualitative and quantitative 

evidence indicates that health care practitioners would use 2D scanning technology more if they did not have 

to switch between data entry methods. Higher rates of scanning are associated with better vaccine 

administration data quality so if all vaccines were 2D barcoded then it would be expected that scanning 

compliance rates would increase and data quality would improve. 

An additional temporal dimension of technology adoption is whether the adoption of the technology 

increases or decreased over time. To understand how the adoption of 2D barcode technology changed over 

time, the project team examined compliance rates throughout the course of the project period. Findings 

indicate that, on average, compliance decreased across the project period. While there were varying patterns 

by facilities, a consistent decline in compliance was observed in all of the facilities with direct measurement of 

2D barcode compliance. There are several potential explanations for the decline: 1) staff growing increasingly 

frustrated with 2D barcode scanners not working as expected; 2) staff turnover resulting in new staff not 

trained in the use of 2D barcode scanners; 3) initial excitement over a new technology but staff eventually 

reverts back to old process, and 4) initial interest and commitment to the pilot that may wane after the 8 

month duration. While these are possible explanations for declining compliance observed over time, none of 

these hypotheses was tested during the current project. However, these findings indicate that efforts should 

be made to encourage continued compliance with 2D barcode scanning, which will benefit data quality.

Evaluation Question 4: What types of errors in data entry occurred?

The fourth evaluation question addresses types of errors common in vaccination administration data from 

EMR systems. Specific examples of errors in data entry were obtained from the User Survey, our review of 

EMR data, and Workflow Analysis site visits. 

Our review of EMR data identified several common types of errors in the lot number field. These errors are 

categorized into four main types. Without observation of each instance, it is not possible to confirm the 

reason for these inconsistences, but feedback indicated that limited restrictions in the EMRs lot number field, 

allowing for manual manipulation of the data in the field is the likely reason; oftentimes data was changed to 

provide additional, internally used only, meaning to lot numbers. 1) Those that occurred as a result of

individuals at a facility adding notation related to a vaccine being publicly funded, whether this is through 

addition of a “P” (7,395 instances of this observed), use of a dash (903 instances), or addition of a space 

(1,160 instances). This type took the form of a single “P”, a dash or space followed by a “P”, or a “VFC” 

addition to the field. 2) Multiple lot numbers in the same field (2,968 instances), perhaps to capture lots from 

multiple parts to a multi-component vaccine. 3) A date added into the lot number field (661 instances) or 

combined with lot number (187 instances). 4) The addition of other punctuation (273 instances), numbers 

(142 instances), or characters (71 instances). 

While most errors are associated with human error in manual entry, there are a few types of intentional edits 

to lot numbers made by facilities, such as type 1) listed above. These types of intentional edits negatively 

affect the lot number total correctness because the lot number can no longer be easily searched for during a 

recall. These intentional edits comprise over 70% of inaccurate lot numbers and over one percent of all of the 

Summary Report – September 2015
vaccine administration records collected during this project. This equates to more than 10,000 vaccine records 

that would be considered correct if these intentional edits had not been made in the lot number field. 

Nearly one-quarter of user respondents reported experiencing an incorrect entry at some point after scanning 

2D barcodes, either while recording vaccines administered (n=19, 22%) or during vaccine inventory (n=8, 

24%). Specific examples of errors with incorrect entries described by users through write-in survey responses 

included: missing or incorrect elements of lot number or expiration date, data entered into the wrong area of 

the EMR, receiving an error message when visual confirmation of information on vaccine label does not 

support, or reports of the EMR linking scanned data to incorrect product or lot numbers. These errors may 

be due to issues with the scanner, user error, or the EMR configuration. 

Two primary types of errors observed during the WFA site visits included: 1) intermittent data entry errors 

with the lot number and expiration date entries and 2) 2D barcode scanning entries interfering with the 

capture of another data field, such as manufacturer name, based on other data entered or pulled into the 

system. In the first example, the barcode appeared to be correctly encoded and the EMR configuration was 

confirmed as correct; however, the lot number and expiration date were not populated correctly within the 

system. This error may have been due to an intermittent issue with the scanner-workstation interface, but 

should be explored further. In the second situation, the scanned data seems to have interfered with other data 

in the system, which may have come from the EMR lookup tables utilized in the application back-end or data 

potentially provided during the vaccine inventory process. This issue does not appear to be attributable to 2D 

barcode scanning itself, but is something that immunizers and EMR vendors should take into consideration 

as they work to improve the EMR-2D barcode scanning integration process. 

Evaluation Question 5: What are practitioners’ perceptions related to data quality and compliance? 

The fifth and final evaluation question presents self-reported information around data quality (accuracy) and 

compliance from practitioners, as collected through the Leader and User Surveys. Findings are presented for 

user characteristics and experiences related to perceptions of data quality and compliance. Further, we 

compared these self-reported survey data to measured data quality and compliance collected through EMR

data. 

Data Quality

Leaders and users provided insight into their level of agreement or disagreement with the statement “I feel 

that 2D barcode scanning leads to more accurate vaccine data than other approaches this facility has used” 

related to recording vaccines administered. A large majority of both leaders and users responding to this 

question agreed or strongly agreed that 2D barcode scanning improves accuracy of data for recording 

vaccines administered (n=36, 86% and n=65, 74%, respectively). Twelve percent of leaders (n=5) and 14% of 

users (n=12) selected the neutral response option for whether 2D barcode scanning improved the accuracy of 

data when recording vaccines administered. Few leaders or users disagreed or strongly disagreed with this 

statement (n=1, 2% and n=11, 12%, respectively). 

Cross-tabulations of responses to the statement (above) about accuracy from the User Survey and respondent 

characteristics identified several potential factors associated with user perceptions. Factors that may affect 

user perceptions of the accuracy of data entered using 2D barcode scanning (compared to other approaches 

previously used) included: 1) specialty of facility in which the respondent worked; 2) percentage of vaccines 

Summary Report – September 2015
within their facility that were 2D barcoded; 3) type of training received; 4) self-reported percentage of 2D 

barcoded vaccines scanned versus entered using another method; 5) ease of scanning vaccine vials or syringes 

successfully with 2D barcode scanning; 6) ability to recover easily after making a mistake using 2D barcode 

scanning; 7) ease of integration into usual work processes; 8) benefit of using 2D barcode scanning being 

worth the change in process; 9) ease of using 2D barcode scanning to record vaccines administered; 10) 

perception of time savings with use of 2D barcode scanning; 11) preference for 2D barcode scanning over 

other approaches previously used; and 12) overall satisfaction with 2D barcode scanning. 

When we compared alignment between measured EMR data for data quality and perceptions reported by 

leaders and users through the online surveys (from the same 27 scan log facilities that provided direct 

assessment of when 2D barcode scanning took place) limited alignment was found. Overall, survey 

respondents from these scan log facilities perceived positive changes in accuracy with 2D barcode scanning, 

with 19 of 22 leaders (86%) and 29 of 38 users (76%) having agreed or strongly agreed that 2D barcode 

scanning leads to more accurate vaccine data than other methods used. However, measured EMR data were 

mixed as to whether scanned or not-scanned vaccine entries provided better accuracy. Just over half of the 27 

facilities showed measured improvements in accuracy with 2D barcode scanning (n=14, 52%), and the 

remaining facilities showed no difference in accuracy between scanning or not-scanning (n=5, 19%) or had 

greater accuracy with not-scanned vaccine entries (n=8, 30%). A mix of alignment and inconsistencies were 

found between observations and survey responses between leaders and users within the same facility. Twelve 

of 26 facilities showed consistent findings between measured accuracy and perceptions by leaders and/or 

users. Inconsistencies were seen when: 1) measured accuracy did not match the perception of survey 

respondents and 2) when disagreement was seen among survey respondents within the same facility. There 

were 14 instances of these inconsistencies (six of which include an overlap of both types of inconsistencies).

Compliance

Leaders8 and users9, 10 disagreed about prevalence of data entry utilizing 2D barcode scanning. As shown in 

Figure 3, Users most frequently indicated scanning “most (76-99%)” or “some (26-50%)” 2D barcoded 

vaccines administered, while leaders primarily indicated their staff scanned “most (76-99%)” 2D barcoded 

vaccines administered. Neither of these respondent groups indicated that “none (0%)” of the 2D barcoded 

vaccines were scanned for vaccines administered. 

Figure 3. Self-Reported Estimate of Percentage of 2D Barcoded Vaccines Scanned Versus Entered 

Through another Method, to Record Vaccines Administered, Leader and User Surveys

8 Leader Survey question: “What percentage of the 2D barcoded vaccines administered to patients have been scanned 

instead of entered through another format (for example, typing data, drop-down menus)? Please estimate this percentage 

from the time that 2D barcode scanners were installed at your facility for this project and you had 2D barcoded vaccines 

available to scan.”

9 User Survey question, “Of the 2D barcoded vaccines administered in your facility that you have personally recorded 

data about, what percentage have you scanned instead of entered through another format (for example, typing data, 

drop-down menus)? Please estimate this percentage from the time that 2D barcode scanners were installed at your 

facility for this project and you had 2D barcoded vaccines available to scan.”

10 Survey participants were provided with six response options to these questions: “All (100%),” “Most (76-99%),” 

“Many (51-75%),” “Some (26-50%),” “Very few (1-25%),” and “None (0%)”.

Summary Report – September 2015

 
We analyzed measured EMR compliance at facilities that had a scan log in relation to several factors from the 

user experience survey—including facility characteristics, user background/experience with barcode scanning, 

patterns of use with respect to 2D barcode scanning of vaccines, as well as several general and specific factors 

that relate to users’ perceptions of 2D barcode scanning and experience. Sample sizes were small; this 

comparison utilized only scan log facility data (in order to assess measured compliance levels) and therefore 

interpretation of several characteristics (those with ten or fewer responses within a given response option) 

were not possible. Cross-tabulations indicate that the following factors may be related to level of scanning 

compliance: 1) previous method of entering vaccines administered; 2) previous use of barcode scanning 

outside of a medical setting; 3) weekly frequency of scanning 2D barcoded vaccines; 4) perceptions that the 

benefit of 2D barcode scanning is worth the change in process; 5) ease of integrating 2D barcode scanning 

into usual work procedures; 6) perceptions of time savings with 2D barcode scanning; 7) ability to record data 

quickly with 2D barcode scanning; and 8) overall satisfaction with 2D barcode scanning. 

The project team found little relationship between self-reported compliance from the Leader and User 

Surveys and compliance measures using the scan log information. The 60 survey respondents (leader and 

user) most frequently noted a medium level of scanning compliance (n=29, 48%), estimating that 26-75% of 

the 2D barcoded vaccines administered were scanned versus entered using another method. Another 17 

respondents (28%) indicated a high level of compliance (76% or more) and the remaining 14 respondents 

(23%) indicated a low level of scanning compliance (25% or less). However, measured compliance indicates 

much lower levels of scanning compliance than self-reported survey responses. Two-thirds of the 27 facilities 

measured showed a low level of compliance with 2D barcode scanning (n=18, 67%), another 8 facilities 

(30%) had a medium level of compliance, and just one facility demonstrated a high level of compliance (4%). 

Little alignment was seen within the same facility between observations and surveys or even between leader 

and user survey responses. Only 3 of 26 facilities showed consistent findings between measured compliance 

and self-reported compliance by leaders and/or users. Inconsistencies were seen when: 1) measured 

compliance did not match the self-reported compliance of survey respondents and 2) there was disagreement 

among survey respondents within the same facility. There were 23 instances of these inconsistencies (12 of 

which include an overlap of both types of inconsistencies) in the 26 facilities. 

0 

17

14

21

43

5 

0 

19

24

21

25

11

0

5

10

15

20

25

30

35

40

45

50

None (0%) Very few

(1-25%)

Some

(26-50%)

Many

(51-75%)

Most

(76-99%)

All (100%)

Respondents (%)

Leader (n=42) User (n=88)

Summary Report – September 2015
User Experience

The third topic examined the implementation of 2D barcode scanning and the experience of leaders and 

direct users at participating facilities within the current project. Two overarching evaluation questions (and 

several sub-questions) were the focus of the User Experience. Broadly, these questions focused on the 

experience of the end-user during the process of adopting 2D barcodes to record vaccine information and the 

potential for sustaining 2D barcode scanning after the current project ends. 

Findings indicate a general receptiveness to 2D barcode scanning among participating users and facilities. 

Many respondents reported that it was easy to learn and use 2D barcode scanning technology in a short 

timeframe. Additionally, many participants appeared satisfied overall with 2D barcode scanning and 

expressed a preference for 2D barcode scanning over other approaches that they had previously used for 

recording vaccine data. Most participants perceived that 2D barcode scanning improves the accuracy of 

vaccine data recorded and considered the benefits of 2D barcode scanning worth the change in process. 

Further, more than half of leaders indicated that it was likely or extremely likely that their facility would 

continue to use 2D barcode scanning after the end of the project. Project participants also provided an initial 

list of improvements that have the potential to increase the likelihood of sustaining the practice of 2D 

barcode scanning to record vaccine information, such as: 2D barcodes on more (if not all) vaccine products, 

more data elements populated in the EMR with a single scan, enhanced scanner ease of use and scanning 

reliability, improved 2D barcode print quality, and that for the benefits to be fully realized, the scanner 

hardware should capture the scan on the first try, which currently is not always the case. 

We present findings on each of the evaluation questions from this below. 

Question 1: What is the experience of the end user during the process of adopting 2D barcodes to 

record vaccine information? 

How often did end users have opportunities to scan 2D barcoded vaccines, and how frequently did they act on these 

opportunities? What was the landscape in which end users were to employ 2D barcode scanning?

In order to assess opportunities to scan 2D barcoded vaccines, we needed to understand whether vaccines 

with 2D barcodes had been received by participating facilities. Responses from both leaders and users 

indicated that staff had moderate to high exposure to 2D barcoded vaccines during the current project. The 

majority of facilities indicated that at least half of their vaccines were 2D barcoded, with 74% of leaders 

(n=31) and 60% of users (n=68) reporting that at least 50% of the vaccines at their facility had 2D barcodes. 

Further, leaders from participating facilities reported receiving 32 different vaccine products with 2D 

barcodes. Eleven different vaccine products were indicated as received with 2D barcodes by 50% or more of 

the facilities administering that type of vaccine product. 

Further, we sought to understand the extent to which individuals had utilized the 2D barcoded vaccines that 

their facility received. Most leaders indicated that their facility had been scanning 2D barcoded vaccines for 

more than four months to record information about vaccines administered (n=32, 76% of 42 responses 

received) and vaccine inventory (n=12, 52% of 23 responses received). Additionally, over half of users who 

personally scanned 2D barcodes to record data for vaccines they administered reported scanning 2D 

barcoded vaccines ten or more times per week (n=46, 52%). Leaders reported more frequently that staff 

Summary Report – September 2015
recorded vaccines using the 2D barcode scanner compared to other methods (e.g., drop-down menu, manual 

entry) than did users. Users most frequently indicated scanning “most” (76-99%) or “some” (26-50%) of the 

2D barcoded vaccines they administered (n=22, 25% and n=21, 24%, respectively). Leaders, on the other 

hand, indicated that their staff scanned “most” (76-99%) (n=18, 43%) of the 2D barcoded vaccines 

administered at their facility, with only 14% of leaders (n=6) indicating “some” (26-50%). Seventeen percent 

(n=7) of leaders and 19% (n=17) of users indicated that using 2D barcode scanning to record vaccines 

administered took place infrequently (1-25% of the time). Higher rates of scanning 2D barcoded vaccines 

were reported for recording vaccines administered, compared to recording vaccine inventory.

To better understand the contextual landscape within which 2D barcode scanning adoption took place, we 

sought to identify the locations within facilities where vaccine data were recorded, as well as the processes 

used for data entry prior to the project.11 Facilities providing responses to the surveys most frequently used

stationary desktop computers outside of immunization rooms to record data for vaccines administered (n=35, 

52%). The next most frequent approach was to use laptops brought into immunization rooms (n=26, 39%) 

or stationary computers inside immunization rooms (n=21, 31%). Respondents indicated that prior to the 

project, staff in their facilities primarily manually typed in data (leaders n=27, 40% and users n=39, 44%) or 

used a combination of drop-down and manual entry (leaders n=23, 34% and users n=36, 41%) to record the 

vaccines they administered. Leaders (n=30, 77%) and users (n=22, 67%) also noted that manual data entry 

was the most common method staff in their facilities used to enter information about vaccines into their 

facility’s inventory system. Few respondents indicated either using drop-down lists exclusively (between 3-

18%, depending on respondent and whether vaccine administration or inventory) or any type of barcode

scanning (0-7% of varying respondents and whether administration or inventory) prior to the pilot project. 

To prepare for 2D barcode scanning at participating facilities, the project team provided on-site trainings at 

the start of the project. At the end of the project, users were asked about how and if they were trained on the 

use of 2D barcode scanners. User Survey respondents indicated that they were primarily trained on the use of 

2D barcode scanners by the 2D barcode project team (n= 59, 51%) or a colleague (n=43, 37%). A small 

number of respondents indicated that they had not been trained (n=13, 11%).

We also wanted to understand when vaccines were recorded in relation to when they were administered, and 

whether this timing differed for 2D and non-2D barcoded vaccines. User Survey respondents indicated that 

both 2D and non-2D barcoded vaccines were most frequently recorded before vaccines were administered to 

patients. However, a slightly higher percentage of respondents indicated recording vaccine data before

administering the vaccine for 2D barcoded vaccines (n=54, 61%), compared to non-2D barcoded vaccines 

(n=49, 56%). Of those who recorded vaccine data after administering the vaccine (for both 2D and non-2D 

vaccines), most recorded the vaccine data immediately after administration (non-2D n=35, 40% and 2D 

barcoded n=30, 34%).

How well was 2D barcode scanning to record vaccine data accepted by intended end users?

Many leaders and users expressed satisfaction with 2D barcode scanning, especially when used to record data 

about vaccines administered. Respondents appeared less enthusiastic about the use of 2D barcode scanning 

to record vaccine information into the inventory system (although it should be noted that only a few facilities 

11 It should be noted that since not all vaccines are 2D barcoded, these same methods would continue to be used by the 

facilities to enter any non-2D barcoded vaccine data.

Summary Report – September 2015

 
were configured by our team to record vaccine inventory, n=6, 7%). Overall, a large portion of respondents 

indicated that 2D barcode scanning was: easy to integrate into usual work processes, easy to use, worth the 

change in process, preferred to other methods previously used to record vaccine data, and, perceived to 

improve the accuracy of data staff recorded. 

The majority of both leaders and users agreed or strongly agreed that it was easy to integrate 2D barcode 

scanning into their usual process of recording data about vaccines administered (n=25, 60% and n=48, 55%,

respectively). With respect to integrating 2D barcode scanning into the process for recording vaccine 

inventory, users most frequently indicated disagreement (n=12, 36%) and leaders most frequently selected the 

neutral response (n=6, 35%).

Most respondents also found 2D barcode scanning easy to use when recording data about vaccines 

administered, as demonstrated by agreement or strong agreement by leaders and users (n=25, 60% and n=50, 

57%, respectively). When respondents disagreed or strongly disagreed that 2D barcode scanning was easy to 

use, these responses most frequently referred to user assessment of 2D barcode scanning to record vaccine 

inventory (n=11, 33%). Users most frequently indicated that peel-off labels (n=23, 67%) and syringes (n=36, 

51%) were easy to scan successfully for the vaccines they administered, and that vials (n=8, 30%) and syringes 

(n=7, 27%) were easy to scan when recording vaccine inventory. Users were able to select all vaccine 

presentations that they had used, thus the numbers add up to more than 100%.

Approximately half of the respondents either agreed or strongly agreed that they were satisfied with 2D 

barcode scanning to record vaccines administered (leaders n=23, 55% and users n=41, 48%). Leaders 

indicated slightly higher satisfaction than users, based on visual comparison of the responses provided. When 

respondents expressed dissatisfaction with 2D barcode scanning, this was most frequently with respect to 

recording vaccine inventory (n=13, 39%). 

A majority of both leaders and users agreed or strongly agreed that the benefits of 2D barcode scanning are 

worth the change in process for recording vaccines administered (n=25, 61% and n=55, 63%, respectively). 

Users also indicated a preference for using 2D barcode scanning over any other approaches they had 

previously tried for recording vaccines administered, where 56% (n=49) agreed or strongly agreed with a 

preference for 2D barcode scanning. 

Respondents reported in very high numbers a positive perception about the increase in accuracy of vaccine 

data when using 2D barcode scanning. A large majority of both leaders and users agreed or strongly agreed 

that 2D barcode scanning improves the accuracy of data recorded about both vaccines administered (n=36, 

86% and n=65, 74%, respectively) and vaccine inventory (n=13, 76% and n=17, 52%, respectively). 

Respondents’ perceptions about time savings when recording vaccine data using 2D barcode scanning were

more mixed than some of the other questions about satisfaction, benefits of 2D barcode scanning, or 

improved accuracy of data with the introduction of 2D barcode scanning. The most frequent response from 

leaders about time savings was to select the neutral option (n=16, 38%), whereas users most frequently agreed 

that 2D barcode scanning saves time when recording vaccines administered (n=30, 34%).

To what extent does the acceptance of 2D barcode technology by end users for recording data about vaccine 

administrations relate to their background and experience?

Summary Report – September 2015
The project team performed a variety of additional descriptive analyses on the User Survey data to better 

understand what factors may have influenced the level of ease or difficulty experienced by users when 

integrating 2D barcodes into their existing process for recording vaccine data. We also investigated factors 

that may have influenced user satisfaction. We compared user feedback on ease of integration and satisfaction 

(both when recording vaccines administered) with several factors of interest, including: facility characteristics, 

respondent background, opportunities to scan, and user experience with 2D barcode scanning. Similar 

patterns were seen in the relationship between several factors and the respondents’ reported ease of 

integration and satisfaction. 

User respondents with higher levels of agreement with ease of integration and satisfaction most often: 

• Came from Federally Qualified Health Centers (FQHCs) compared to other facility types

• Were trained on using 2D barcode scanners by 2D project team members

• Had high or moderate levels of comfort using barcode scanning technology

• Scanned 2D barcoded vaccines more frequently

• Recorded vaccine data by scanning 2D barcodes more often than by using other methods (such as 

drop-down or manual entry)

• Were less likely to have experienced an incorrect entry after scanning a 2D barcode 

• Were less likely to have experienced inconsistencies with 2D barcode scanning 

• Found it easy to scan 2D barcoded vials/syringes/peel-off labels successfully 

• Preferred 2D barcode scanning over other recording methods they had used previously

• Considered the benefits of 2D barcoded vaccines worth the change in process

• Found 2D barcode scanning easy to use, and 

• Perceived improved accuracy and time savings with use of 2D barcode scanning

We observed differences in some instances between ease of integration and satisfaction with comparison to 

the factors studied. Ratings of ease of integration did not vary greatly across method of data entry previously 

used, years of user experience recording administered vaccines, previous experience scanning barcodes in a 

medical setting, and whether vaccines were recorded before or after administration, although variation in 

responses were seen in user satisfaction for these factors. Higher proportions of 2D barcoded vaccines 

received by the participating facilities seemed related to the ease of integration, but did not seem to influence

user satisfaction ratings. 

What challenges and benefits do end users most frequently report? 

Inconsistencies with scanning and scanners, faded barcodes, and incorrect entries after scanning were most 

frequently identified by project participants as problems. Survey respondents described many of these 

challenges, with additional examples gathered through the WFA interviews and observations. Beyond 

challenges, participants also provided information on approaches they have used to counter these challenges. 

Benefits identified by respondents focused mostly on improved accuracy, improved efficiency, and less 

manual entry of vaccine data. 

The inconsistent functioning of 2D barcode scanners was a major challenge identified by respondents and 

this was supported through observation during workflow analysis. Although we do not know the technical 

Summary Report – September 2015
cause of the challenges or the precise number of times this occurred, users frequently indicated the need to 

scan 2D barcodes more than once to successfully record data about the vaccines they administered (n=62, 

70%) or added to inventory (n=23, 70%). The majority of users indicated the need to attempt scanning more 

than once “frequently” for recording data for the vaccines they administered (n=33, 58%) or added to 

inventory (n=14, 70%). Previous experience with scanning inconsistencies was found to be related to lower 

levels of satisfaction and more difficulties with ease of integration, compared to users who had not 

experienced this challenge. Thirty-seven percent (n=22) of users who indicated inconsistencies with 2D 

barcode scanning and the need to scan vaccines multiple times agreed they were satisfied, compared to 80% 

(n=16) of users who had not experienced these issues previously. Additionally, 48% (n=30) of users who 

indicated issues with the inconsistency of 2D barcode scanning agreed with ease of integration, compared to 

70% (n=14) of users who had not experienced these issues.

Users reported encountering faded barcodes more frequently than smeared barcodes (n=27, 31% and n=9, 

10%, respectively) when recording vaccines administered. However, users encountering these issues appeared

equally as likely to report a problem scanning and recording data from these labels, whether the barcode was 

faded (n=21, 78%) or smeared (n=7, 78%). Respondents provided write-in responses indicating that they 

perceived that light coloring and poor quality of printed 2D barcodes made it difficult for scanners to record 

the data. 

Nearly one-quarter of users with direct experience scanning 2D barcoded vaccines indicated encountering an 

incorrect entry at some point while recording the vaccines they administered (n=19, 22%) or added to vaccine 

inventory (n=8, 24%). Specific errors noted by respondents included missing or incorrect elements of lot 

number or expiration date, data entered into the wrong area of the EMR, receiving an error message when 

visual confirmation of information on vaccine label does not support, or reports of incorrect alignment of the 

product within the system; these were not instances that were observed regularly in the workflow analysis. 

Further analyses were conducted through cross-tabulation comparisons of incorrect entries and other factors. 

Previous experience with an incorrect entry was related to lower levels of satisfaction and more difficulties 

with ease of integration, compared to users who had not experienced this challenge. Thirty-three percent 

(n=6) of users who experienced an incorrect entry were satisfied, compared to 51% (n=35) of users who had 

not experienced an incorrect entry previously. Additionally, 37% (n=7) of users who experienced an incorrect 

entry agreed that it was easy to integrate 2D barcode scanning into their normal work procedures, compared 

to 59% (n=41) of users who had not experienced an incorrect entry previously.

Overall, users did not indicate experiencing challenges with the amount of space taken up by scanners, the 

location of scanners, or the number of scanners available at their facility. When challenges with equipment 

and placement did arise, they appeared to be related to processes involved with recording vaccine inventory 

rather than recording data about vaccines administered. 

Project participants identified a number of approaches for overcoming challenges, including: practice and 

repetition of scanning techniques, finding new ways to align the barcode and scanner, making modifications 

to the barcode or setup to aid scanning, sharing lessons learned with colleagues, rebooting the system, 

receiving additional training, and “being patient.”

Project participants identified several benefits of using 2D barcode scanning to record vaccine data. These 

benefits included: enhanced accuracy and completeness of data, improved efficiency and time savings, greater 

convenience and ability for less manual entry with 2D barcode scanning, improved patient safety, added 

Summary Report – September 2015
support with inventory tracking, a new ability to capture additional data elements in a single scan, and 

increased accountability. Additionally, many respondents indicated conditional benefits, where they saw the 

potential for benefits that had not yet been fully realized (most often due to challenges experienced, such as 

two data entry processes still in place, since not all vaccines are 2D barcoded, inconsistencies with scanning, 

or other challenges).

Question 2: What is the potential for sustaining 2D barcode scanning after the current project? 

The potential for sustaining 2D barcode scanning at participating facilities appears promising. Leaders 

described why it was likely (or unlikely) that their facilities would continue 2D barcode scanning after the end 

of the project and provided suggestions for changes to increase the likelihood of sustaining the practice of 2D 

barcode scanning. More than half of leaders noted that it was likely or extremely likely that their facility would 

continue scanning 2D barcodes after the project to record the vaccines they administer (n=42, 63%). 

However, the majority of leaders were neutral about continued 2D barcode scanning to record vaccine 

inventory (n=84, 59%). Reasons given for why their facilities were likely to continue scanning included 

increased accuracy and reduction in data entry errors when recording information about vaccines 

administered. Others remarked that scanning is a “time saver,” is “easy-to-use,” and “makes life easier.” 

Reasons for it being unlikely that the facility would continue 2D barcode scanning after the project related to 

challenges with the performance of the technology, problems in scanning, the additional time added when 

using 2D barcode scanning, and the lack of 2D barcodes on some vaccines.

More than half of leaders (n=40, 60%) and users (n=64, 55%) stated that additional 2D barcoded vaccines 

were necessary to sustain regular scanning activities by their facilities. When asked what percentage of 

vaccines would need to be 2D barcoded to sustain scanning at their facilities, a large majority of leaders 

(n=35, 88%) and many users (n=43, 68%) indicated at least 76% needed to be 2D barcoded12 (with n=7, 18% 

of leaders and n=24, 38% of users wanting 100% of vaccines 2D barcoded). Leaders and users also indicated 

a desire to have additional data elements populate upon scanning (n=24, 36% and n=40, 34%, respectively). 

Users in particular voiced a need for 2D barcode scanning to align better with their current workflow (n=43, 

37%). 

We also examined the likelihood of facilities continuing to scan 2D barcoded vaccines after the project with 

respect to various facility characteristics, respondent background characteristics, opportunities for 2D 

barcode scanning and patterns of use, and user experience with 2D barcode scanning during the project. 

Visual comparison of descriptive statistics indicates that leaders responding as follows may be more likely 

than their counterparts to express that it is likely or very likely that their facilities will continue using 2D 

barcode scanning after the project ends:

• Reported their facility manually typed data into the EMR system prior to the project

• Reported personally scanning 2D barcoded vaccines

• Expressed ease with integrating 2D barcodes into their facility’s existing work process

• Considered the benefits of 2D barcode scanning to be worth the change in process

• Reported being satisfied with the use of 2D barcode scanning at their facility, and

• Perceived that 2D barcode scanning saves staff time when recoding vaccine data

12 These percentages represent the combined response options of “Most (76-99%)” and “All (100%)”.

Summary Report – September 2015

 
Respondents provided suggestions for how to improve the process of recording vaccine data by scanning 2D 

barcodes. These included: improving efficiency and ease-of-use, improving barcode print quality, increasing 

the number of 2D barcoded vaccines, having additional data elements populate with a single scan, enhancing 

the workstation setup, enabling a safety alert to flag mismatched vaccine, improving the accuracy of 

information scanned into the system, providing additional information technology (IT) and training support, 

and enhancing EMRs.

Strengths and Limitations of the Evaluation

In reflecting upon the project and evaluation, we identified a number of strengths and limitations that are 

important to consider when interpreting aforementioned key findings. One of the lessons learned from our 

work is that great variation exists among the workflows, technology, and general operations at participating 

facilities. It is important to document these differences so they can be controlled for in data analyses or 

included in descriptions of the context within which data analyses should be interpreted. One way the project 

team attempted to capture this type of data was through the recruitment and installation logs. It should be 

noted that findings presented are not intended to be generalized beyond the current project or participating 

respondents and facilities involved in each data collection process. 

Strengths 

This project had a very high retention rate. Ninety three percent of the practices that initially enrolled in the 

pilot stayed involved until the end of the project. While we sought participation from a broader number of 

facilities, our final sample contained a variety of types of sites, including public health departments, 

independent and retail pharmacies, mass vaccinators and hospital based clinics. This allowed us to study the 

implementation and adoption of 2D barcode technology in a variety of diverse settings. Understanding the 

differences among settings can better inform stakeholders as to the ease of integration or challenges 

experienced in the field. In addition to the diversity of facility types, we were able to gather a large amount of 

vaccination data. Close to 1 million records were collected throughout the pilot period and included in the 

analysis. 

The completeness and representation of data used in this evaluation were strong. For example, one leader 

from 83% of participating facilities and at least one direct user from 78% of the participating facilities 

completed their respective Leader and User Surveys. Additionally, survey respondents were those who were 

most likely to engage with the 2D barcode scanning technology — individuals who reported personally 

recording vaccine administration or vaccine inventory data as well as persons who provide direct supervision 

of recording vaccines administered. 

The inclusion of Workflow Analysis site visits and data collection provided an on-the-ground experience to 

talk with practitioners and observe their work practices, including scanning 2D barcoded vaccines. The depth 

of insights we gained about the challenges users experienced and the approaches they used to address or work 

around these challenges would not have been possible with data collection solely from the online surveys. 

Observing the staff using the 2D barcode scanners while conducting the time measurements provided the 

WFA technicians an opportunity to better understand potential drivers of efficiencies or delays experienced. 

Follow up discussion during the staff interviews allowed for in depth discussion and further probing in areas 

of interest or unique observations. Insights gained during this process supported the interpretation of the 

Summary Report – September 2015
statistical analysis. Further, utilization of a rigorous yet simple and targeted time measurement protocol 

allowed for multiple high quality time measurements to be captured at each facility. 

The process maps were designed with the end goal in mind, which was to better understand the impact of 2D 

barcode scanning on the data entry process. In that sense, more detail is provided for steps impacting the data 

entry (e.g., fields populated, connecting scanner to laptop) and less detail was provided for steps unlikely to 

affect data entry (e.g., patient registration, consultation, etc.). To the extent possible, consistent terminology 

was used across the process maps to allow for better readability and process step comparisons across 

facilities, if needed. 

In addition to the number of records collected, the 2D barcode market has matured since prior evaluations. 

As of August 2015, there are 32 2D barcoded vaccine products. In previous evaluations, many facilities 

reported that they had not yet received 2D barcoded vaccines, whereas in this project, every facility had 

received at least one 2D barcoded product prior to the start of the evaluation period. Practitioners had more 

exposure to barcoded vaccines, more opportunities to scan barcoded vaccines and consequently greater 

opportunity to evaluate how 2D barcoding influenced their work flow and provided feedback on the benefits 

and limitations of 2D barcode scanning. 

A major strength of this project was the availability of a direct indicator of 2D barcode scanning to confirm 

that a 2D barcodes product was, in fact, scanned. A substantial proportion of participating facilities had a 

scan log, either native to their EMR or installed, and these logs directly indicated in the EMR which vaccines 

were scanned and which were entered through traditional methods. Previous projects had to rely on a 

composite indicator of multiple factors to estimate vaccine scanning. The direct indicator is a marked 

improvement and allows for greater certainty in the estimation of vaccines actually scanned and the 

contribution of 2D barcode scanning technology to lot number and expiration date data quality. 

In addition to validating scanning frequency data, the direct indicator of scanning also affords the ability to 

directly measure compliance. Previous projects relied on a self-reported measure of compliance while this 

project can calculate compliance directly. This allows for exploratory analysis related to factors that affect 

compliance and the testing of hypotheses around the adoption of 2D barcode scanning. This analysis can 

better inform stakeholders of potential barriers to 2D barcode scanning adoption and implementation. 

Limitations 

Although the data used for this report are rich and reasonably complete, there are some limitations to note. 

As noted previously, limitations with technology, such as the EMR limitations with product identifier or 

inconsistencies with scanner or 2D barcode quality, could create limitations in the project; however, even 

with these limitations, the results still demonstrated improvement in data quality. 

The consistency of data collected for the WFA might have been stronger had all site visits and data recording 

been performed by a single WFA technician instead of two. However, it should be noted that no statistical 

difference was found in time measurements between these two data collectors. Only a small number of 

facilities (n=22) were included in WFA segment of the project and only 16 of these facilities were included in 

the time measurement analysis13. Inclusion of additional facilities might have enriched the breadth of 

13 Reasons for the six facilities not included in the time measurement analyses included: 1) a facility that was included to pre-test the 

proposed methodology and data collection procedures for the WFA, 2) facilities with no or a very small volume of vaccines 

administered during the WFA visit, and 3) the facility having not yet reached a steady state with their new 2D barcode scanning data 

Summary Report – September 2015

 
experiences observed and feedback obtained. The intention with the inclusion of the WFA was not to visit 

each participating facility, but rather to gather detailed contextual and qualitative information from a smaller 

number of facilities; however, the selection of these facilities and small number of facilities visited limits the 

information obtained through this data source. 

Our team made every effort to select WFA facilities that were expected to have a sufficient daily volume of 

vaccines administered. The selection process included reviewing responses to the recruitment survey and 

direct communication with the facility point of contact to establish an estimate of the volume of 2D barcode 

scans performed per day/week. Despite these efforts, mock scans were needed during some WFA visits to 

reach the target volume for analysis. Although the WFA technician made efforts to control the measurement 

environment to be as close to the real case scenario as possible, the mock scans tended to be faster than the 

real ones; the difference of timed scans was not statistically significant14.

Fewer responses were received to the User Survey than we had hoped. Although at least one user from most 

facilities completed a User Survey, feedback from multiple users across a larger number of facilities might

have provided additional insights.

Selection bias is another factor to consider, both for project participation overall and specific to various data 

collection efforts. Participants determined whether they would respond to data collection activities such as 

Leader and User Surveys and WFA site visits. It may have been that participants with particularly positive or 

negative experiences with 2D barcode scanning were more inclined to participate more fully and provide 

feedback. The level of any bias this may have introduced is unknown. Further, survey responses and 

interviews during WFA were self-reported, which may have introduced some inaccuracies or bias.

We sought out participation from retail and pharmacy facilities to get a better idea of the use and 

implementation of 2D barcode scanning in these settings. We anticipated that adoption may be easier in these 

settings due to a high volume of a specific vaccine that is administered within a short period of time (i.e., 

influenza vaccine). Unfortunately, we were able to gather only a limited amount of data from these entities 

due to low response rate, minimal interest, and limited involvement from these types of facilities in both 

surveys and WFA participation (including lack of sufficient time measurements). This may indicate an 

opportunity for further needs assessment work.

The data completeness rate of facilities included in this project was high. In a comparison of baseline data 

quality, those facilities that did not complete the project had significantly worse baseline data quality than 

those that completed the current project. This may indicate the presence of a self-selection bias, whereby 

facilities that had poorer data quality at the start were less likely to complete the project, leaving those 

facilities with already higher data quality in the final sample. 

An additional limitation is that not all facilities included in the project had a direct indicator of scanning either 

installed by the project team for the purposes of this project or native to their EMR system. This resulted in 

some facility types being excluded from the data quality analysis conducted on sites with scan logs. Future 

entry process (including facing issues with staff not using the scanners consistently, not knowing how to scan properly or with 

scanners not being configured properly for 2D scanning).

14 Mock scan time measurements were compared to actual scans. Mock scans did not significantly affect time to enter vaccine 

administration data; therefore, the team considered them valid time measurements.

Summary Report – September 2015

 
efforts to evaluate 2D barcode technology could focus on those facilities such as pharmacies and military 

vaccinators that were not included in the final analysis. 

Conclusions

This project provides evidence of measured time savings and improved data quality associated with the use of 

2D barcode scanning to record vaccines administered. Although the time saved is relatively small and may 

not be meaningful for low volume practices, practices administering a large volume of vaccines or using 

manual entry to record vaccines administered may find particular incentive in the potential for time savings 

with the adoption of 2D barcode scanning. Given the limited number of pilot practices with inventory 

software capabilities, conclusions could not be drawn related to the use of 2D barcode scanning to record 

vaccine inventory. 

Practitioners who participated in the project acknowledged several realized or potential benefits of 2D 

barcode scanning for vaccine administration data entry, including increased accountability and convenience 

fostered by the new technology, in addition to the time savings compared with manual data entry. However

and perhaps most importantly, practitioners also indicate that for benefits to be realized, the technology needs 

to work as expected, which currently is not always the case. Additionally, barcode scanning technology is not 

universally accepted by facility staff and the compliance rate and perception of benefit varies with individual 

experience, training, and a number of other factors.

It is clear from this project that there are key technology-driven challenges that introduce process 

inefficiencies and prevent the immunizer community from fully embracing 2D barcode scanning. First, 

scanning is not always reliable, resulting in scan failures and repeat scans which require practitioners to 

manipulate (e.g., twist, turn) the product several times to obtain a successful scan. Nearly one-quarter of user 

respondents reported experiencing an incorrect entry at some point after scanning 2D barcodes, while 

recording vaccines administered. Issues with the print quality and fading of some barcodes and the curvature 

of barcode labels appears to exacerbate scanning reliability problems. Even when 2D barcodes are 

successfully scanned there are instances where incorrect data are entered in the EMR (e.g., wrong lot number, 

extra digit, and transposed expiration date); because there is not a standard format for lot numbers, EMRs 

cannot check for adherence to a format (as with a date format), therefore, there is nothing to prevent a user 

from manually entering data or changing what was captured. 

Furthermore, practitioners indicate that not having 2D barcodes on all vaccine products and not having 

additional data elements (e.g., manufacturer name, National Drug Code [NDC]) populated in the EMR with a 

single scan introduces inefficiencies into their process for recording vaccine data and can reduce compliance 

with barcode scanning among staff. Several challenges noted were not directly related to the process of 

scanning the 2D barcode, but instead related to the functionality of the EMR, configuration of the EMR or 

scanner, the change required in procedures, the availability of scanners in the facility, or the number of 2D 

barcoded vaccines present in the facility. 

Nonetheless, despite these challenges, project findings indicate that 2D barcode scanning improves lot 

number and expiration date data quality. Although levels of observed compliance were relatively low, we still 

measured improvement in correctness of lot number and expiration date data, compared with manual data 

entry. This improvement in data quality would like be greater with higher rates of compliance. Several factors 

appear to affect vaccine compliance, including staff training, vaccine type, time of year and vaccine 

Summary Report – September 2015
manufacturer. Additionally, compliance decreased over the life of the project. These factors suggest that 

workload and turnover, along with several factors identified through the surveys, affect the realization of 2D 

barcode technology benefits. 

Feedback from facility leaders and staff indicate a general receptiveness to 2D barcode scanning. Many survey 

respondents reported that it was easy to learn and use 2D barcode scanning technology in a short timeframe. 

Additionally, many participants appear satisfied overall with 2D barcode scanning and expressed a preference 

for 2D barcode scanning over other approaches that they had previously used for recording vaccine data. 

Most participants perceived that 2D barcode scanning improves the accuracy of vaccine data recorded and 

considered the benefits of 2D barcode scanning worth the change in process. Further, more than half of 

leaders indicated that it was likely or extremely likely that their facility would continue to use 2D barcode 

scanning after the end of the project. 

Despite the positive feedback by many project participants, there are several challenges that put the path to 

full adoption at risk. One of the intentions of the current project was to see whether user experiences differ in 

an environment where more 2D barcoded vaccines are in the supply chain than were available during the 

original 2D pilot project. While the number of 2D barcoded vaccines available and the proportion of total 

vaccines that are 2D barcoded has increased greatly since the previous 2D pilot project, staff continue to note 

that challenges remain because of the limited number of 2D barcoded vaccines available. The need to use two 

different methods to record vaccines based on whether they have a 2D barcode continues to negatively affect 

the sustained use of this technology. Project participants offered specific suggestions to increase the 

likelihood of sustaining the practice of 2D barcode scanning to record vaccine information, including: 2D 

barcodes on more (if not all) vaccine products, more data elements populated with a single scan, enhanced 

scanner ease of use and scanning reliability, and improved 2D barcode print quality. 

Recommendations 

Analyses of data gathered from project participants indicate several opportunities for improving the accuracy, 

time savings, and user experience of 2D barcode scanning to record vaccine data in healthcare facilities.

Suggested improvements that hold potential include: populating additional data elements, beyond lot number 

and expiration date, into EMRs with a single scan, having 2D barcodes on all vaccine products, and 

enhancing the scanning experience for users. Several members of the immunization community have an 

important role in taking action on these findings; however, there is a need to ensure that these actions are well 

coordinated to fully realize the anticipated benefits. We propose the following recommendations and task 

leads below (Table 5). 

Table 5. Recommendations to Improve 2D Barcode Scanning Accuracy, Time Savings and User 

Experience 

Recommendation Opportunities for Action Potential Leads

Fully integrate 2D 

barcode scanning 

with EMR systems 

Fully integrate 2D barcode scanning capabilities 

to expand information populated and benefits to 

providers 

• Enable population of lot number, 

expiration date, and NDC for all 2D 

barcode scans

EMR Vendors

Summary Report – September 2015
Recommendation Opportunities for Action Potential Leads

• Expand algorithms and look-up tables 

behind the scenes (to enable conversion 

of the product identifier (e.g. NDC )to 

derive additional information, such as 

manufacturer and dose)

• Add functionality or new field to allow 

practitioners to include notes or funding 

source field (such as indication of 

funding source, e.g. VFC) to eliminate 

need for data editing, which can 

introduce errors, reduce data quality, 

and hamper recall efforts.

• Provide support to practitioners to 

identify and address compatibility issues 

between 2D barcodes, 2D barcode 

scanners, and EMR system

Provide support and crosswalks for NDC and 

lot number patterns by manufacturer to make 

development of algorithms and look-up tables 

by EMR vendors easier and more readily 

available

CDC to maintain code set tables

Providers to communicate to 

EMR Vendors

Develop and promote a standard (or bestpractice) format for lot numbersVaccine Manufactures

FDA

Request incorporation of EMR scanning 

functionality in day to day interactions with 

EMR vendors.

Client facilities, practitioners and 

physician associations.

Increase the number 

of opportunities for 

scanning 2D 

barcoded vaccines 

by introducing 2D 

barcodes on all

Units of Use 

Practitioners to make request for 2D barcodes 

on all vaccine units of use to manufacturers

Vaccine Manufacturers to 

implement;

Practitioners and Associations 

make request

Alert the immunization community of the 

anticipated dates each vaccine product will 

include a 2D barcode

Vaccine Manufacturers, 

CDC to publish on their website

Develop communications materials to encourage 

the use of 2D Barcodes on vaccines and 

promote consistency of use among providers. 

CDC to develop materials;

Practitioners and American 

Academy of Pediatrics (AAP) and 

other practitioner associations to 

encourage use

Take actions to 

improve scanning 

experience (both 

reliability/ 

consistency and ease 

of use) 

Enhance the reliability of 2D barcode scanners 

in order to provide a more consistent scanning 

experience for users.

• Share approaches to address known 

issues (such as a transposed 7 and &)

• Research improvements for scanning 

vials and syringes

CDC to investigate frequency of 

occurrence of scan issues; 

2D Scanner Vendors to review 

and adjust technology

Summary Report – September 2015
Recommendation Opportunities for Action Potential Leads

Test and subsequently address barcode issues 

that hinder scanning success. Indications from 

the current data suggest that the following items 

require further examination but have the 

potential to lead to improved scanning:

• Enhancing the quality of barcodes by 

improving the quality and resolution of 

printing and including high contrast 

images 

• Altering the barcode style in ways that 

may aid scanning such as increasing the 

placement of the barcode on the label or 

utilizing alternatives to traditional labels 

(e.g., through using peel-off labels or 

labels that unfold and can be flattened 

easily during scanning)

Vaccine Manufacturers and 

contracted printers – continue to 

make improvements to printing 

and production process

FDA – label format approval

CDC in collaboration with 

partners and GS1 to identify ideal 

image size, color, and placement. 

Make training and technical assistance or 

support resources available on the use of 2D 

barcode scanning and scanners and resolution of 

challenges that may arise

CDC – online references;

2D Scanner Vendors – technical 

assistance; 

Physician associations – share best 

practices

Develop a barcode scanning protocol to include:

• Staff training on the use of 2D barcode 

scanners and barcode scanning 

protocol. 

• Scanner testing to ensure that scanners 

are working properly though proper 

handling and regularly testing (e.g: scan 

into a word document to look at 

output). 

• Support information for issues

Leaders at facilities

Practitioner associations

Pharmacy associations

Identify, develop, disseminate, and use best 

practices for use of 2D scanners in vaccination 

settings.

AAP or other associations to 

disseminate; 

CDC to publish on website 

Summary Report – September 2015