Implementation Pilot for TwoDimensional (2D) Vaccine Barcode 

Utilization: 

Additional Workflow Analysis 

(WFA) Final Report 

Prepared for 

Immunization Services Division 

National Center for Immunization and Respiratory Diseases

Centers for Disease Control and Prevention 

Prepared by 

Deloitte Consulting LLP 

October 2014 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Contents 

Executive Summary............................................................................................................................................. 3
 

1. Introduction ................................................................................................................................................ 7
 

2. Methodology............................................................................................................................................... 8
 

2.1. Selection of practices .......................................................................................................................... 8
 

2.2. Data collection .................................................................................................................................... 9
 

2.3. Data analysis ..................................................................................................................................... 14
 

3. Results....................................................................................................................................................... 15
 

3.1. Practice characteristics ..................................................................................................................... 15
 

3.2. Vaccine-specific administration and inventory workflows ............................................................... 17
 

3.2.1. Vaccine administration ............................................................................................................. 17
 

3.2.2. Vaccine inventory...................................................................................................................... 18
 

3.3. Changes in amount of time to record vaccine data.......................................................................... 18
 

3.3.1. Vaccine administration ............................................................................................................. 19
 

3.3.2. Vaccine inventory...................................................................................................................... 23
 

3.4. Staff opinions about 2D barcode scanning ....................................................................................... 24
 

4. Strengths and Limitations ......................................................................................................................... 29
 

5. Conclusions and Recommendations ......................................................................................................... 30
 

Appendix A: Email Disseminated to Candidate Practices ................................................................................. 34
 

Appendix B: Interview Guide ............................................................................................................................ 36
 

Appendix C: Practice Summaries ..................................................................................................................... 38
 

Practice ID: 1187 ........................................................................................................................................... 38
 

Practice ID: 2091 ........................................................................................................................................... 49
 

Practice ID: 2133 ........................................................................................................................................... 59
 

Practice ID: 2168 ........................................................................................................................................... 64
 

Practice ID: 2299 ........................................................................................................................................... 69
 

Practice ID: 2413 ........................................................................................................................................... 74
 

Practice ID: 2429 ........................................................................................................................................... 80
 

Practice ID: 2452 ........................................................................................................................................... 87
 

Practice ID: 2508 ........................................................................................................................................... 94
 

Practice ID: 2510 ........................................................................................................................................... 99
 

2 
 

 

 

 

 

 

 

 

 

 

 

 

 

Executive Summary 

In 2011, CDC’s Immunization Services Division within the National Center for Immunization and Respiratory 

Diseases (CDC/NCIRD/ISD) contracted with Deloitte Consulting (Deloitte) to conduct the Implementation 

Pilot for Two-Dimensional (2D) Vaccine Barcode Utilization. As part of that pilot project, Deloitte performed 

workflow analyses (WFAs) in a subset of practices to describe how the introduction of 2D barcoded vaccine 

scanning changed vaccination administration and inventory processes and affected the time it takes 

practitioners to record vaccine data (i.e., lot number, expiration date, and product identifier) . Although 

valuable lessons were learned during these WFAs, the limited number of 2D barcoded vaccines available 

during the pilot project observation period1 hindered the ability to develop a full understanding of the effect 

of 2D barcodes on recording data about vaccine administrations and vaccine inventory. As of May 2014, 

multiple 2D barcoded presentations of vaccines which were not available in the original pilot timeframe had 

come to market, therefore, Deloitte performed a second set of WFAs to answer the following questions: 

(1) In what ways, if any, does the use of 2D barcoded vaccines affect the workflow procedures
 

associated with recording vaccine data as part of the vaccine administration and inventory
 

processes?
 

(2) Does the use of 2D barcodes to record vaccine data during the vaccine administration and inventory 

processes change the amount of time it takes to record vaccine data? If so: 

a.	 What is the direction and magnitude of this change? 

b.	 What factors contribute to the changes in recording time? 

c.	 How do these time differences relate to modifications made in the recording procedures 

when using 2D barcode scanning? 

(3) What are staff opinions about the use of 2D barcode scanning to record vaccine data in their 

practice? 

a.	 To what extent do staff report improvements in the use of 2D barcode scanning given the 

increase in 2D barcoded vaccines in their practice? 

b.	 What has facilitated the continued use of 2D barcode scanning to record vaccine data in the 

practice and what challenges remain? 

Methods 

To be included in the current WFA, practices included in the original pilot had to meet the following 

inclusion criteria: (1) All above the 33rd percentile in estimated volume of 2D barcoded vaccines 

administered in their practice, (2) Continued use of 2D barcode scanners to record data for vaccines 

administered after the data collection phase of the pilot ended in March/April 2013, (3) Encountered 2D 

barcoded vaccines more frequently than once a month, and (4) Expressed an interest in participating in the 

1 By the end of the implementation period the majority of practices (n=111, 71%) received 2D barcoded Menactra; however, other 

2D barcoded vaccines were infrequently received during the original pilot. 

3 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

WFA. Our team screened the 217 practices from the original pilot project for their eligibility using existing 

data and requesting additional information via email or phone. 

Ten practices were selected for inclusion in the current WFA, as well as one practice that served to pilot test 

the proposed WFA data collection procedures. One project team member conducted all WFA practice visits. 

This WFA consultant performed three data collection activities: (1) Observations of the vaccine-specific 

administration and inventory workflow processes including data entry, (2) Time measurements for each step 

in these workflow processes as well as specifically for entering vaccine data using 2D barcode scanning and 

traditional methods of data entry and, (3) Semi-structured interviews with practice staff with direct 

experience recording vaccine administration or vaccine inventory data with 2D barcode scanners . 

The WFA consultant used the detailed notes and time measurements taken during observations of the 

entire workflow process within each practice (for vaccine administrations and vaccine inventory) to develop 

process maps. The analysis team reviewed these process maps and extracted information about how the 

workflow processes differed when linear barcoded vaccines or 2D barcoded vaccines were in use. We 

identified the steps in the workflow that were commonly added or omitted between and within practices 

when using vaccines with different types of barcodes. A member of the analysis team transcribed digital 

recordings of the interviews, extracted, and coded relevant text from each transcript in response to the third 

evaluation question noted above. 

We analyzed time measurements using descriptive statistics, and then tested for differences between the 

average time to record vaccine administration data using 2D barcode scanning compared to traditional data 

entry methods within practices with a t-test. We used the Wilcoxon Signed-Rank Test to examine whether 

this difference was statistically significantly different at the aggregate practice level. Additionally, to account 

for the effects of potential confounding factors, we performed random intercept linear mixed effects 

analysis of the relationship between time to record vaccine data and barcode type. Due to the limited 

sample size, we used descriptive statistics to examine differences in the time to record vaccine data for 

inventory when using 2D barcode scanning relative to traditional data entry methods. 

Results 

The majority of WFA practices specialized in pediatrics (n=6, 60%), were small in size (five or fewer 

physicians) (n=6, 60%), and used 2D barcode scanners only to record vaccine administrations (n=7, 70%). 

Two of the 10 participating practices had an EMR that is 2D barcode capable2

. In over half of the WFA 

practices, approximately 26% to 50% of their vaccine supply is public purchase (n=6, 60%). 

The workflow procedures associated with patient visits where one or more vaccines were administered 

were more likely than the processes associated with vaccine inventory to differ with the introduction of 2D 

barcode scanning. However, minimal changes occurred in this procedure for the 10 practices observed. All 

practices, with the exception of one, had 2D barcode scanners located at every data entry site in the 

practice. As a result, no changes beyond those associated with physically scanning the barcode occurred in 

2 2D capable means an EHR can process the scan from a 2D barcode scanner (not configured using our configuration utility) to 

deposit the data into their system. 

4 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

the overall workflow procedures when a 2D barcode vaccine was recorded. The WFA consultant observed 

three practices performing inventory data entry. In general, the procedures associated with recording 

vaccine inventory changed minimally in these three practices between the activities performed for recording 

data from linear barcoded vaccines and 2D barcoded vaccines—with the major difference being the use of a 

2D barcode scanner. 

Across all 10 practices, the average time to record vaccine administration data using 2D barcode scanning 

was 26.75 seconds and the average time to record vaccine administration data from a linear barcode using 

traditional methods was 28.47 seconds. This difference was not statistically significantly different3

. The 

largest predictor of differences in time to record vaccine data in the random intercept linear mixed effects 

analysis was administering nurse—this explained 17% of the variation in time. Practice and vaccine type 

both individually explain 9% of the variability in time to record vaccine data. 

For five practices, the average time to record data about vaccines administered was lower when using 2D 

barcode scanners than recording data from linear barcoded vaccines using traditional methods. For five 

practices, the average time to record data about vaccines administered increased. Mean differences 

between 2D barcode scanning and using the traditional method of data entry to record data for vaccines 

administered ranged between 16.68 seconds to -16.79 seconds. Only one practice experienced a 

statistically significant decrease in the average time to record data for vaccines administered, where the 

average time to record using traditional methods was 44.55 seconds (SD=16.39) and the average time to 

record using 2D barcode scanning was 27.76 seconds (SD=16.39). 

Many individuals interviewed were not certain as to why their practice continued using 2D barcode scanning 

after the original pilot project came to a close (n=13 interviews of 264

). Interviewees who provided a 

specific answer about why their practice continued to scan 2D barcodes to record vaccine data most 

frequently noted a perceived increase, or potential for improved, accuracy of records (n=11 interviews). In 

21 of 27 interviews, individuals conveyed that staff had noticed an increase in the number of vaccines with 

2D barcodes since the pilot end date; however, opinions were mixed about the extent to which this increase 

had improved the process of using 2D barcode scanners to record vaccine data. The most frequent benefit 

of using 2D barcode scanning mentioned by interviewees was a perceived improvement in the accuracy of 

vaccine data. The most frequent challenge cited was difficulties with scanning barcodes and the level of 

inconsistency with which the scanning process actually works. 

Conclusions and recommendations 

This WFA provides some limited indications that 2D barcode scanning can save time when entering data 

about vaccines administered. The use of 2D barcode scanning in one of the 10 practices resulted in a 

3 V=344, p=0.437 4 Interviews performed were semi-structured in nature. This format allowed the interviewer to adapt the interview as 

needed based upon the context. Therefore, in some instances all questions on the interview guide were not asked in 

every interview. 

5 
 

 

 

 

 

 

 

 

 

 

 

 

 

statistically significant decrease in average time to record vaccine data. We hypothesize that such 

differences may become more widespread when the practice of 2D barcode scanning to record vaccine data 

is common in health care practices and encourage future research once all vaccines are 2D barcoded. In the 

interim, there are several findings from this WFA that suggest steps that could improve the process of using 

2D barcode scanning to record vaccine data in health care practices. Based upon these findings we 

recommend that: 

1.	 Practices considering 2D scanning adoption should invest time up front to plan how 2D barcode 

scanning will work best with their operations to help determine the appropriate number and 

placement of scanners so as not to introduce operational impediments. 

2.	 The CDC and FDA should collaborate with the manufactures to investigate the quality and 

consistency of the 2D barcode labeling. Consistent reports from interviewees that there is 

variability in the ease of scanning 2D barcodes by vaccine type warrants further investigations about 

the quality of 2D barcode labeling and the ability to consistently produce batches with labels that 

practitioners can easily scan. 

3.	 The EMR software should be configured or set up to read the entire lot number which will prevent 

default selection by the system of lots with similar first characters. Solution vendors incorporating 

2D barcode scanning of vaccines into their applications should read the entire lot number from the 

encoded data string. 

4.	 Industry should further investigate inconsistencies of 2D barcode scanning. Non-specific, but 

frequent, concerns from interviewees about the inconsistency of 2D barcode scanning suggests that 

follow-up product evaluations that examine the possible root causes of these reported 

inconsistencies would be helpful. Such investigations might examine the comparative effectiveness 

of 2D barcode scanning using a variety of conditions, such as: (a) placement of the barcode in 

different locations on the label, (b) other quality features of barcodes (e.g., different background 

colors, ink used for 2D barcode), or (c) peel-off labels versus labels on vials or syringes. 

6 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

1. Introduction 

In 2011, CDC’s Immunization Services Division within the National Center for Immunization and Respiratory 

Diseases (CDC/NCIRD/ISD) contracted with Deloitte Consulting (Deloitte) to conduct the Implementation 

Pilot for Two-Dimensional (2D) Vaccine Barcode Utilization. As part of that pilot project, Deloitte performed 

workflow analyses (WFAs) in a subset of practices to describe how the introduction of 2D barcoded vaccine 

scanning changed vaccination administration and inventory processes and affected the time it takes 

practitioners to record vaccine data (i.e., lot number, expiration date, and product identifier). Although 

valuable lessons were learned during these WFAs, the limited number of 2D barcoded vaccines available 

during the pilot project observation period5 hindered the ability to develop a full understanding of the effect 

of 2D barcodes on recording data about vaccine administrations and vaccine inventory. 

As of May 2014, multiple 2D barcoded presentations of vaccines which were not available in the original 

pilot timeframe had come to market. This development provided an opportunity to examine the potential 

effects of 2D barcode scanning in an environment where 2D barcode vaccine volume has increased. As a 

result, Deloitte performed a second set of WFAs from June-July 2014 with a new subset of the 217 practices 

from the pilot project. In this report, we present the findings from this second WFA, specifically addressing 

the following questions: 

(1) In what ways, if any, does the use of 2D barcoded vaccines affect the workflow procedures
 

associated with recording vaccine data as part of the vaccine administration and inventory
 

processes?
 

(2) Does the use of 2D barcodes to record vaccine data during the vaccine administration and inventory 

processes change the amount of time it takes to record vaccine data? If so: 

a.	 What is the direction and magnitude of this change? 

b.	 What factors contribute to the changes in recording time? 

c.	 How do these time differences relate to modifications made in the recording procedures 

when using 2D barcode scanning? 

(3) What are staff opinions about the use of 2D barcode scanning to record vaccine data in their 

practice? 

a.	 To what extent do staff report improvements in the use of 2D barcode scanning given the 

increase in 2D barcoded vaccines in their practice? 

b.	 What has facilitated the continued use of 2D barcode scanning to record vaccine data in the 

practice and what challenges remain? 

5 By the end of the implementation period the majority of practices (n=111, 71%) received 2D barcoded Menactra; however, other 

2D barcoded vaccines were infrequently received during the original pilot. 

7 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

2. Methodology 

2.1. Selection of practices 

The target number of practices for participation in this WFA was 10. To select practices for inclusion, we 

assessed several factors. The first factor of interest in this WFA was the anticipated volume of 2D barcoded 

vaccines administered by the practice. Our goal was to obtain participation from practices that likely 

administer the largest volume of 2D barcoded vaccines out of all practices that participated in the pilot 

project. Due to the timeline for the current WFA and the effort that would be involved in obtaining data 

from each pilot practice regarding the volume of 2D barcoded vaccines they administered in the months 

directly preceding this WFA, we leveraged existing data from the pilot project to generate an estimate of the 

possible 2D barcoded vaccine administration volume within each practice. 

To estimate the volume of 2D barcoded vaccines administered, our team obtained the National Drug Code 

(NDC) from manufacturers for all 2D barcoded vaccines as of May 7, 2014. We then used the NDC to 

identify the brand and type of vaccines. Our team used the NDC, brand, manufacturer, and vaccine type to 

identify and label each vaccine administered in the pilot project Immunization Information Systems (IIS) data 

file as 2D barcoded or not. Seasonal vaccine products, specifically influenza vaccines, were excluded from 

this file since practices frequently change the manufacturers from which they order product between 

seasons. Subsequently, we ran a frequency distribution to group practices into three categories – low, 

medium, and high volumes of anticipated 2D barcoded vaccines administered. Practices at or below the 33rd 

percentile in anticipated volume based upon the number of 2D barcoded, non-influenza, vaccines they 

administered between November 2011 and March/April 2013 were categorized as low volume (range: 2 to 

625 vaccines administered), those above the 33rd percentile and at or below the 66th percentile were 

categorized as medium volume (range: 628 to 1,919 vaccines administered), and those above the 66th 

percentile were categorized as high volume (range: 2,013 to 33,165 vaccines administered). 

Subsequently, our team contacted practices assigned to the high or medium volume categories via email. 

This email included a brief description of the purpose and timing of the WFA, and asked for additional 

information from the practice to determine their eligibility for participation (Appendix A). Practices were 

eligible for inclusion if they reported: 

1.	 Continued use of 2D barcode scanners to record data for vaccines administered after the pilot 

ended in March/April 2013 

2.	 Encountering 2D barcoded vaccines more frequently than once a month 

3.	 Interest in participating in the WFA 

Members of our team placed at least one phone call to each of the practices that did not respond to the 

email as well as to practices that did respond and met the eligibility criteria. For practices meeting the 

eligibility criteria, we requested additional data about the practice including: (1) the name of their EMR, (2) 

the number of physicians in the practice, (3) the approximate percentage of vaccines that are privatepurchase, (4) the approximate percentage of vaccines that are public-purchase, (5) when practice staff 

8 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

typically record data about vaccines administered, and (6) where 2D barcode scanners used for recording 

vaccine administrations are located within the practice. The original intention of gathering this additional 

data was to provide further information helpful to selecting the practices for the WFA (e.g., obtain an even 

split of practices with 2D enabled EMR systems vs. non-2D enabled). 

2.2. Data collection 

One project team member conducted all WFA practice visits. Prior to the in-person visits, the WFA 

consultant met with the Deloitte team project manager and lead evaluator to become familiar with the 

current protocol, the specific data collection and documentation procedures, and who to contact in the 

event questions arose in the field. The WFA consultant also spoke with individuals who conducted WFAs for 

the original pilot to become familiar with the lessons they learned. The first WFA visit was used as a “pilot 

test” for the procedures outlined in the project protocol6

. Following this initial pilot visit, the Deloitte lead 

evaluator, project manager, and WFA consultant discussed the feasibility of the protocol procedures and 

revised the data collection procedures to improve data documentation. 

The consultant performed three data collection activities for the WFA: (1) Observations of the vaccinespecific administration and inventory workflow processes including data entry, (2) Time measurements for 

each step in these workflow processes as well as specifically for entering vaccine data and, (3) Interviews 

with practice staff. Each data collection activity is described in greater detail below. 

2.2.1. Observations of vaccine-specific administration and inventory workflows 

The WFA consultant performed observations of vaccine administration and vaccine inventory (where 

applicable) processes while on-site at each practice. For these observations the consultant shadowed at 

least two staff members7 while they performed the entire process associated with patient visits in which 

one or more vaccines were administered, and at least one individual while they performed the process 

involved with recording vaccine inventory data (where applicable). The process for selecting individuals to 

observe varied slightly by practice; however, the general rule followed by the WFA consultant was to 

observe all individuals who administered vaccines and were working during the days of the WFA visit. The 

WFA consultant typically did not observe individuals who noted they never personally use the 2D barcode 

scanners to record vaccine data. 

For both the vaccine administration and vaccine inventory workflows, the WFA consultant documented each 

step in the workflow process for each observation instance and assigned a unique identifier code that 

represented both the observed staff member and the observation instance. Our team also gathered 

additional descriptive information for each observation instance (Table 1). In addition to these elements, 

6 Since this practice was used as a pilot data collection effort a limited amount of data obtained is included in our analyses.
 

Specifically, we included feedback obtained through staff interviews. 7 One exception to this process occurred at the WFA visit to practice 2299. There is only one nurse who administers vaccines in this
 

office, therefore the WFA consultant observed only this individual. A medical assistant who does not administer vaccines on a
 

typical day, but does so when the nurse is not in the office was asked to participate in mock scans.
 

9 
 

•

•

•

•

•

•

•

 

 

 

 

 

 

 

 

the WFA consultant recorded the practice identification number, the name of the EMR used, and the 

traditional procedures used to record vaccine data (e.g., drop down, manual entry) for each practice visited. 

Table 1. WFA data elements gathered during observations of workflow processes 

Process Observed Data Elements Collected 

Vaccine 

Administration 

• Unique identifier for staff member being observed 

Number of vaccines administered to patient 

Names of vaccines 

Number of 2D barcoded vaccines administered to patient 

Number of linear barcoded vaccines administered to patient 

Name of the key step in the process under observation* 

Length of time (in seconds) for the key steps in the process 

When vaccine administration data were recorded (Before or after administration) 

Physical location where vaccine data were entered into the EMR 

Additional comments regarding process or WFA data collection 

•

•

•

•

•

•

•

•

•

Vaccine Inventory Unique identifier for staff member being observed 

Type of vaccine being recorded into inventory (linear or 2D) 

Name of vaccine being recorded into inventory (e.g., Menactra) 

Name of the step in the process 

Length of time (in seconds) for the key steps in the process 

Physical location where vaccine data were entered into inventory 

Additional comments regarding process or WFA data collection 

*General steps as defined by the consultant based on each site’s workflow process. 

2.2.2. Time measurements 

The WFA consultant recorded time measurements associated with the four processes depicted in Table 2. 

Specifically, the consultant recorded times for each key step in the entire process associated with patient 

visits in which one or more vaccines were administered, and for the process involved with recording vaccine 

inventory data (where applicable). Additionally, the consultant recorded the time it took practitioners to 

enter data, either by scanning or traditional methods, for each vaccine that was administered to a patient or 

recorded for the purpose of vaccine inventory. The WFA consultant recorded times using TimeStudy,8

an 

application specifically designed for use in time and motion studies. 

8

Information about TimeStudy is provided at: https://itunes.apple.com/us/app/timestudy-by-nuvizz/id504948284?mt=8 

10 
 

 

 

 

 

 

 

 

 

 

Table 2. Processes for which time was measured with start and stop times 

Process * Observation start Observation end 

1. Vaccine inventory 

process 

Retrieve vaccine from shipment box Save vaccine inventory record in 

electronic system 

2. Vaccine-specific 

patient visit 

Review patient chart† Save patient record in electronic 

system 

3. Data entry for vaccine 

inventory 

For 2D barcoded vaccines this included 

picking up the scanner. For linear 

barcoded vaccines this is entering the 

lot number and expiration date 

Complete data entry of data element 

of interest in 2D barcode scanningeither lot number or expiration date 

4. Data entry for 

vaccines administered 

For 2D barcoded vaccines this included 

picking up the scanner. For linear 

barcoded vaccines this is entering the 

lot number and expiration date 

Complete data entry of data element 

of interest in 2D barcode scanningeither lot number or expiration date 

*

Each process listed was timed for both 2D barcoded and linear barcoded vaccines. For the second process listed – “vaccine-specific 

patient visit”—practitioners often administered more than one vaccine. Therefore observations included those in which the 

practitioner administered only linear barcoded vaccines, only 2D barcoded vaccines, or a mixture of linear and 2D barcoded 

vaccines. †

The start times for the vaccine-specific patient visit frequently varied as these depended upon the process employed 

within a given practice. 

After observing the workflow process to garner a general understanding of the typical steps and procedures, 

the WFA consultant timed as many instances of vaccine administrations and vaccine inventory as possible 

with a goal of 30 observations each for both 2D barcode scanning and traditional methods in each 

participating practice. In small practices, the consultant attempted to capture times associated with all 

vaccine-specific patient visits that took place during the days of the WFA visit. In larger practices with more 

than one practitioner, the WFA consultant attempted to maximize the number of vaccine-specific patient 

visits observed while also spreading these observations across the practitioners, in an effort to obtain 

approximately the same number of observations per practitioner. 

Table 3 provides an example of how time measures were recorded for each step of the workflow process 

associated with one vaccine-specific patient visit. The specific steps recorded in the first column varied by 

each observation instance however, consistent terminology for similar steps was used throughout the data 

collection effort. We used a similar procedure to document times associated with the steps involved in 

vaccine inventory. 

11 
 

 

 

 

 

 

 

 

 

 

 

 

Table 3.Example of time measurements of vaccine administration process 

Step in Process* Time (sec) 

Patient counseling 5.21 

Login 32.13 

Data Entry 17.16 

Walking 10.36 

Wash hands 33.64 

Vaccine Prep 59.28 

Data Entry 9.11 

Linear input 21.13 

2D input 25.78 *

The “2D input” step in this chart refers to the specific time it took 

to perform the steps specific only to 2D scanning: picking up the 

scanner, scanning the barcode for lot # and expiration date, and 

pressing the ‘scan’ button when necessary. The “linear input” steps 

refer to the specific steps that would be performed in the 

traditional method instead of the 2D scanning process—so either 

manually typing or selecting from a drop box the lot number and 

expiration date. The “Data Entry” step refers to the steps that are 

performed for both processes-such as typing or drop down 

selection of manufacturer, input location, etc. The WFA consultant 

wanted to single out the steps that are directly comparable to 

measure the differences for the two processes: linear vs. 2D. 

When documenting the time associated with recording vaccine data elements for a single vaccine 

administered or entered into inventory, the WFA consultant also documented several variables of potential 

interest to the analysis (Table 4). Per the WFA protocol, our intention was to record 30 time measurements 

of the linear barcode recording process and 30 time measurements of the 2D barcode recording process in 

each location (administration and inventory) per facility (Table 5). In the event that 2D barcoded vaccines 

were not administered, or administered with limited frequency, during the date and time of the visit the 

WFA consultant requested practitioners engage in “mock” scanning procedures. Towards the end of the 

WFA visit, the WFA consultant reviewed the number of observations made and worked with those who she 

was observing to obtain time measures for mock procedures9

. It should be noted, however, that three 

practices were uncomfortable with the mock scanning process and declined to participate due to concerns 

about the mock scans introducing data that might interfere with the inventory reconciliation and 

management process. The decision to engage in mock scanning was usually dependent on whether the 

EMR system easily allowed the staff to create a mock patient (which the EMR knew not to subtract the 

inventory for) or whether the staff could easily perform the process completely without saving the vaccine 

administration into the EMR. 

Table 4. Variables recorded for instances of vaccine data entry (administration and inventory) 

9 Detailed information about the mock data entry procedures are provided in the WFA protocol. 

12 
 

 

 

 

 

 

 

 

 

 

 

 

 

Variable Description 

Staff ID Unique identifier for staff member observed 

Vaccine Name Name of the vaccine (e.g., Menactra) 

Barcode Type 2D barcode or linear barcode 

Procedure Process observed—administration or inventory 

Real or mock scan Real or mock scan observed 

Time (seconds) Time in seconds that elapses between the instant when a staff member 

accesses the vaccine administration or vaccine inventory module in their 

electronic system to enter data until all data is entered for that vaccine. 

Physical location where vaccine was 

recorded- administration 

Location in practice where vaccine data was recorded. Options include: nurses 

station; refrigeration area/prep station; patient area/point of care; other 

When vaccine administration data 

was recorded 

Before or after vaccine was administered 

Physical location where vaccine was 

recorded- inventory 

Text based description 

Comments Any additional comments about the process observed or efforts to record 

information about the data entry process 

Table 5. Proposed number of observations per process for data entry time measures 

Process Administration Inventory 

2D barcode scanning 30 30 

Traditional method 30 30 

Total (practice- level) 60 60 

2.2.3. Staff Interviews 

The WFA consultant conducted in-person, semi-structured interviews with a subset of staff at each practice. 

Specifically, this activity targeted performing interviews with three10 to five individuals who had direct 

experience recording vaccine administration or vaccine inventory (where applicable) data with 2D barcode 

scanners. Interviewees were selected for participation based upon their schedule availability, and in some 

practices, based upon the preference of the practice’s point of contact. On occasion we conducted 

interviews with a representative from information technology or a nurse manager who provided oversight 

to the medical staff depending upon what seemed most appropriate in the practice context. Given the busy 

nature of the practices, we ensured that no interviews exceeded 15 minutes in duration. The WFA 

consultant obtained permission to digitally record the interview. Our team transcribed the interviews prior 

to analysis. In the event that a digital recording was unavailable, the team used notes taken during the 

10 These estimates assumed that the facility was large enough to have at least three individuals who were directly involved in 

recording vaccine administration or vaccine inventory data. In the event this assumption was violated we proposed to speak with all 

individuals who were involved in this process (i.e., one or two persons). 

13 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

course of the interview in the analysis11. The semi-structured interview guide used by the WFA consultant 

for these staff interviews is provided in Appendix B. 

2.3. Data analysis 

2.3.1. Comparison of vaccine-specific administration and inventory workflows 

One primary question of interest in this evaluation is—“ In what ways, if any, does the use of 2D barcoded 

vaccines affect the workflow procedures associated with recording vaccine data as part of the vaccine 

administration and inventory processes?” The WFA consultant used the detailed notes and time 

measurements taken during observations of the entire workflow process within each practice (for vaccine 

administrations and vaccine inventory [where applicable]) to develop process maps. The analysis team 

reviewed these process maps and extracted information about how the workflow processes differed when 

linear barcoded vaccines or 2D barcoded vaccines were in use. We identified the steps in the workflow that 

were commonly added or omitted between practices when using vaccines with different types of barcodes 

and present these findings in the results section. We also report on unique changes to individual practice 

workflows. Appendix C includes process maps and detailed case descriptions for each practice. 

2.3.2. Changes in amount of time to record vaccine data 

Another central question of interest to this evaluation concerns whether or not using a 2D barcode scanner 

to record vaccine data is more or less expedient than entering data from vaccines without a 2D barcode – 

“Does the use of 2D barcodes to record vaccine data during the vaccine administration and inventory 

processes change the amount of time it takes to record vaccine data?” To address this question, the 

analysis team combined and manually cleaned the datasets created for each practice during the WFA visits 

and performed descriptive analyses12 on the data to examine measures of central tendency, dispersion, and 

shape of the distribution. The team used these descriptive analyses to consider whether transformation 

procedures were needed for any of the key variables prior to producing regression models. 

Since the underlying assumptions of the parametric t-test were not met, the analysis team used the nonparametric Wilcoxon Signed-Rank Test to examine whether differences existed between the time to record 

data using 2D barcode scanners compared to recording data using tradition methods in the aggregated 

practice data. Additionally, to account for the effects of potential confounding factors, we performed 

random intercept linear mixed effects analysis of the relationship between time to record vaccine data and 

barcode type. As fixed effects, we entered barcode and EMR into the model. As random effects, we had 

intercepts for vaccine name, staff ID and practice ID. To adhere to the model’s assumption of linearity in the 

outcome measure, time, we performed a log transformation of time to adjust for right skew in the time 

variable. The final model took the form of: 

yy (LLLLLL LLoo tttttttt) = ββ0 + ββ1 ∗ (2DD bbbbbbbbLLbbttbb) + ββ2 ∗ (EEEEEE) + υυ(VVbbbbbbttVVtt) + υυ(PPbbbbbbttttbbtt) + υυ(NNNNbbNNtt) εε

11 This situation only occurred for a portion of one interview; all others included a complete digital record.
 12 Included measures of central tendency (mean, median, and mode), dispersion (standard deviation), and shape (skew and
 

kurtosis). Additionally we examined visual plots of data including histograms and boxplots.
 

14 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

2.3.3. Staff opinions about 2D barcode scanning 

The final question of interest in this evaluation relates to the staff’s opinions of using 2D barcode scanning 

to record vaccine data. As previously noted, the WFA consultant took detailed notes and digital recordings 

during the interviews. Prior to performing analysis of the interviews, a member of the analysis team 

transcribed each recording and extracted relevant text from each transcript that included opinions from 

staff regarding the following topics: 

•	 Factors that facilitated the continued use of 2D barcode scanning to record vaccine data 

•	 Challenges that remain in using 2D barcode scanning to record vaccine data 

•	 Reported improvements in using 2D barcode scanning in a context where more vaccines are 2D 

barcoded than during the pilot project observation period 

•	 Benefits of recording vaccine data using 2D barcode scanning as compared to other methods 

•	 Drawbacks of recording vaccine data using 2D barcode scanning as compared to other methods 

The analyst further examined these excerpts to identify additional themes within these categories (e.g., 

commonly held opinions about benefits). 

3. Results 

In this section we provide an overview of the characteristics of practices participating in the WFA as well as a 

summary of each of the evaluation questions posed for this WFA. We provide detailed, de-identified 

summaries for each practice visited as part of the WFA, in Appendix C. 

3.1. Practice characteristics 

Of the 217 practices that participated in the original pilot, we estimated that 145 would likely administer a 

high or medium volume of vaccines that were currently 2D barcoded. Each of these practices was contacted 

by our team—90 provided the majority of data requested and 55 did not provide a response to our request 

after multiple attempts. Of the 90 practices that provided data, 74 were determined ineligible based on the 

inclusion criteria described in Section 2. Of the 16 eligible practices, three were associated with a single 

facility in one state and grouped together as a single practice for the WFA. Another three were associated 

with a single local health department, with two of these practices available for a visit. 

Our team successfully scheduled and performed WFA visits at 11 out of 13 eligible practices. One of the 

eligible practices was experiencing computer issues that were unlikely to be resolved as of the scheduled 

time for the WFA; so we excluded that practice. We were unable to accommodate the schedule for another 

practice—this practice had one individual user of the 2D barcode scanner and this person was unavailable at 

the time open on the WFA visit schedule. One of the 11 practices the WFA consultant visited served as a 

pilot for the procedures outlined in the protocol. This resulted in 10 practices where we conducted WFA 

visits. 

15 
 

Location of practice* 

Michigan 3 

Washington 3 

New York 2 

Florida 1 

New Jersey 1 

Practice specialty 

Pediatrics 6 

Public health department 3 

Family practice 1 

Number of physicians 

5 or fewer 6 

6 to 15 3 

16 or more 1 

MEMR 

EPIC 3
 

Mitchell and McCormick† 2
 

Aprima 1 

E-MDs 1 

GE Centricity 1 

Insight Netsmart 1 

Sage Intergy 1 

Estimated 2D barcode volume administered‡ 

High (Range: 2,157 to 9,515 vaccines administered) 6 

Medium (Range: 672 to 1,779 vaccines 

administered) 

4 

Process for which 2D scanning is used 

Vaccine Administrations 7 

Vaccine Inventory 0 

Both Processes 3 

Percentage of public vaccines (approximate) 

0%-25% 1 

26%-50% 6 

51%-75% 1 

76%-100% 2 

 

 

An overview of the demographics for the 10 participating practices in the WFA is provided in Table 6. The 

majority of WFA practices specialized in pediatrics, were small in size (five or fewer physicians), used 2D 

barcode scanners only to record vaccine administrations, and had an EMR that is not 2D barcode capable13. 

In over half of the WFA practices, approximately 26% to 50% of their vaccine supply is public purchase. 

Table 6. Characteristics of participating WFA practices 

Characteristic Frequency 

13 2D capable means an EHR can process the scan from a 2D barcode scanner (not configured using our configuration utility) to 

deposit the data into their system. 

16 
 

 

 

 

 

 

 

 

 

 

* Location of practice relates to the Immunization Awardee jurisdiction within which 

the practice resides. †2D barcode capable. ‡This represents the number of 2D 

barcoded vaccines (non-influenza) that would have been administered by the 

practice during the original pilot project time period (Nov 2011-March/April 2013) 

given which vaccines were 2D barcoded as of May 7, 2014). 

3.2. Vaccine-specific administration and inventory workflows 

The WFA consultant observed over one hundred vaccine-specific administration workflows from start to 

finish across the 10 participating practices. The smallest number of observations was six, the largest number 

was 20. Observations of the inventory process were only possible in two of the 10 participating practices, 

with six observations total (Table 7). 

Table 7. Number of observations by process 

Practice Administration Inventory 

1187 15 5 

2413 17 -­

2133 8 -­

2508 6 -­

2091 8 1 

2452 7 -­

2299 11 -­

2429 14 -­

2168 20 -­

2510 6 -­

Total 112 6 

3.2.1. Vaccine administration 

The workflow processes associated with patient visits where one or more vaccines were administered were 

more likely than the processes associated with vaccine inventory to differ with the introduction of 2D 

barcode scanning. However, minimal changes occurred in this procedure for the 10 practices observed. All 

practices, with the exception of the pilot practice and Practice 2168, had 2D barcode scanners located at 

every data entry site in the practice. As a result, no changes beyond those associated with physically 

scanning the barcode occurred in the overall workflow procedures when a 2D barcode vaccine was 

recorded. In one location where 2D barcode scanners were not readily accessible at each recording station, 

staff reported in the interviews that they purposefully changed their regular workflow to accommodate the 

2D barcode scanning observations for the WFA. 

Only one of the observed practices incorporated the use of a portable tablet device to record data while in 

the patient room. The staff informed the WFA consultant that the tablet was unable to use the 2D barcode 

scanner because it was not configured to perform scanning and the only available 2D barcode scanners at 

this specific practice were located at the stationary nursing stations. 

17 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

3.2.2. Vaccine inventory 

The WFA consultant observed two practices performing inventory data entry because only those two 

incorporated the use of 2D barcode scanners into their normal inventory recording process. The practices 

used these inventory processes to keep an accurate count of vaccines in stock within the facility. 

Additionally, the data entered at inventory populates drop down menus in practices that have EMRs that 

make use of drop down menus to enter data (such as lot numbers, vaccine names, and vaccine 

manufacturers) when recording data about vaccines administered. The two practices where the inventory 

process was observed (Practice 2091 and Practice 2452) were affiliated with the same local health 

department where inventory was a centralized function. 

In general, the procedures associated with recording vaccine inventory changed minimally in these three 

practices between the activities performed for recording data from linear barcoded vaccines and 2D 

barcoded vaccines—with the major difference being the use of a 2D barcode scanner. In the observed 

practices, the manual entry of lot number and expiration date was no longer necessary when a practitioner 

scanned a 2D barcode. Practitioners in practices 2091 and 2452 used the Vendor B EMR which required 

them to press a “scan” button prior to scanning the 2D barcode on each vaccine to activate the scanning 

capability in the EMR. This was recognized by nurses as a challenge when entering data for multiple 

vaccines administered to a single patient since it adds an extra step to the process for each vaccine entered. 

It did not, however, seem to present similar difficulties in recording vaccine inventory. 

One of the practices observed (practice 1187), relocated the inventory process to a different location since 

the start of the pilot study as part of an internal initiative to improve efficiency; the addition of scanners was 

a step included in the update of this inventory process. The previous inventory recording process was done 

at the front desk where patients checked in. The same staff member still performs the inventory input; 

however, there is now a computer station with a scanner in the inventory storage room, and the vaccines 

are now either scanned or input traditionally in the storage room whenever a new shipment arrives. The 

staff member noted that this cuts down on the number of interruptions and therefore likely cuts down on 

the number of data errors made as a result of having to stop and return to the process several times. 

We should also note that individuals performing data entry at inventory noted that they preferred to scan 

the 2D barcode located on the box over having to open the box and scan the individual vials. One practice 

noted that these lot numbers were initially different, but it was decided as an office policy to always use the 

box barcode instead. 

3.3. Changes in amount of time to record vaccine data 

One of the purposes of the WFA is to evaluate whether a statistically significant difference exists in the time 

it takes to record vaccine data from a 2D barcode compared to a linear barcode. As described in the 

methodology section, the WFA consultant obtained measurements of the time it took practitioners to 

record data for vaccines administered and vaccine inventory. Results for each of these processes are 

provided below. 

18 
 

 

 

 

 

 

 

 

 

 

 

 

3.3.1. Vaccine administration 

The WFA consultant captured time measurements for 129 instances of recording data using 2D barcode 

scanning and 186 instances of recording data from linear barcoded vaccines using the practice’s traditional 

method of data entry. As seen in Table 8, few observations were of mock procedures. 

Table 8. Number of time measurements for recording vaccine administration data* 

Attribute 2D Linear Total 

Real Scan 93 158 251
 

Mock Scan 36 28 64
 

Total Observations 129 186 315
 * Observations performed in N=10 practices 

The average time to record vaccine administration data using 2D barcode scanning was 26.75 seconds. The 

average time to record vaccine data from a linear barcode using traditional methods was 28.47 seconds. 

Thus, when data across all 10 practices were aggregated, we saw an improvement of 1.72 seconds when 

recording data with a 2D barcode scanner versus recording data from a linear barcoded vaccine using 

traditional methods. 

We conducted a paired-sample t-test to examine whether the difference in the time it takes to record 

vaccine data by scanning a 2D barcode is a statistically significant improvement over the time it takes to 

record the same information using traditional data entry procedures with a linear barcoded vaccine. Our 

analysis revealed that the assumption of normal distribution was not met within the time to record vaccine 

data; therefore we selected the nonparametric paired sample Wilcoxon Signed-Rank Test which does not 

assume an underlying distribution of the paired differences. The results from this test indicate, that the 

average time it took to record vaccine data using 2D barcode scanning is not statistically significantly 

different than the time it takes to record vaccine data using traditional data entry procedures with a linear 

barcoded vaccine (Table 9). 

Table 9. Results of Wilcoxon Signed-Rank Test 

Barcode 

Type 

Average Time 

(all data) Difference V * P-Value 

Linear 28.47 

2D 26.75 

1.72 344 0.437 

*V is the test statistic output for the Wilcoxon Signed-Rank Test. It is equal to the absolute 

value of the sum of signed ranks. This test was conducted using the R function Wilcox.test 

Although there was not a statistically significant difference between the time to record vaccine data using a 

2D barcode scanner relative to entering data using traditional methods when considering all of the practices 

together, there were several additional practice-specific factors that could affect recording time. These 

factors included EMR type, vaccine type, and idiosyncratic differences between nurses and differences 

between the practices. For instance, Figure 1 and Table 10 show there were substantial differences 

between practices in time to record vaccine data. The shortest average time to record vaccine data was 11 

19 
 

 

 

 

 

 

 

seconds. The longest average time to record vaccine data was 36 seconds. That is a 25-second difference in 

average time to record vaccine data regardless of the type of barcode on the vaccine. The shortest average 

time to record data from linear barcoded vaccines was 10 seconds with the longest being 44 seconds. The 

shortest average time to record data from 2D barcoded vaccines was 14 seconds with the longest being 39 

seconds. Differences in average recording times for linear barcoded vaccines indicate there were unique 

differences between practices in scanning that should be considered when measuring changes in average 

time to record data from 2D barcoded vaccines. Variability in recording time was also present within 

practices—some practices had substantially more variation in recording times than others. 

Figure 1. Time to record vaccine administration data by practice 

0 

10 

20 

30 

40 

50 

1187 2091 2133 2168 2299 2413 2429 2452 2508 2510 

Seconds 

Average Time to Record Vaccine Data 

2D versus Traditional 

2D Barcode Linear Barcode 

20 
Linear Barcoded 

Mean (SD) 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Table 10. Descriptive statistics of time to record vaccine data by practice 

Mean 

Difference 

(2D-Linear) 

Overall 

Mean (SD) 

2D Barcoded 

Mean (SD) Practice ID Observations 

1187 40 26.45 (9.84) 22.63 (9.15) 28.29 (9.78) -5.66 

2091 19 28.77 (18.27) 39.31 (22.87) 22.63 (12.22) 16.68 

2133 21 26.00 (12.73) 28.16 (17.33) 24.66 (9.47) 3.5 

2168 43 16.41 (6.97) 18.83 (7.11) 14.67 (6.46) 4.16 

2299 25 35.45 (24.66) 29.15 (19.36) 43.46 (29.08) -14.31 

2413 40 31.52 (9.90) 29.58 (10.21) 32.46 (9.8) -2.88 

2429 31 27.05 (20.61) 32.08 (28.08) 22.34 (8.04) 9.74 

2452 17 11.82 (4.01) 14.90 (4.26) 10.53 (3.26) 4.37 

2508 63 36.02 (21.32) 27.76 (16.39) 44.55 (22.66) -16.79* 

2510 16 27.11 (9.33) 20.03 (3.22) 29.47 (9.57) -9.44 

*Difference between the mean time to record data from 2D barcoded vaccines and linear barcoded vaccines is statistically significant 

at a .05 level after Bonferroni adjustment (t=-3.36) 

For five practices, the average time to record vaccine data was lower when using 2D barcode scanners than 

recording data from linear barcoded vaccines using traditional methods. For five practices, the average time 

to record vaccine data increased. As noted in the above table, these differences were not significant, with 

one exception. Those practices that saw improvement in time to record vaccine data with 2D barcode 

scanners versus traditional methods had higher than average time to record linear vaccines. One practice 

experienced a decrease of 14.31 seconds between traditional methods and 2D barcode scanning but they 

had an average time for traditional methods 15 seconds higher than the overall average of 28 seconds. 

Similarly, a practice that experienced a 16 second improvement from traditional methods to 2D barcode 

scanning had an average scanning time, however, the traditional method was 16 seconds higher than the 

overall average recording time using the traditional method. Practices that saw time to record vaccine data 

increase with 2D barcode scanning versus traditional methods from linear barcoded vaccines had lower than 

average time to record data from vaccines with linear barcodes. For instance, one practice that experienced 

a large increase in time to record data from 2D barcodes versus the traditional method had a traditional 

average time 6 seconds under the average recording time overall. 

Although the EMR and general data entry procedures were similar between the two practices under the 

same local health department (practices 2091 and 2452), these practices had large differences in the 

average time to record vaccine data using 2D barcode scanning versus traditional methods. In both 

practices, it took practitioners longer to record vaccine data using 2D barcode scanners; however, this 

difference was appreciably larger for Practice 2091 (16.68 seconds compared to 4.37 seconds in Practice 

2452). This difference was the result of two outlier observations where the Vendor B EMR system froze at 

21 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Practice 2091 when using a 2D barcode scanner14. However, when the Vendor B system15 did not freeze 

(Practice 2452), the average time to record a 2D barcoded vaccine was the smallest of any practice observed 

(14.90 seconds). 

Practices where the largest decrease in time occurred when using 2D barcode scanners (Practice 2299 and 

Practice 2508) both manually entered lot number and expiration date (as opposed to using a drop down 

menu) in the absence of a 2D barcoded vaccine. In Practice 2299, the lot number which had to be manually 

typed into the system was case sensitive meaning that upper and lower case letters must match exactly to 

the lot number on the vaccine. This case sensitivity resulted in a more challenging data entry procedure. 

One practice (Practice 2429) had a relatively large standard deviation for recording vaccine data with 2D 

barcode scanners. All of the individuals observed at this practice were hired after the pilot observation 

period; variation in their exposure to scanning technologies prior to joining this practice may be one reason 

for the large variation in time to record vaccine data. Additionally, observations suggested that the 

practitioners at this location were not in the habit of using the 2D barcode scanners as part of their normal 

routine. 

The largest predictor of differences in time to record vaccine data in the random intercept linear mixed 

effects analysis was the nurse administering the vaccines. Observations indicated that data entry times 

differed as a result of how practitioners entered data into the EMR—for example, whether they elected to 

use a mouse to move between data entry cells or tab between cells and whether they focused solely on 

entering data or engaged in other tasks while performing data entry. These differences explain 17% of the 

variation in time. Practice and vaccine type both individually explain 9% of the variability in time to record 

vaccine data. We anticipated that practice differences could arise due to a difference in recording 

processes, expectations, and staff in each practice. 

We expected that vaccine name could affect time to record data due to differences in vaccine manufacturer 

and packaging. Vaccines that take substantially longer than average to record may indicate a packaging 

issue such as the barcode being placed in a hard-to-record location. The linear vaccine that took the longest 

to record was IPV with an average recording time of 57 seconds or two times the average recording time for 

linear vaccines. The 2D barcoded vaccine that took the longest to record was DTaP at an average of 33 

seconds, compared to an overall 2D barcoded vaccine average of 26 seconds. Despite having the highest 

average, 33 seconds is a seven second improvement over the average time to record data for the linear 

barcoded DTaP vaccine. Among all the vaccines we observed that had both 2D barcode and linear barcode 

presentations, it took less time to record data using a 2D barcode scan than traditional procedures for data 

entry with linear barcoded vaccines. 

14 Removal of the two outliers decreased the amount of time it took practitioners, on average, to record data using a 2D barcode 

scanner. The average 2D barcode scanning time dropped to 30.43 seconds (SD=20.89), resulting in an average difference of 7.81 

seconds in recording time between 2D barcode scanning and entering data from a linear barcoded vaccine. This difference 

remained statistically insignificant (T=0.78, p=0.47). 

15 It should be noted that although both practices used the Vendor B EMR they reside under the same health department system 

and therefore likely have in common other software and hardware. The EMR in use is only one of several factors that could relate to 

the freezing issues experienced within these practices. 

22 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Conducting the mixed model analysis allows for the examination of the net effects other variables (EMR, 

nurses, vaccine name, and practice) may have on the time it takes to record vaccine data when scanning a 

2D barcode relative to entering data from a linear barcoded vaccine using traditional methods. The final 

analysis revealed that the use of a 2D barcode scanner is associated with a decrease of .89 seconds in time 

to scan vaccine data when the net effect of these other factors is included. Visual inspection of residual 

plots did not reveal any obvious deviations from homoscedasticity or normality indicating that the 

underlying assumptions of this statistical approach were met. 

3.3.2. Vaccine inventory 

There were only two practices where inventory processes were observed. Between these two practices the 

WFA consultant took 33 time measurements– nine at the first practice and 24 at the other practice (Table 

11). The small number of observations did not afford us the opportunity to conduct inferential statistics for 

the inventory process, thus we present descriptive statistics only in this section. 

Table 11. Number of time measurements for recording vaccine inventory data* 

Attribute 2D Linear Total 

Real Scan 11 13 24 

Mock Scan 4 5 9 

Total Observations 15 18 33 

* Observations performed in N=2 practices. All mock scans occurred in one practice. 

The overall average recording time for the nine observations in the first practice was 21.6 seconds. This 

practice did experience a drop in the average time to record vaccine inventory data from 31.8 seconds using 

their traditional method of data entry to 13.4 seconds using 2D barcode scanners. This is an 18 second 

improvement in average time to record vaccine inventory data16. The second practice had an overall 

average vaccine inventory recording time of 26.3 seconds. Unlike the first practice, this practice 

experienced an increase in average time to record vaccine data using 2D barcode scanners compared to 

their traditional method of data entry. The average time practitioners took in this practice to record 

inventory data using their traditional method of data entry was 19.4 seconds, while the average time they 

took to record vaccine inventory data using 2D barcode scanning was 40.1 seconds. This is an increase of 

20.7 seconds using 2D barcode scanners versus their traditional data entry method (Figure 2). 

16 Due to the low sample size there is no statistically sound method to determine if this is a statistically significant change. 

23 
 

Average Time to Enter Inventory Data*
 

Traditional versus 2D Barcodes 

45.0 

40.0 

35.0 

30.0 

25.0 

20.0 

15.0 

10.0 

5.0 

0.0 

2091 

Seconds 

31.8 

19.4 

13.4 

40.1 

1187 

Traditional 2D 

 

 

 

Figure 2. Time to record vaccine inventory data by practice 

*

All observations of inventory data entry at practice 1187 were mock scans. No mock scans were observed at practice 2091. 

3.4. Staff opinions about 2D barcode scanning 

The WFA consultant obtained staff opinions about using 2D barcode scanning to record vaccine data 

through in-person semi-structured interviews while visiting each practice (See Appendix B). A total of 27 

interviews were performed with 32 interviewees. The largest number of interviews the WFA consultant 

performed in any single practice was five (with six individuals). In two practices the WFA consultant 

interviewed one individual. Table 12 summarizes the number of interviewees and interviews performed by 

practice. 

Table 12. Number of interviewees and interviews by practice 

Practice Number of 

interviewees* 

Number of 

interviews 

2091 3 3 

2452 3 2 

1464† 3 3 

2413 3 3 

1187 6 5 

2133 2 2 

2510 2 2 

2508 6 3 

2299 1 1 

2429 1 1 

2168 2 2 

TOTAL 32 27 

24 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

*

Some interviewees participated in the same interview. 

†

Pilot practice
 

for current 2D WFA.
 

Factors facilitating continued use of 2D 

Interviewees who provided a specific answer about why their practice continued to scan 2D barcodes to 

record vaccine data most frequently noted a perceived increase, or potential for improved, accuracy of 

records (n=11 interviews17). Interviewees, although less frequently, also mentioned a perceived time 

savings from using the 2D barcode scanner to enter data (n= 3). There was also a general sense among 

many who were interviewed that just being a part of the pilot helped to stimulate continued use (n=6)—for 

example, some of these interviewees noted that they were accustomed to using 2D barcode scanners since 

they did so during the pilot project observation period. Since they already had the equipment and didn’t 

experience any serious issues using it, they simply continued on the same course. During one interview, it 

became apparent that a group of nurses at one practice were unaware that the pilot project observation 

period had come to a close. 

In many interviews, the interviewee mentioned that they did not know why their practice continued to use 

2D barcode scanning or that they were hazarding a guess about why this process was still in place (n=13 of 

26 interviews where question was specifically addressed18). What is clear is that continued or sustained use 

of 2D barcode scanning in these practices was rarely, if ever, an intentional or planned act. None of the 

interviewees noted that there was a specific policy or procedure in place requiring staff to scan 2D barcoded 

vaccines. In one practice there was a sense from an interviewee that although there was not a formal policy 

in place that 2D barcode scanning was strongly recommended – “They don’t have an option not to use it, so 

if it is there I want them using it”. However, this statement was tempered by two other interviewees who 

indicated that the practice was recommended or that they did it because of participation in the pilot. For 

other practices, there was the general sense from interviewees that scanning 2D barcodes was simply a 

continued practice and that individual staff were allowed to scan or not based upon their own preference. 

Improvements given increase in 2D barcoded vaccines 

During 21 of the 27 interviews (78%), interviewees conveyed that staff had noticed an increase in the 

number of vaccines with 2D barcodes since the pilot end date. In some of these instances, although the 

respondent was asked about 2D barcode volume since the end date, it was unclear what their referent 

period of time was. Some interviewees appear to have referred to the time since the pilot began rather 

than came to a close. Nevertheless, there was wide recognition that a greater number of 2D barcoded 

17 All n’s listed for interview data in Section 3.4 represent the frequency of interviews in which a given theme was 

mentioned. 

18 Interviews performed were semi-structured in nature. This format allowed the interviewer to adapt the interview as 

needed based upon the context. Therefore, in some instances all questions on the interview guide were not asked in 

every interview.” 

25 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

vaccines were in the supply chain than in the past. Although we know that 52 presentations of 22 types of 

vaccines were available as of August 4, 2014, the interviews helped to confirm that 2D barcoded vaccines 

are making it through the supply chain and into some practices. 

During two interviews respondents clearly used the March/April 2013 end date of the pilot project as their 

referent and noted that more 2D barcoded vaccines had not come out since this time, but did mention an 

increase during the course of the pilot project. Those responding (n=4) that they did not know if there was 

an increase in availability of the 2D barcodes in the supply chain had either stopped using the barcode 

scanners, did not administer vaccines very frequently, were new to the office, or were not responsible for 

administering vaccines (e.g., information technologist). 

During the original 2D pilot project a large percentage of respondents (n= 161, 77%) to the second User 

Experience Survey indicated the fact that “only a small percentage of vaccines have 2D barcodes” presented 

a challenge to integrating 2D barcodes into their practice’s process for recording information for vaccine 

administration or inventory. Since additional vaccines are now in the supply chain with 2D barcodes, we 

inquired during the course of interviews as to whether (and how) this increase in the percentage of 2D 

barcoded vaccines in the practice affected their ability to use 2D barcode scanning to record vaccine data. 

Responses to this interview question were mixed. Interviewees noting that there was an effect of the 

increase in 2D barcoded vaccines (n=9) often provided fairly general feedback about the process being 

easier or quicker. Other interviewees indicated that since there were more 2D barcoded vaccines, they 

became more efficient at scanning and were better able to integrate it into their routine. One interviewee 

commenting on her routine also called to attention the desire to have more 2D barcoded vaccines because 

now there is an extra step in the process—having to look to check if the vaccine is 2D barcoded or not. She 

pointed out that the more 2D barcodes there are, the more it became a part of her routine. Another 

interviewee noted that the increased flow of 2D barcoded vaccines resulted in her looking for the 2D 

barcode more frequently than she did in the past—increasing her behavior to see if a barcode was available. 

Another interviewee noted that there was a negative effect on her process—that more 2D barcoded 

vaccines slowed her down because the scanner did not read the 2D barcodes consistently; when she had to 

scan a few times per barcode it disrupted her normal process. 

Multiple interviewees, however, specifically remarked that the increase in 2D barcoded vaccines did not 

affect their ability to use scanning to record vaccine data (n=7). Based upon our estimates of the current 2D 

barcode vaccine administration volume within the participating WFA practices—meaningful differences did 

not appear between practices where interviewees indicated that this increased volume has no effect and 

those where the interviewees reported the increased volume having a positive effect on their ability to use 

scanning to record vaccine data. 

Reported benefits of 2D barcode scanning 

Interviewees most frequently reported that the benefit of using 2D barcode scanning is improved accuracy 

of vaccine data (n=18). Many of the interviewees who noted this advantage specifically mentioned this 

benefit relative to accurate lot numbers. Some individuals specifically commented on the difficulties they 

26 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

experienced when recording lot numbers manually. Specific examples of challenges recording lot numbers 

included having difficulties reading the small numbers associated with the lot and difficulties with 

distinguishing between characters that look very similar. 

“...honestly, sometimes like when you get a row of 5 in a thing you actually have to count 

how many 5’s are there—it isn’t easy to pick it up because they are so small.” 

“I do find it easier when it works, I like not having to look at the tiny little numbers and 

guess if it is a six or an eight but they are so small that I can’t tell which one is which. So 

I like that part of it.” 

Interviewees also frequently mentioned (n=7) that scanning 2D barcodes was quicker than their traditional 

method of data entry or felt that not having to enter as much vaccine data by hand or drop down was a 

benefit. 

Some interviewees in practices where scanning was used to record vaccine data at inventory also mentioned 

the benefit of having accurate lot numbers for the purpose of reconciliation. Several also noted that they 

enjoyed the addition of 2D barcodes to vaccine boxes, as it helped to make the scanning process easier at 

inventory. 

“I think a week ago we typed in the wrong lot number for a vaccine we received in 

because it didn’t have the 2D barcode and once those are administered to our patients 

then when we find out after so many have been administered, someone notices ‘oh the 

wrong lot number was put in’ we have to take all of those immunizations out of the 

client’s records, fix the mistake we made, re-enter it into [the EMR] and then re-enter it 

into all of the client records. And it’s okay if it is like 5 but we have done 75 before and 

I’ve put the lot number and someone even double checked me. So scanning is good 

because it is accurate.” 

“And they also put it on the outside of boxes. Glaxo does a lot with it on the outside 

which is nice, because you don’t have to open up the box. I still open up the box of the 

ones that don’t show, just to make sure because some of them still have the 2D on it 

without it being on the outside of the box. But that is the only thing that I would rather 

not do is open the box.” 

Reported challenges and drawbacks of 2D barcode scanning 

By far, the most frequent comment regarding the challenges of using 2D barcode scanning to record vaccine 

data related to difficulties scanning barcodes and the level of inconsistency with which the scanning process 

actually works. Many interviewees remarked that some vaccine types scanned easier than others. The 

influenza vaccine was mentioned by several interviewees working in different practices as difficult to scan. 

We are not aware, however, of which specific influenza vaccines these practices use. Some interviewees did 

not offer information about specific vaccines rather they described this challenge in general terms, noting 

that it sometimes took a long time to scan or after multiple attempts failed to scan. 

27 
 

 

 

 

 

 

 

 

 

 

 

 

Interviewee: “I think it is very easy, the only issue is some of the vaccines don’t scan as 

well as the others”. Interviewer: “Do you know which ones in particular?” Interviewee: 

“Hep A I think the barcode is too light and sometimes the influenza—influenza does not 

scan, hardly ever”. 

“...the only challenging I would say is some barcodes are not as easy to get to scan. I 

found that with the flu vaccine, they were very difficult to get. I would think that was the 

hardest. But for example, DTaP and the Menactra are very easy barcodes to use and 

Hepatitis A is a very easy barcode to use.” 

“...sometimes you are able to scan and sometimes the scanner doesn’t work and by the 

time you get it to scan you could have entered the numbers in, you know, in less time.” 

A few individuals offered other challenges including having to remember to look to see if a vaccine is 2D 

barcoded, not having a mechanism for scanning multi-component vaccines, and still having to open some 

boxes to record vaccine inventory. Some individuals also noted a desire to have more information populate 

into their EMR about the vaccine when scanning a 2D barcode (such as vaccine name, manufacturer, NDC, 

as opposed to only populating lot number and expiration date). Analyses of user experience data collected 

during the pilot project observation period also highlighted this as a challenge, however, such functionality is 

a function of EMR capabilities. EMRs that are 2D capable are able to receive and process the data elements 

from scanning a 2D barcoded vaccine (i.e., lot number, expiration date, and product ID) to identify and input 

additional variables for the vaccine. 

There were two challenges that arose in a subset of practices visited for the WFA. The first concerned the 

wrong lot number entering into an EMR when a 2D barcoded vaccine was scanned. This happened for two 

practices where the lot number is selected from a drop down list. It appears that lot numbers are selected 

in these EMR systems via an auto-populate feature which may inadvertently select the incorrect lot number 

based upon how the EMR selects a match. For example, if the configuration is set up to look for a match 

with the first three digits of a lot number and there are two lot numbers in the EMR system with the same 

first three digits, the wrong lot number could be selected by the EMR. 

The diversity of EMR solutions observed in the pilot made implementation of a programmatic interface 

between the 2D barcode scanner and each EMR infeasible within the pilot timeline. Therefore, to enable 

scanners to work with EMRs for the pilot we used a configuration utility. The configuration utility enables 

the 2D barcode scanner to read the data into the correct fields in the EMR but does not modify the 

programmatic behavior of the EMR. If an EMR selects the lot number based on the first few lot characters 

keyed in and there are multiple lots in the system whose lot numbers begin with the same characters, 

scanning a 2D barcode results in the first match being selected. One interviewee noted that it was 

important to point out that as a result, 2D barcode scanning was not always a “fail-safe” method for 

entering the correct lot number. 

A second issue noted by interviewees, and witnessed during the course of observations, was the freezing of 

the EMR system used in the practices located under the same local health department (Practice 2091 and 

28 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Practice 2452). Interviewees reported that the system would intermittently crash when using the 2D 

barcode scanner. While in Practice 2091 the WFA consultant observed this occurring in real time. During 

one 2D barcode scan, the influenza vaccine did not register a scan within 30 seconds. The EMR system 

subsequently froze and the nurse had to exit out of the EMR and start data entry over. The nurses at this 

practice noted that the system freezing is inconsistent—sometimes they can use the scanner for extensive 

periods without the system freezing. The system froze twice during the observations, both times when 

influenza vaccine was being administered. 

4. Strengths and Limitations 

There are several strengths and limitations associated with the WFA we present in this report. One of the 

strengths relates to the consistency and quality of data collection activities. Throughout the course of the 

entire WFA project, we incorporated multiple checkpoints to test and examine the quality of our data 

collection procedures. Prior to data collection, the WFA team’s project manager, evaluator, and WFA 

consultant met in person to review the data collection procedures and discuss any initial questions. 

Additionally, we used one practice visit as a pilot test to examine the proposed data collection activities in 

action to fully assess their feasibility. Following the pilot test, the WFA team regularly checked in via email 

and phone while the WFA consultant was in the field to identify any emergent issues in data collection and 

actively discuss next steps. 

A related strength of the WFA concerns the type of data gathered and the specific procedures used. For 

example, the WFA consultant performed interviews with practice staff while on-site. During these 

interviews, individuals were asked to step away from their regular work to reflect specifically on the 

questions posed absent distractions. The WFA consultant digitally recorded these interviews to ensure that 

we fully captured the content and tone of interviews. Additionally, in the current WFA we opted to gather 

time measurement data for the main steps in the workflows performed in the participating practices. Such 

data helps to provide information about the context within which data entry about vaccines occurs, and can 

suggest the proportion of time out of the whole workflow process that is affected by procedures such as 2D 

barcode scanning. 

The main limitation in this WFA concerns the selection of practices. As described in the methodology 

section, the practices participating in this WFA were not randomly selected from all practices that 

participated in the pilot project. As such, the practices that elected to participate may be more likely to 

have successfully integrated 2D barcoding into their workflows and have more positive opinions about their 

use than other practices from the pilot project. Additionally, we were unable to visit every practice that met 

the inclusion criteria because of the short time period available to perform data collection activities. Two 

practices experienced scheduling conflicts that our team could have accommodated with a longer data 

collection window. 

Other limitations associated with the WFA include the number of data entry observations the WFA 

consultant was able to observe and the limited number of practices visited. Several practices visited do not 

specifically focus on administering vaccines, as such only a subset of patient visits relate to vaccinations. 

29 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Since the WFA visits did not occur during the “busy season” for vaccination such as the school season or 

influenza season, we were not able to observe as many data entry efforts as we originally targeted. When 

possible, the WFA consultant did work to obtain mock scans, however three practices did not participate in 

mock scanning due to staff concerns that the falsely entered data would result in issues around 

reconciliation of inventory. 

Finally, we were unable to explore the specific differences in data entry among staff with respect to the data 

entry process. These differences made up much of the variability in the time to record vaccine data at 

administration; however, we did not obtain specific information about factors that may have contributed to 

these differences such as the work history or other demographics of each nurse performing data entry. We 

were, however, able to identify some sources of this variability between nurses as part of the detailed 

observations performed (discussed in Section 5). 

5. Conclusions and Recommendations 

The analyses we performed of time to record vaccine administration data indicated no statistically 

significant difference between the average time it took practitioners to record vaccine data using 2D 

barcode scanning compared to using their traditional method to record data from vaccines with a linear 

barcode (e.g., manual entry or selecting from a drop down menu). This finding is inconsistent with the 

results of the first WFA performed during the pilot observation period. In the initial WFA, we found that it 

took, on average, 3.6 seconds longer to record vaccine data using 2D barcode scanning than entering data 

using traditional procedures. This difference was statistically significant at α=.05. 

There are multiple potential reasons that we found no differences in data entry time between 2D barcode 

scanning and traditional procedures for recording data about vaccines administered. First, we noted a clear 

relationship between the percent of 2D barcoded vaccines and the time to record vaccine data in the initial 

WFA—with practices that had a higher percentage of 2D vaccine administration volume also generally 

having smaller data entry time. This could indicate that more exposure to 2D barcode scanning influences 

overall scanning time. Practices in the current study were exposed to 2D barcode scanning for longer than 

those in the initial WFA, and also experienced a recent increase in the number of 2D barcoded vaccines 

available. While we hesitate to state a direct relationship, it may be the case that as health care 

practitioners increase their level of comfort with 2D barcode scanning, the scan time will increasingly 

approach the manual/drop down entry time. 

The majority of interviewees (78%) did note that they witnessed an increase in the number of vaccines with 

2D barcodes since the pilot project19. However, interviewee perceptions were mixed regarding whether this 

increased volume affected their ability to record vaccine data. Some interviewees felt that this change 

made it easier or quicker to scan, others specifically pointed to an increase in efficiency as they got more 

19 These interviewees were typically able to specifically name which vaccines were 2D barcoded during the pilot project observation 

time period and which ones were now 2D barcoded, indicating that an availability heuristic was unlikely to be occurring. 

30 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

used to the practice and were better able to integrate it into their routine, and others did not feel the 

increase affected their ability to scan. 

Second, practices in the current WFA that saw time to record data about vaccines administered increase 

with 2D barcode scanning versus traditional methods from linear barcoded vaccines had lower than average 

time to record data from vaccines with linear barcodes. For instance, one practice that experienced a large 

increase in time to record data from 2D barcodes versus the traditional method had a traditional average 

time 6 seconds under the average recording time overall for traditional methods. Third, the practices we 

included in the current WFA are different from those we included in the initial WFA. Since the current WFA 

does not include the same practices these findings are not directly comparable to the initial WFA—it is 

possible that underlying difference between the practices in the initial WFA and the practices in the current 

WFA drive the disparate findings. Fourth, different individuals recorded data for the initial WFA than for the 

current WFA. Small differences in recording procedures could also contribute to different findings. 

Finally, there could be a unique uncaptured driving factor for why the findings from the current WFA differ 

from the findings of the initial WFA. Additional factors that may affect this relationship include a health 

practitioner’s tenure, volume of vaccine administration, and workforce culture around 2D barcoded 

vaccines. For example, the mixed model presented in Section 3.3 estimated that “administering nurse” 

explains 17% of the variation in time to record vaccine data. Although we did not capture potential factors 

for these idiosyncratic differences in recording times using quantitative variables, the WFA consultant’s 

qualitative observations suggested that the manner in which health care practitioners recorded vaccine data 

varied widely. Observations indicated that data entry times differed as a result of how practitioners entered 

data into the EMR—for example, whether they elected to use a mouse to move between data entry cells or 

tab between cells and whether they focused solely on entering data or engage in other tasks while 

performing data entry. 

Several other findings from the current WFA warrant mention. First, for most practices, there were minimal 

changes to the workflows associated with entering data between linear and 2D barcoded vaccines. 

Information from the first workflow analysis and User Experience surveys conducted during the observation 

period of the pilot project indicated major disruptions in some practice’s workflows with the introduction of 

2D barcode scanning—with some practices having to physically retrieve a 2D barcode scanner from another 

room, shelf, or drawer to scan. In the current WFA, all practices (with the exception of one clinic and the 

pilot practice) had a 2D barcode scanner located at every data entry point in their practices. We 

hypothesize that the number and placement of scanners within these practices contributed, at least in a 

small way, to more frequent and continued use of 2D barcode scanning within the participating practices. 

Additionally, findings from the interviews suggest that challenges remain for full integration of 2D barcode 

scanning into practice workflows. The primary challenge concerns the consistency with which 2D barcode 

scanning results in successfully registering data into the practice’s EMR. Interviewees perceived differences 

in the ease with which scans registered from different 2D barcoded vaccines. They also noted general 

challenges with getting scans to register. In some cases, the interviewees noted that they would attempt to 

31 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

scan a vaccine one or two times and then revert to their traditional method of data entry if the scan was 

unsuccessful. For some, however, this inconsistency resulted in no longer using the 2D barcode scanner. 

In some practices (two observed by the WFA consultant), the EMR system allows a user to select a lot 

number from a drop-down menu and automatically advances the user to a lot number based on characters 

keyed by a user. Some interviewees who interfaced with such EMRs noted that 2D barcode scanning 

sometimes selected the incorrect lot number. This issue was a product of the way 2D barcode scanners 

interfaced with EMRs for the pilot project. The diversity of EMR solutions observed in the pilot made 

implementation of a programmatic interface between the 2D barcode scanner and each EMR infeasible 

within the pilot timeline. Therefore, to enable scanners to work with EMRs for the pilot we used a 

configuration utility. The configuration utility enables the 2D barcode scanner to read the data into the 

correct fields in the EMR but does not modify the programmatic behavior of the EMR. If an EMR selects the 

lot number based on the first few lot characters keyed in and there are multiple lots in the system whose lot 

numbers begin with the same characters, scanning a 2D barcode results in the first match being selected. 

One interviewee noted that it was important to point out that as a result, 2D barcode scanning was not 

always a “fail-safe” method for entering the correct lot number. 

The majority of individuals interviewed continued to see the value in using 2D barcode scanners to record 

vaccine data. Several specifically pointed out the difficulties they have with correctly deciphering and handentering the lot number into their EMR and noted that 2D barcode scanning does improve accuracy by 

populating the correct lot number into the system. Additionally, one might infer that 2D barcode scanning 

can be readily adopted into the regular workflows of practices at least partially due to its acceptance among 

end users. None of the practices participating in this WFA had a policy in place requiring their staff to use 

the scanners, however, in several instances practitioners still elected to use 2D barcode scanning to record 

vaccine data. 

Based upon the findings from this WFA we recommend that: 

1.	 Practices considering 2D scanning adoption should invest time up front to plan how 2D barcode 

scanning will work best with their operations to help determine the appropriate number and 

placement of scanners so as not to introduce operational impediments. 

2.	 The CDC and FDA should collaborate with the manufactures to investigate the quality and 

consistency of the 2D barcode labeling. Consistent reports from interviewees that there is 

variability in the ease of scanning 2D barcodes by vaccine type warrants further investigations about 

the quality of 2D barcode labeling and the ability to consistently produce batches with labels that 

practitioners can easily scan. 

3.	 The EMR software should be configured or set up to read the entire lot number which will prevent 

default selection by the system of lots with similar first characters. Solution vendors incorporating 

2D barcode scanning of vaccines into their applications should read the entire lot number from the 

encoded data string. 

32 
 

 

 

 

 

 

 

4.	 Industry should further investigate inconsistencies of 2D barcode scanning. Non-specific, but 

frequent, concerns from interviewees about the inconsistency of 2D barcode scanning suggests that 

follow-up product evaluations that examine the possible root causes of these reported 

inconsistencies would be helpful. Such investigations might examine the comparative effectiveness 

of 2D barcode scanning using a variety of conditions, such as: (a) placement of the barcode in 

different locations on the label, (b) other quality features of barcodes (e.g., different background 

colors, ink used for 2D barcode), or (c) peel-off labels versus labels on vials or syringes. 

This WFA provides some limited indications that 2D barcode scanning can save time when entering data 

about vaccines administered. We hypothesize that the time saving differences may become more 

widespread when the practice of 2D barcode scanning to record vaccine data is common in health care 

practices. Future research is needed once all vaccines are 2D barcoded. 

33 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

___ ___ 

___ 

___ 

___ 

___ 

___ _________________)
 

 __ ___

___ ____ 

Appendix A: Email Disseminated to Candidate Practices 

From: support [mailto:support@2dvaccineadoptionpilot.com] 

Sent: Date 

To: Practice Point of Contact 

Subject: 2D Vaccine Barcode Pilot - Work Flow Analysis 

Dear {Point of Contact}-

Thank you for your participation in CDC’s Implementation Pilot for Two-Dimensional (2D) Vaccine Barcode 

Utilization. Since the pilot, vaccine manufacturers have continued to affix 2D barcodes to their vaccine 

products. Many more vaccines currently have 2D barcodes—approximately 24 vaccines have 2D barcodes 

today as compared to the eight with barcodes during the original pilot. We are fortunate to have an 

opportunity to follow-up with a subset of practices who participated in the pilot project to learn more about 

the workflow associated with scanning 2D barcodes to record vaccine data now that more vaccines have 2D 

barcodes and are asking for your assistance in this effort. 

Based upon the data we gathered during the pilot, we have identified your practice as a potential candidate 

for the upcoming workflow analysis (WFA) project. To determine if you are eligible for this project, we ask 

that you complete the following three questions. We would greatly appreciate answers to these questions 

even if you do not have an interest in participating. Practices not interested in participating will be screened 

out of the project and not contacted by our team. 

1.	 Since the implementation pilot came to a close, has your practice continued 2D barcode scanning? 

Yes No 

If yes to question 1, please answer the following: 

2.	 During which of the following procedures do practice staff use 2D barcode scanners to record 

vaccine information? 

a. Inventory: Yes No 

b. Vaccine administration: Yes No 

3.	 With what frequency do you typically encounter 2D barcoded vaccines?
 

Daily
 

Weekly
 

Monthly
 

Quarterly
 

Other (Please describe: 

4.	 As part of the workflow analysis project, a member of our team will visit practices to observe the 

process of recording vaccine information and conduct brief interviews with up to five staff members 

34 
 

 

 

 

 

___ 

to learn about their experience scanning 2D barcoded vaccines. This visit will occur over the course 

of a regular workday and will be arranged at a time that is most convenient to participating practices 

in June 2014. Given this description, would your practice be interested in participating in the 

proposed WFA? Please note that responding “yes” in no way commits your practice to participation 

at this time. 

___ Yes No 

Thank you very much for your time, if you have indicated an interest in participating in the WFA a member 

of our team from Deloitte will contact you within the next week to inform you of your practice’s eligibility, 

ask some additional questions (if needed), and potentially schedule a WFA visit. 

Thank you for your continued interest in 2D barcoding. 

~ Deloitte Team 

35 
 

 

 

 

 

 

 

o	 

 

 

 

 

 

 

 

Appendix B: Interview Guide 

Introduction: 

Hello [name of interviewee], I am [name of interviewer] from Deloitte Consulting. Thank you for agreeing to 

participate in an interview. The purpose of this interview is to learn about your experience using 2D 

barcode scanning to record vaccine data since the pilot came to a close earlier this year. We will analyze the 

data you provide through this interview alongside interview data we obtain from staff from nine other 

practices that are participating in this project to identify and report out on common experiences. 

The aggregate findings from these interviews will be used by the CDC and others interested in 2D barcoding 

of vaccines to better understand the experience of practitioners employing this technology. Although we 

may use quotes from this interview in our final report on this project, we will not associate your name with 

these quotes. 

I will be taking detailed notes during our interview, however, to make sure I capture important points 

correctly I would like to also digitally record this conversation. Do I have your permission to record this 

interview? [If yes]- Thank you, if at any point in this interview you would like me to stop this recording 

please just let me know and I will do so. 

Section 1: General information 

•	 First, please tell why your practice elected to continue scanning 2D barcodes to record vaccine data. 

•	 Beyond the items you just mentioned, were there other actions taken within or outside of your 

practice to help facilitate the continued use of 2D barcode scanning (e.g., policies requiring staff to 

use this process, recommended procedures by leadership, IIS policies)? 

[If yes]- Please tell me about these actions. 

Section 2: Personal use of 2D barcode scanning 

Now I would like to ask you some questions about your experience using 2D barcode scanning to record 

vaccine data. 

•	 First, please tell me a little about your experience using 2D barcode scanning to record vaccine data. 

To what extent do you find this activity to be easy or challenging? Why? What makes this 

challenging/easy? Please provide me with an example. 

•	 Since the pilot came to a close in March 201420, would you say that you have seen more 2D 

barcoded vaccines? [If yes]- Please tell me more about the types of vaccines you have seen and the 

frequency with which you administer these vaccines. 

•	 To what extent has the increased flow of vaccines affected your ability to use 2D barcode scanning 

to record vaccine administrations/inventory, if any? {probe for specifics and examples} 

20 Should have been March 2013. Frequently this exact language was not used in the course of interviews (due to the semi-structure 

nature) and therefore was unlikely to have affected interviewee responses. 

36 
 

 

 

 

 

 

Section 3: Benefits and drawbacks of using 2D barcode scanning 

•	 What do you perceive to be the benefits of using 2D barcode scanning to record vaccine 

administration/inventory data? {probe for specific examples of these benefits in their practice} 

•	 What do you feel are some of the drawbacks of using 2D barcode scanning to record vaccine 

administration/inventory data that you haven not already mentioned during our conversation? 

{probe for any implications} 

Section 4: Close-out 

•	 Those are all of the questions I have for you. 

•	 Are there any items we have discussed that you would like to elaborate on further? [If yes]- which 

ones? {ask for elaboration if needed} 

•	 What other items do you think would be helpful for us to know about the experience of using 2D 

barcode scanning to record vaccine data that we haven’t covered, if any? 

Thank you very much for your time. Do you have any questions for me before we end our interview? 

37 
 

 

 

Practice Attributes 

MI 

Practice Type
 Public Health Department
 

Practice Size (# of physicians) 5 or fewer 

EMR System Vendor F 

Estimated 2D vaccine administration volume 3953- High 

Estimated % public vaccines 33% 

Administration and 

Process observed 

Inventory 

 

 

 

 

 

 

 

 

 

Appendix C: Practice Summaries 

Practice ID: 1187 

Overall Workflow Process 

Practice 1187 administers vaccines in nurse working stations that are set up in two separate patient care 

areas with 2D barcode scanners. There are laptops set up as the computers for these working stations, 

however, these laptops remained stationary during observations with the 2D barcode scanners attached to 

the laptops for the entire duration of the visit. 

The nurses always entered or scanned vaccine data before administering vaccines. While talking to the 

nursing manager, she noted there was a study done on-site previously that determined that order of 

operations was the most effective and efficient way to capture vaccine administration data with the least 

amount of errors. Four different nurses were observed during the visit—they comprise all of the nurses 

that use the 2D barcode scanners in this practice. There is an IT representative that rotates through all of 

the different practices in the county and is available to answer any IT related questions. He was on-site 

during the WFA visit, participated in all of the interviews, provided the WFA consultant a tour of the facility, 

and was the key point of contact prior to and during the visit. 

A refrigerator with approximately one-day’s supply of vaccines is located in a refrigeration preparation 

room next to the patient care rooms/nursing stations. A much larger refrigeration area, containing four 

large-scale refrigerators, is located at the back of the facility. Staff referred to this location as the “storage 

room” since it is the physical location where they store the remaining vaccines on-site. During the day of the 

visit, the varicella vaccine supply was only located in the storage room given a refrigeration issue in the 

preparation room. There is usually an additional smaller refrigerator in the preparation room where these 

are stored, but the refrigerator was not working on the days of observations. 

38 
Administration process 

 

 

 

 

 

 

The WFA consultant observed four individuals performing vaccine administration procedures while at this 

practice. These were all of the nurses that were administering vaccines on the days of observations. The 

typical steps during administration started with staff at the front desk preparing patient information and 

providing the applicable VIS statements to the patient’s parents. The nurse then calls the patient to the 

patient room where the nurse reviews each vaccine and the potential side effects, verifies the patient’s 

information, and reviews the vaccine schedule. Following this interaction with the patient, the nurse 

retrieves the vaccines to prepare—either from the refrigeration preparation room (which is between twenty 

and forty yards away from the patient rooms) or the storage room in the back of the building (during this 

visit nurses had to retrieve varicella from this location). 

Once the nurse returns to the patient room, they wash their hands, prepare the vaccine(s), and enter data 

into the practice’s EMR while sitting at the nurse’s station located in the patient room. Multiple data 

elements have drop down functionality in the EMR which pulls possible response options from the inventory 

records. 2D barcode scanners used at administration in this practice are configured to capture the lot 

number and expiration date. Vaccine data elements are saved once after the nurse enters data about all 

vaccines administered. After saving the data, the nurse completes any remaining vaccine preparation steps 

(such as gathering Band-Aids, alcohol swabs, etc.) and administers the vaccine(s). The nurse then updates 

the hard copy patient chart and writes updates on a yellow information sheet to give to the patient’s 

parents for their records. 

39 
Figure C-1. Overall vaccine administration process 

Chart Review 

Call patient into 

room 

Patient 

counseling 

Data Entry 

Vaccine 

Preparation 

Waste disposal 

Patient 

Counseling 

Vaccine 

preparation 

Waste Disposal 

Vaccine 

Administration 

Manual hard 

copy updates 

Save 

Immunization
 

Event 

Recorded
 

40 
 

 

 

 

 

 

 

 

 

 

The average time to perform all steps in the process depicted in Figure C-1 was 186.91 seconds (SD=89.11). 

The longest step in the process on average was vaccine administration whereas the shortest step was 

logging into the EMR. Table C-1 provides the average length of time for each step as well as the standard 

deviation associated with each of these steps. Table C-1 shows the steps captured for all observations; 

Figure C-1 shows the WFA of the most common steps that create the normal process for vaccine 

administration at this location. 

Table C-1. Average time to perform steps* in vaccine administration workflow 

Observations Mean 

2D input 17 20.79 8.87 

Call Patient into room 5 40.76 29.42302 

Chart Review 8 34.79 25.43955 

Data Entry 23 17.91 18.84738 

Linear input 31 27.14 10.96769 

Login to EMR 45 12.72 15.05072 

Patient Counseling 26 64.17 71.55357 

Vaccine Admin 3 89.00 54.10967 

Vaccine Prep 20 74.77 57.59485 

VIS Statement gathering 2 64.20 69.08295 

Walking 26 31.28 35.92996 

Wash Hands 10 24.80 5.702507 

Waste Disposal 4 31.13 22.64557 

*The “2D input” step in this chart refers to the specific time it took to perform the steps specific only to 2D scanning: picking up the 

scanner, scanning the barcode for lot # and expiration date, and pressing the ‘scan’ button when necessary. The “linear input” steps 

refer to the specific steps that would be performed in the traditional method instead of the 2D scanning process—so either 

manually typing or selecting from a drop box the lot number and expiration date. The “Data Entry” step refers to the steps that are 

performed for both processes-such as typing or drop down selection of manufacturer, input location, etc. The WFA consultant 

wanted to single out the steps that are directly comparable to measure the differences for the two processes: linear vs. 2D. 

Inventory process 

One individual, the front desk representative, is responsible for recording vaccine inventory data and was 

observed during the visit. Each time the practice receives a new shipment, she retrieves the boxes of 

vaccines from the shipping package and organizes the vaccines received by lot number and private versus 

public in a storage room located at the back of the office prior to scanning. She then enters data into both 

the EMR and the Michigan Care Improvement Registry (MCIR) state tracking system at a computer located in 

the back room. 

She first enters data into the EMR and subsequently enters data for all private vaccines into MCIR. Data for 

VFC vaccines is already included in MCIR (by the state) therefore it does not need to be entered by the 

practice. After data entry is complete, she puts the vaccine boxes back in the fridge and files the packing slip 

at the front desk clerical station. 

41 

Standard 

Deviation 
Michigan Care Improvement Registry EMR Inventory Process Inventory Process 

Saved to Saved to Inventory Inventory 

Retrieve box of 

vaccines from 

package 

Add a new lot 

number 

Group by Lot # 

Select VFC or 

Private 

Select Quantity 

Select 

Manufacturer 

Scan 2D Barcode 

Manually enter 

the Lot # 

Manually enter 

Expiration date 

Select Vaccine 

Save 

Separate VFC 

and Private 

Add a new lot 

number 

Select ‘Private’ 

for private 

vaccines 

Select Vaccine 

name 

Select Quantity 

Select 

Manufacturer 

Scan 2D Barcode 

Manually enter 

the Lot # 

Manually enter 

Expiration date 

Type NDC 

Save 

 

 

Figure C-2. Overall vaccine inventory process 

Blue fill indicates a data entry process step 

The average time to perform all steps in the process depicted in Figure C-2 is 38.61 seconds (SD=9.73). There 

was one person observed for the inventory process. The longest step in the process on average was 

recording of data from a linear barcoded vaccine whereas the shortest step was retrieving the packing slip. 

Table C-2 provides the average length of time for each key step as well as the standard deviation associated 

with each of these steps. 

42 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Table C-2. Average time to perform key steps* in vaccine inventory workflow 

Observations Average* 

2D input 8 8.400135 4.449885 

Linear input 4 22.00558 12.15877 

Data Entry 15 11.73131 11.28233 

Retrieve Packing Slip 6 6.099342 5.174005 

*The “2D input” step in this chart refers to the specific time it took to perform the steps specific only to 2D scanning: picking up the 

scanner, scanning the barcode for lot # and expiration date, and pressing the ‘scan’ button when necessary. The “linear input” steps 

refer to the specific steps that would be performed in the traditional method instead of the 2D scanning process—so either 

manually typing or selecting from a drop box the lot number and expiration date. The “Data Entry” step refers to the steps that are 

performed for both processes-such as typing or drop down selection of manufacturer, input location, etc. The WFA consultant 

wanted to single out the steps that are directly comparable to measure the differences for the two processes: linear vs. 2D. 

Data Entry Process & Time Measurements 

The same EMR is used for inventory and administration, and several comments were made throughout the 

day that the combination of both processes made sense to the nurses and that the two processes tie 

together nicely. The lot numbers were populated into the EMR during the inventory process, so that they 

automatically populate the drop down menu options available when entering vaccine administration data. 

The general procedure for entering vaccine inventory and administration data is described below and 

depicted in Figures C-3 and C-4. 

The general process for entering vaccine inventory data involves: 

•	 Entering data into the EMR by... 

•	 Selecting Private or VFC vaccine from drop down menu 

•	 Selecting vaccine name from drop down menu 

•	 Selecting vaccine manufacturer from drop down menu 

•	 Manually entering quantity 

•	 Entering lot number and expiration date: For vaccines with a linear barcode this data is 

manually entered, for 2D barcoded vaccines this data is scanned into the system 

•	 Saving the record after each vaccine’s data is entered 

•	 Entering data into MCIR by... 

•	 Selecting ‘Private’ from a drop down menu 

•	 Double checking for duplicate lot numbers 

•	 Entering vaccine name, manufacturer, lot number, NDC, number of doses, current date, and 

expiration date. Vaccine name and manufacturer are available in a drop down menu. NDC, 

number of doses, and current date are manually entered for all vaccine barcode types (i.e., 

linear or 2D). For 2D barcoded vaccines lot number and expiration date are input from the scan. 

• Saving vaccine data after entering each individual vaccine, 

The general process for entering data about vaccine administrations into the EMR includes: 

• A person from the front desk prepares patient information and gives VIS Statements to patient 

43 

Standard 

Deviation 
 

 

 

 

 

 

 

 

 

 

 

o	 

•	 Nurse calls patient and escorts them to the patient room 

•	 Nurse goes over each vaccine and details/side effects of the vaccine with the patient and verifies 

patient information and vaccine schedule 

•	 Nurse walks to the nurses room to get vaccines and for vaccine preparation 

Some of the vaccines required the nurse to walk to the supply room; one example was the 

varicella vaccine (Note: this is usually kept in the nurses room but the freezer is not working 

today) 

•	 Walks back to patient room and washes hands 

•	 Vaccine preparation 

•	 Data entry into the EMR (Select drop downs for linear: vaccine name, lot number, manufacturer, 

expiration date, administration location. If scan for 2D: lot number and expiration date populate, but 

select remaining from drop downs.) 

•	 Vaccine preparation and patient counseling at the same time-prepping for side effects/what to 

expect, (Note: prepping vaccines in patient area in front of patient) 

•	 Vaccine administration in patient room 

•	 Waste disposal 

•	 Update the patient chart/information manually on hard copy of vaccine records 

•	 Meet with parents to give information 

The process for entering a linear barcoded vaccine involves nurses selecting the correct lot number from a 

drop down menu. Vaccine data is saved at the end, and multiple vaccines can be entered in a list format 

before saving. The nurses generally knew which vaccines had 2D barcodes based on manufacturer based on 

memory and frequency of administration, but did not organize in a particular way during the administration 

process. There is an auto-populate feature in the EMR that automatically populates the lot number if there 

is only one lot number in the inventory, so if there was only one lot number, no scanning or selection of 

dropdowns was necessary. The NDC number is tied to the lot number during inventory, so it was not 

necessary to input again during the administration process. 

44 
 

Linear 2D 

Select Vaccine 

name 

Select Vaccine 

name 

Select Lot 

Number 

Select 

Manufacturer 

Select expiration 

date 

Scan 2D barcode 

on vaccine 

Select admin 

location Save 

Save 

Select admin 

location 

Select 

Manufacturer 

Figure C-3. Vaccine administration data capture process 

45 
 

 

 

12 time measurements were taken of the vaccine inventory data entry process at this practice for the on

individual observed. 8 time measurements were taken for data entry using 2D barcode scanners and 4 

measurements were recorded for linear barcoded vaccines. On average, it took 18.35 seconds less for the

front desk staff member to enter vaccine administration data using 2D barcode scanners (Mean = 13.4, SD

=5.41) than it did for them to manually enter data from vaccines with a linear barcode (Mean = 31.75, 

SD=11.87). 

 

Figure C-4. Vaccine inventory data capture process 

Add a new lot 

number 

Select VFC or 

Private 

Select Quantity 

Select 

Manufacturer 

Scan 2D Barcode 

Manually enter 

the Lot # 

Manually enter 

Expiration date 

Select Vaccine 

Save 

Blue fill indicates a data entry process step 

The WFA consultant recorded 40 time measurements of the vaccine administration data entry process at 

this practice for the four individuals observed. Twenty seven time measurements were taken for data entry 

using 2D barcode scanners and 13 measurements were recorded for linear barcoded vaccines. On average, 

it took 5.66 seconds less for nurses to enter vaccine administration data using 2D barcode scanners 

(Mean=22.63, SD=9.15) than it did for them to manually enter data from vaccines with a linear barcode 

(Mean=28.29, SD=9.78). 

e 

46 
 

 

 

 

 

 

Figure C-5. Time to record vaccine administration data by barcode type 

31.75 13.4 

0 

10 

20 

30 

40 

50 

A V E R A G E T I M E T O R E C O R D 

C O M P A R I S O N O F R E C O R D I N G T I M E 

2 D VE R SU S N ON -2 D B A R C O DE S 

P R A C T I C E 1 1 8 7 I N V E N T O R Y 

Linear 2D 

Figure C-6. Time to record vaccine inventory data by barcode type 

31.75 13.4 

0 

10 

20 

30 

40 

50 

A V E R A G E T I M E T O R E C O R D 

C O M P A R I S O N O F R E C O R D I N G T I M E 

2 D VE R SU S N ON -2 D B A R C O DE S 

P R A C T I C E 1 1 8 7 I N V E N T O R Y 

Linear 2D 

Other observations 

There was no written or formal policy at this location requiring staff to use the scanners, but the practice of 

using 2D barcode scanning to record vaccine data was heavily engrained in the organizational culture. The 

leadership at this practice supported using 2D barcode scanning, and continuously encouraged use of the 

scanners by pointing out the current and future benefits to the nurses. In response to asking if policies are 

in place, the Nursing Manager said, “There are not policies yet, but those will come now that we have it in 

place and we are using it and it is improving. At first they really did struggle with it, some of the codes 

weren’t as clear, the labels weren’t as dark so it was hard to pick it up but the staff told me that is 

improving. So now we are going to continue to use it, especially since the inventory balances nicely. So we 

47 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

are going to continue to use it, so now we put policies and procedures in place. They don’t have an option 

not to use it, so if it is there I want them using it. “ 

This practice appeared to use the scanners more frequently than any other practice observed for the WFA. 

The two nurses that were very positive regarding the use of 2D barcode scanners appeared to use them 

whenever possible. However, it should be noted that there was a recognition among staff that some of the 

nurses were hesitant to use the barcode scanners at first and that there was a learning curve before they 

were comfortable using the scanners. Staff noted that initially, the use of the scanners interrupted the 

normal routine however, it was noted that having a key information technology contact who could address 

issues as they arose was helpful. 

The nurses mentioned that more vaccines were currently 2D barcoded than during the pilot project—during 

the pilot project observation period the clinic only had Menactra vaccines with 2D barcodes and now they 

estimate that 50% of the vaccines have 2D barcodes. One of the nurses mentioned, “The benefit is that in 

case there was ever a recall you would know that it was in there correctly, you would be getting a hold of 

everyone who had received that vaccine-which is the whole purpose behind entering the lot numbers.” 

Some issues with the specific vaccines were noted by multiple nurses during the interviews and throughout 

the day. The influenza vaccine was singled out as the one that was the most challenging to scan. Nurses 

reported that all of the 2D barcodes on a curved surface (syringe) are the most difficult to scan. Also, some 

noted that Menactra is sometimes printed partially off of the label, and these were not able to be read by 

the scanner. However, this seemed to be a batch issue, because the newer shipments of Menactra do not 

have this issue. 

HepA, DTaP, and Polio consistently have 2D barcodes in this practice. HepA and DTaP were recognized as 

being the easiest to scan. Nurses noted that they currently have to look at each vaccine to see if there is a 

2D barcode available and that they believe the process will be faster and easier when all vaccines are 2D 

barcoded. Two nurses reported that they really liked the 2D barcode scanners and spoke very highly of 

them; one nurse did not think it helped the process and had trouble getting the scanners to quickly scan so 

did not think it saved time or effort—“ if you have multiple vaccines that is a challenge in the clinic room. 

From a nursing standpoint in the clinic I see no advantages to the 2D scanning barcode because we already 

find the lot number in a drop box. So we find the lot number and then we scan it so it is an extra step for us 

in the clinic room.” 

48 
Practice Attributes 

MI 

Practice Type
 Public Health Department
 

Practice Size (# of physicians) 5 or fewer 

EMR System Vendor B 

Estimated 2D vaccine administration volume 6295 -High 

Estimated % public vaccines 50% 

Administration and 

Process observed 

Inventory 

 

 

 

 

 

 

 

Practice ID: 2091 

Overall Workflow Processes 

Practice 2091 is one of two practices visited in the same local county health department (the other is 

practice 2452). Each patient room has a nursing station inside the room that includes a desktop computer 

with a barcode scanner attached. Each nurse is assigned to work in a single patient room. This practice 

keeps both electronic and hard copies of patient records. Notes are taken on the hard copy of the patient 

record while visiting with the patient and entered in the computer later in the workflow process. Labels 

from vaccines are removed from the vials and adhered to the hard copy record. 

Administration Process 

There are six nurses in this practice, four of which were observed for the administration process during the 

WFA. All nurses scheduled to work during the WFA were observed. The administration process begins with 

the nurse reviewing the patient information and charts. The nurse then retrieves the vaccine(s) from the 

refrigerator, brings the vaccines to the nursing station, and writes the names of the vaccines and lot 

numbers on the hard copy patient record. If the vaccine has a removable label, the nurse removes the label 

and attaches it to the hard copy patient record. Next, the nurse orders the vaccines through EMR system. 

Subsequently, the nurse calls the patient into the patient room, reviews the patient information with the 

family, provides the appropriate VIS statements, and performs patient counseling. Next, the nurse performs 

the vaccine preparation activities, administers the vaccine, disposes of waste, and updates information in 

the EMR. Finally, the nurse provides an updated immunization card to the parents for their records. 

49 
Access 

System
 

Review Charts 

Order vaccines 

in EMR 

Manually write 

Lot # on hard 

copy 

Remove labels 

from vaccines 

for hard copy 

records 

Give VIS 

statements 

Vaccine 

Preparation 

Save Record 

Vaccine 

Administration 

Waste Disposal 

Patient 

Counseling 

Update patient 

information 

Data Entry 

Handwritten 

updates to 

patient records 

Immunization
 

Event 

Recorded
 

Figure C-7. Overall vaccine administration process 

50 
 

 

 

 

 

The average time to perform all steps in the process depicted in Figure C-7 was 264 seconds (SD=160). The 

longest step in the process on average was calling the patient whereas the shortest step was logging into the 

electronic system. Table C-3 provides the average length of time for each step as well as the standard 

deviation associated with each of these steps. 

Table C-3. Average time to perform key steps* in vaccine administration workflow 

Observations Mean 

2D input 13 26.37 20.20 

Chart Review 10 41.92 61.09 

Data Entry 8 22.03 13.50 

Linear input 15 23.37 16.43 

Login 28 16.41 21.37 

Vaccine Admin 11 74.04 91.06 

Patient counseling 6 73.47 74.52 

Waste Disposal 7 19.56 16.13 

VIS Statement 5 51.36 43.39 

Call Patient 3 279.63 457.37 

Vaccine Prep 7 65.97 68.26 

*The “2D input” step in this chart refers to the specific time it took to perform the steps specific only to 2D scanning: picking up the 

scanner, scanning the barcode for lot # and expiration date, and pressing the ‘scan’ button when necessary. The “linear input” steps 

refer to the specific steps that would be performed in the traditional method instead of the 2D scanning process—so either 

manually typing or selecting from a drop box the lot number and expiration date. The “Data Entry” step refers to the steps that are 

performed for both processes-such as typing or drop down selection of manufacturer, input location, etc. The WFA consultant 

wanted to single out the steps that are directly comparable to measure the differences for the two processes: linear vs. 2D. 

Inventory Process 

The inventory process is performed centrally for all practices associated with the local county health 

department. Data is entered once a month by two Biological Clerks and always takes place at the receiving 

desk. Data is entered directly into the EMR. As a general procedure, when the lot number on the box is 

different from the lot number on the vial the clerks input the box lot number. One clerk was observed during 

the WFA. 

51 

Standard 

Deviation 
Saved to 

Inventory 

Log in to EMR 

Type Site Name 

Manual input 

Transaction Type 

Manual input 

Manufacturer 

Populate 

expiration date 

Type in Unit 

amount 

Click “Scan” 

Scan 2D Barcode 

on Vaccine 

Type in NDC 

Manual input Lot 

# 

Save 

Retrieve Packing 

Slip 

 

 

Figure C-8. Overall vaccine inventory process 

Blue fill indicates a data entry process step 

The average time to perform all steps in the process depicted in Figure C-8 was 191 seconds (SD=186.8)

21. 

The longest step in the process on average was entering 2D barcoded vaccine data whereas the shortest 

step was entering other data. As seen in Table C-3, the standard deviation associated with the longest step is 

21The average and standard deviation are based upon observations of the full inventory process for four times. 

52 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

o	 

o	 

o	 

o	 

o	 

very high. This is because the clerk observed used the 2D barcode scanner until it registered the data in the 

EMR—in some instances this took three or four tries before the data registered in the system. The normal 

process would have been to try once or twice and then revert to manual entry of the data. If the scan had 

worked the first time in each instance, this average time would be noticeably shorter. Table C-4 provides the 

average length of time for each step as well as the standard deviation associated with each of these steps. 

Table C-4. Average time to perform key steps* in vaccine inventory workflow 

Observations22 Average Standard 

Deviation 

2D input 11 40.0848 55.1664 

Linear input 13 19.44525 7.897434 

Enter Data 2 3.900254 0 

Retrieve Packing Slip 4 16.03948 10.61519 

*The “2D input” step in this chart refers to the specific time it took to perform the steps specific only to 2D scanning: picking up the 

scanner, scanning the barcode for lot # and expiration date, and pressing the ‘scan’ button when necessary. The “linear input” steps 

refer to the specific steps that would be performed in the traditional method instead of the 2D scanning process—so either 

manually typing or selecting from a drop box the lot number and expiration date. The “Data Entry” step refers to the steps that are 

performed for both processes-such as typing or drop down selection of manufacturer, input location, etc. The WFA consultant 

wanted to single out the steps that are directly comparable to measure the differences for the two processes: linear vs. 2D. 

Data Entry Process & Time Measurements 

This practice uses one of the few EMRs that are 2D barcode enabled, as such upon scanning a 2D barcode 

the information encoded on the barcode (lot number, expiration date, and product identifier) are entered 

into the record along with several other data elements specific to the vaccine scanned. 

The typical process used for recording data about vaccine inventory includes: 

•	 Log in to EMR 

•	 Type an “R” into the Transaction Type field to indicate the vaccine is from ‘Receiving’ 

•	 For linear barcoded vaccines... 

Type in the practice name for the destination location and enter the lot number 

Select manufacturer from a drop down menu. 

Type the expiration date, amount of units and NDC 

•	 For 2D barcode scan... 

Press “scan” icon on the screen 

Scan 2D barcoded vaccine – populates the expiration date and lot number; if the date is 

ever entered incorrectly, the date must be selected from the calendar view to continue. 

She will then type in the amount of units and NDC. 

22 In some of the processes observed the individual entered more than one vaccine into inventory (2D or linear) and for 

some of the processes observed the individual did not have to enter additional data. 

53 
 

 

 

 

 

 

 

 

 

 

 

 

o	 

o	 

o	 

o	 

o	 

o	 

o	 

o	 

The EMR links the vaccine information entered during inventory to pull vaccine information such as lot 

number, vaccine name, and NDC from input during the inventory scanning process. This is used to populate 

the drop down menus. If the lot number is already in the system, it will populate everything except for the 

units. 

To record data about vaccines administered the nurse... 

•	 Logs into EMR 

•	 For linear barcoded vaccines... 

Select lot number, expiration date, manufacturer, vaccine location from drop down menus 

that are pre-populated from the inventory process. 

•	 For 2D barcode scan... 

Presses the ‘scan’ button icon on the screen 

Scans a single 2D barcoded vaccine – which inputs the lot number and expiration date. The 

vaccine name is pulled from when the vaccines were previously ordered, and the 

manufacturer and location is selected with pre-populated drop down menus that are prepopulated from the inventory input. 

Repeat above process for each 2D barcoded vaccine administered 

After data for all vaccines has been entered the information is saved. The typical process used for recording 

data about vaccine inventory includes: 

•	 Log in to EMR 

•	 Type an “R” into the Transaction Type field to indicate the vaccine is from ‘Receiving” 

•	 For linear barcoded vaccines... 

Type in the practice name for the destination location and enter the lot number 

Select manufacturer from a drop down menu. 

•	 For 2D barcode scan... 

Press “scan” icon on the screen 

Scan 2D barcoded vaccine – populates the expiration date, and if the date is ever 

entered incorrectly, the date must be selected from the calendar view to continue. She 

will then type in the amount of units and NDC. If the lot number is already in the 

system, it will populate everything except for the units. 

54 
Linear 2D 

Scan 2D Barcode 

on Vaccine 

Select 

Manufacturer 

Manually key in 

Expiration Date 

Select Lot # 

Select 

Manufacturer 

Select location 

Select location 

Nineteen time measurements were taken of the vaccine administration data entry process at this practice 

for the 4 individuals observed. Twelve time measurements were taken for data entry using 2D barcode 

scanners and seven measurements were recorded for linear barcoded vaccines. On average, it took 16.68 

more seconds for nurses to enter vaccine administration data using 2D barcode scanners (Mean=39.31, SD 

=22.87) than it did for them to manually enter data from vaccines with a linear barcode (Mean = 22.63, 

SD=12.22). 

Figure C-9. Vaccine administration data capture process 

55 
 

Linear 2D Barcode Scan 

Log in to EMR 

Type Site Name 

Manual input 

Transaction Type 

Saved to 

Inventory 

Manual input 

Manufacturer 

Populate 

expiration date 

Type in Unit 

amount 

Type in NDC 

Manual input Lot 

# 

Save 

Log in to EMR 

Type Site Name 

Manual input 

Transaction Type 

Manual input 

Manufacturer 

Type in Unit 

amount 

Click “Scan” 

Scan 2D Barcode 

on Vaccine 

Type in NDC 

Save 

Saved to 

Inventory 

Figure C-10. Vaccine inventory data capture process 

Blue fill indicates a data entry process step 

Twenty-four time measurements were taken of the vaccine inventory data entry process at this practice for 

the one individual observed. Thirteen time measurements were taken for data entry using 2D barcode 

scanners and 11 measurements were recorded for linear barcoded vaccines. On average, it took 20.63 more 

seconds for nurses to enter vaccine administration data using 2D barcode scanners (Mean=40.08, SD 

=55.17) than it did for them to manually enter data from vaccines with a linear barcode (Mean =19.45, 

SD=7.90). 

56 
 

 

 

 

Figure C-11. Comparison of time to record vaccine administration by barcode type 

22.63 39.31 

0 

10 

20 

30 

40 

50 

60 

70 

A V E R A G E T I M E T O R E C O R D 

C O M P A R I S O N O F R E C O R D I N G T I M E 

2 D VE R SU S N ON -2 D B A R C O DE S 

P R A C T I C E 2 0 9 1 A D M I N I S T R A T I O N 

Linear 2D 

Figure C-12. Comparison of time to record vaccine inventory by barcode type 

19.45 40.08 

-40.00 

-20.00 

0.00 

20.00 

40.00 

60.00 

80.00 

100.00 

120.00 

A V E R A G E T I M E T O R E C O R D 

C O M P A R I S O N O F R E C O R D I N G T I M E 

2 D VE R SU S N ON -2 D B A R C O DE S 

P R A C T I C E 2 0 9 1 I N V E N T O R Y 

Linear 2D 

Other observations 

Accuracy was noted as being the greatest advantage to using 2D barcodes scanners. One individual working 

on inventory provided an example of such benefits, “I think a week ago we typed in the wrong lot number 

for a vaccine we received in because it didn’t have the 2D barcode and once those are administered to our 

patients then when we find out after so many have been administered, someone notices ‘oh the wrong lot 

number was put in’ we have to take all of those immunizations out of the client’s records, fix the mistake we 

made, re-enter it into [EMR name] and then re-enter it into all of the client records. And it’s okay if it is like 

57 
 

 

 

 

 

 

 

 

 

 

5 but we have done 75 before - and I’ve put the lot number and someone even double checked me. So 

scanning is good because it is accurate.” 

Nurses reported a similar issue to what was experienced at the other practice observed in this local county 

health department with respect to the EMR system freezing up with the use of 2D barcode scanners. An 

episode of the system freezing was seen during the WFA visit. During one scan the flu vaccine did not 

register a scan within 30 seconds. The EMR system subsequently froze and the nurse had to exit out of the 

EMR and start data entry over. The nurses noted that the system freezing is inconsistent—sometimes they 

can use the scanner for extensive periods without the system freezing. The flu vaccine froze the system 

twice during the observations. 

The nurses mentioned being nervous about the possibility of the system freezing and this nervousness 

affected how they elected to use the scanners. For instance, one nurse mentioned avoiding the use of 2D 

barcode scanners when their patient is a crying baby. Additionally, nurses always scanned the vaccines with 

2D barcodes first in the process so that if the system froze during the process it would not adversely affect 

the inventory numbers (lot numbers are pulled from the inventory). 

Staff noted that some shipments of the same vaccines have 2D barcodes and some do not. Examples include 

HepA, Pediarix, and the different Flu vaccines administered in this practice. Nurses also mentioned that 

having to think to look for the 2D barcode was a challenge and decreased the frequency with which they use 

the scanner. They noted that if all of the vaccines were 2D barcoded, this would eliminate confusion, and 

potentially increase the frequency of use. 

58 
 

Practice Attributes 

WA 

Practice Type Pediatrics 

Practice Size (# of physicians) 5 or fewer 

EMR System Vendor D 

Estimated 2D vaccine administration volume 2688- High 

Estimated % public vaccines 100% 

Process observed Administration 

 

 

 

 

 

 

 

 

 

 

Practice ID: 2133 

Overall Workflow Processes 

Practice 2133 has three barcode scanners located at three nursing stations. Two of these nursing stations 

are located central the patient rooms and the third is in the refrigeration room/prep area. The preparation 

area is directly to the right of the refrigerator. 

Two nurses were observed at this location. These are the two nurses that use the scanners on a regular 

basis. There was a college nursing student in training shadowing one of the nurses during the visit23. All 

vaccinations? were observed unless there was overlap between the two nurses. 

The vaccine administration process24 begins when the nurse obtains VIS sheets and pulls up the patient’s 

information on the computer in the exam room. A system alerts the nurses’ station that the patient is 

present and the nurse proceeds to the waiting area to call the patient to the exam room where they review 

the patient information and cover general wellness topics. During this visit the nurse updates the patient’s 

information and general vaccine information in the EMR at a computer located in the exam room. The nurse 

then reviews wellness questions, updates the patient chart, and takes measurements (e.g., weight). 

Following the visit with the nurse, the patient sees the physician who tells the nurse which vaccine(s) are 

approved for the patient. The nurse retrieves the vaccine(s) from the refrigeration room, writes down the 

patient’s information on the syringes, prepares the vaccines (in the preparation area), and enters data about 

the vaccine into the EMR at the nurses station located outside of the exam room. 

23 This differed from other practices, therefore interpretation of overall workflow times should take into account the time it takes to
 

explain a procedure, etc. to the trainee.

24 Due to office policies, observations were not made in the room with the patients at this facility, so the “vaccine administration” 

was timed as when the nurse went into the room after inputting the information. Because this facility input the information “before
 

administration” the steps in the data capture process were observed at the nurse’s station prior to vaccine administration.
 

59 
Access 

System
 Gather VIS 

sheets 

Review patient 

information 

Call patient 

Order Vaccines 

Vaccine 

Preparation 

Save Record 

Data Entry 

Type any 

additional notes 

Patient 

counseling 

Vaccine 

Administration 

Immunization
 

Event 

Recorded
 

The average time to perform all steps in the process depicted in Figure C-26 was 109.83 seconds (SD=62.34). 

The longest step in the process on average was vaccine administration whereas the shortest step was login. 

Table C-5 provides the average length of time for each step as well as the standard deviation associated with 

each of these steps. 

Figure C-13. Overall vaccine administration process 

60 
 

 

 

 

 

 

 

 

 

 

 

 

Table C-5. Average time to perform key steps* in vaccine administration workflow 

Observations Mean 

2D input 9 27.14 16.49765 

Chart Review 1 16.20 

Data Entry 7 11.29 10.23495 

Linear input 14 23.19 10.63652 

Login 18 4.23 3.034865 

Vaccine Admin 2 210.54 297.1116 

Vaccine Prep 7 132.22 72.22315 

Walking 7 12.73 2.437497 

*The “2D input” step in this chart refers to the specific time it took to perform the steps specific only to 2D scanning: picking up the 

scanner, scanning the barcode for lot # and expiration date, and pressing the ‘scan’ button when necessary. The “linear input” steps 

refer to the specific steps that would be performed in the traditional method instead of the 2D scanning process—so either 

manually typing or selecting from a drop box the lot number and expiration date. The “Data Entry” step refers to the steps that are 

performed for both processes-such as typing or drop down selection of manufacturer, input location, etc. The WFA consultant 

wanted to single out the steps that are directly comparable to measure the differences for the two processes: linear vs. 2D. 

Data Entry Process & Time Measurements 

The first time point at which the nurses access the EMR is to enter a standard set of data during the patient 

examination. A standard list of questions regarding patient information is displayed in the EMR in survey 

format. During the patient visit, the nurse asks this set of questions and enters the data using a series of 

drop down menus. 

The data entry process specifically related to vaccines administered begins after the physician leaves the 

patient room and tells the nurse which vaccines should be administered. When entering data about linear 

barcoded vaccines the nurse selects the vaccine name, lot number, manufacturer, administration location, 

expiration date, and clinic location using drop down menus. When entering data about 2D barcoded 

vaccines, the nurse scans the 2D barcode and the lot number is entered into the EMR. After the lot number 

is entered, the nurse must select the vaccine name, manufacturer, administration location, expiration date, 

and clinic location using drop down menus. For both types of vaccines, the nurse completes the data entry 

process by manually typing any additional notes and saves the record after entering each vaccine. 

61 

Standard 

Deviation 
 

 

 

 

 

Linear 2D 

Select Vaccine 

name 

Select Lot 

Number 

Select 

Manufacturer 

Select expiration 

date 

Select admin 

location 

Save 

Select admin 

location 

Select Vaccine 

name 

Scan 2D Barcode 

on Vaccine 

Select 

Manufacturer 

Select admin 

location 

Save 

Select expiration 

date 

Select admin 

location 

 

Figure C-14. Vaccine administration data capture process 

The WFA consultant recorded 21 time measurements of the vaccine administration data entry process at 

this practice for the 2 individuals observed. Thirteen time measurements were taken for data entry using 2D 

barcode scanners and 8 measurements were recorded for linear barcoded vaccines. On average, it took 3.5 

seconds longer for nurses to enter vaccine administration data using 2D barcode scanners (Mean =28.16, SD 

=17.33 ) than it did for them to enter data from vaccines with a linear barcode (Mean = 24.66, SD=9.47). 25 

25 Due to the anonymity of data collection we do not have a nurse specific identifier to assess what, in any, effect the student nurse 

may have had on the time to record vaccine data at this location. The student nurse was not observed as part of the study; she was 

also observing the process for learning purposes. 

62 
 

 

 

 

 

Figure C-15. Time to record vaccine administration data by barcode type 

24.66 28.16 

0 

10 

20 

30 

40 

50 

A V E R A G E T I M E T O R E C O R D 

C O M P A R I S O N O F R E C O R D I N G T I M E 

2 D V E R SU S N O N -2 D B A R C O DE S 

P R A C T I C E 2 1 3 3 

Linear 2D 

Other observations 

There are no formal policies at this practice requiring nurses to use the 2D barcode scanners, however, the 

leadership and physicians encourage use of the scanners because they perceive it increases the accuracy of 

lot numbers. Nurses mentioned some inconsistencies with scanning that presented challenges— 

inconsistency in one scanner’s performance and some variability between the ease with which they were 

able to scan different types of vaccines. 

Two scanners were used at this location, but one of the barcode scanners was not scanning as consistently 

as the other. When cleaned with an alcohol swab, it seemed to work more effectively. During the 

interviews, one of the nurses mentioned that some vaccines do not scan as well as others and noted 

difficulties scanning the HepA and influenza vaccines. Nurses noted that some shipments of vaccines have 

2D barcodes and some do not. The vaccines nurses recognize as consistently having 2D barcodes are: 

Pentacel, DTaP, HepA, Pediarix, and Menactra. 

It should be noted that due to scheduling challenges, the WFA consultant visited this practice during its least 

busy days – Tuesday and Wednesday. 

63 
 

 

 

 

 

The typical vaccine administration process26 started with the nurse administering the vaccines speaking 

with the physician to determine which vaccines will be ordered. While in the room with the patient, the 

physician places an order for the vaccines. Once this order is placed, the nurse receives an alert on the 

computer to retrieve the vaccine(s). The nurse enters data about the vaccine(s) into the EMR after retrieving 

them from the refrigeration preparation area. Nurses always enter vaccine data before administration in this 

practice. Several of the nurses grouped the vaccines based on whether there was a 2D barcode on the 

vaccine or not prior to recording the vaccine data. After the vaccine data is saved in the EMR, the nurse 

prepares the vaccine(s) while in the refrigeration area. In addition, the nurse writes information about the 

vaccines on hard copy of daily office vaccine data sheet. After disposing of waste from the vaccine 

preparation, the nurse obtained the appropriate VIS sheets and rubber gloves when exiting the refrigeration 

room on the way to the patient room where the nurse administered vaccines. 

 

Practice ID: 2168 

Practice Attributes 

NJ 

Practice Type Pediatrics 

Practice Size (# of physicians) 6 to 15 

EMR System Vendor G 

Estimated 2D vaccine administration volume 2517- High 

Estimated % public vaccines 40% 

Process observed Administration 

Overall Workflow Processes 

At Practice 2168, the preparation rooms are located on the opposite ends of the building with six patient 

rooms surrounding each. A nursing station with a computer and 2D barcode scanner were set up in both 

refrigeration preparation areas. Both nursing stations had desktop computers with scanners attached that 

were within five feet of the refrigerator; there was a portable tablet used at this location also, but there was 

no scanner attached to the tablet. The inventory process was not observed because 2D scanners are not 

involved in that process. 

Six different nurses were observed during the vaccine administration process. Ten different nurses rotate 

through this office; all six that were working at this practice during the two days were observed. Each of the 

nurses has received training on the 2D barcoding process; however nurses can choose whether regular use 

is helpful. The primary contact was the nursing supervisor, and she was included in the six observed in the 

study. 

26 Observations related to patient engagement (e.g., patient counseling and the vaccine administration) were not observed at this 

practice. The WFA consultant was allowed to view processes that took place in the refrigeration preparation and nursing station 

areas. 

64 
Access 

System 

Front Desk 

update patient 

info and alert 

nurses when 

patient is ready 

Chart Review 

Login to Orders 

vaccines 

Data Entry 

Vaccine 

Preparation 

Give hard copy 

records back to 

front desk 

Administer 

Vaccines 

Vaccine 

preparation 

VIS Sheets 

Waste Disposal 

Update patient 

chart 

Save 

Patient 

Counseling 

Immunization 

Event 

Recorded
 

Figure C-16. Overall vaccine administration process 

65 
 

 

 

 

 

 

 

 

 

 

The average time to perform all steps in the process depicted in Figure C-23 was 194.13 seconds 

(SD=210.61). The longest steps in the process on average were patient counseling and vaccine 

administration whereas the shortest step was waste disposal. Table C-6 provides the average length of time 

for each step as well as the standard deviation associated with each of these steps. 

Table C-6. Average time to perform key steps* in vaccine administration workflow 

Observations Mean 

2D input 19 15.85 9.31 

Chart Review 31 36.92 58.26 

Data Entry 6 27.88 14.37 

Linear input 38 14.19 6.89 

Login to EMR 30 5.67 5.63 

Vaccine Admin 6 147.18 64.15 

Patient 

Counseling 5 147.39 162.93 

Waste Disposal 5 5.59 3.34 

Vaccine Prep 29 59.46 46.78 

VIS Statement 6 33.62 32.84 

Walking 3 49.63 64.71 

*The “2D input” step in this chart refers to the specific time it took to perform the steps specific only to 2D scanning: picking up the 

scanner, scanning the barcode for lot # and expiration date, and pressing the ‘scan’ button when necessary. The “linear input” steps 

refer to the specific steps that would be performed in the traditional method instead of the 2D scanning process—so either 

manually typing or selecting from a drop box the lot number and expiration date. The “Data Entry” step refers to the steps that are 

performed for both processes-such as typing or drop down selection of manufacturer, input location, etc. The WFA consultant 

wanted to single out the steps that are directly comparable to measure the differences for the two processes: linear vs. 2D. 

Data Entry Process & Time Measurements 

Most of the information about a vaccine administered is pulled into the patient record from the inventory 

record, therefore, the manual typing in this system was minimal irrespective of whether a linear or 2D 

barcoded vaccine was recorded. In all cases, the ‘auto-complete’ function of the lot number field pulled 

from the last lot number that was entered for that vaccine in the system. When recording about a vaccine 

with a linear barcode, the nurse manually types in the vaccine lot number and expiration date. For the 2D 

process, the nurse scans the 2D barcode to record the lot number and expiration date. After the lot number 

and expiration date are in the system, all other information is pulled from the inventory record, with the 

exception of the vaccine manufacturer which the nurse must select from a drop down list. The nurse verifies 

that the data in the record is correct and saves the record. Right after the nurse enters the vaccine data into 

the EMR, they were required to separately enter the vaccines in the State Registry system. 

66 

Standard 

Deviation 
 

‘ ’ 

 

 

Figure C-17. Vaccine administration data capture process 

Select Vaccine 

name 

Press Scan 

Button 

Select Vaccine 

name 

Select Lot 

Number 

Select 

Manufacturer 

Select expiration 

date 

Scan 2D barcode 

on vaccine 

Linear 2D 

Select admin 

location 

Select VFC or 

Private 

Save Save 

Select admin 

location 

Select 

Manufacturer 

Select VFC or 

Private 

The WFA consultant recorded 43 time measurements of the vaccine administration data entry process at 

this practice for the 6 individuals observed. Twenty five time measurements were taken for data entry using 

2D barcode scanners and 18 measurements were recorded for linear barcoded vaccines. On average, it took 

4.16 seconds longer for nurses to enter vaccine administration data using 2D barcode scanners (Mean = 

18.83, SD =7.11) than it did for them to manually enter data from vaccines with a linear barcode (Mean = 

14.67, SD=6.46). 

67 
“If it worked all of the time, I don’t see that 

there would be a drawback. Because it is quick, 

it is efficient, you have no errors. I think it is 

fantastic. But it doesn’t work all of the time.” 

Nurses identified several challenges in recording vaccine data. First, the nurses noted that the Prevnar and 

Flu vaccines were the most difficult to scan. Additionally, some of the scanners perceived as easier to use 

than others. Nurses also noted frustrations with having to look to see if a 2D barcode is on each vaccine and 

noted that they would be more likely to use 2D barcode 

scanning if on a regular basis if all vaccines had them. One 

of the nurses stated that she is bothered by the red light on 

the scanner because she feels it is blinding, so she prefers 

not to use it; she also stated that she thinks it takes too long 

to scan sometimes. A notable benefit identified by staff 

from this practice was not having to decipher numbers in 

the lot numbers so they could type them into the EMR—one nurse provided a specific example of it being 

difficult to tell the difference between a six and eight in some of the lot numbers. 

Figure C-18. Time to record vaccine administration data by barcode type 

14.67 18.83 

0 

5 

10 

15 

20 

25 

30 

A V E R A G E T I M E T O R E C O R D 

C O M P A R I S O N O F R E C O R D I N G T I M E 

2 D V E R SU S N O N -2 D B A R C O DE S 

P R A C T I C E 2 1 6 8 

Linear 2D 

Other Observations 

Individuals in this practice generally thought that the greatest benefit of using 2D barcode scanners was that 

it eliminated human error of inputting lot numbers. The scanners were provided for use, but not required— 

nurses could use them or not as they chose. 

68 
 

Practice Attributes 

FL 

Practice Type Pediatrics 

Practice Size (# of physicians) 5 or fewer 

Vendor C (different from 

EMR System Pilot EMR) 

Estimated 2D vaccine administration volume 672- Medium 

Estimated % public vaccines 40% 

Process observed Administration 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Practice ID: 2299 

Overall Workflow Process 

Practice 2299 is small in size and is currently set up to use 2D barcode scanning to record vaccines 

administered. The practice configuration enables lot number and expiration date to scan into the EMR. The 

practice comprises one patient room, with a refrigeration and preparation station in a room located across 

the hall. There is one nurse’s station in the practice which is set up directly to the left of the refrigerator. 

This is the only nurse’s station and the only barcoding scanner observed at this location. There are two 

additional scanners kept on-site, but only one of them is used on a consistent basis. The other two are for 

back-ups and to use on very busy days. 

Two individuals were observed while at this practice. One nurse is primarily responsible for administering 

vaccines in this practice and therefore regularly uses the 2D barcode scanner to record vaccine 

administration data. The second individual observed (a Medical Assistant), administers vaccines when the 

main nurse is out of the office, and therefore has an opportunity to use the 2D barcode scanner. 

The general workflow associated with vaccine administration begins with the patient visiting with the 

physician who subsequently places an order for the vaccines. The nurse then retrieves the patient’s 

information on the computer and reviews their chart for insurance information and immunization records. 

The nurse also reviews paper copies of records if needed. Subsequently the nurse pulls the vaccines 

ordered by the physician from the refrigerator and separates the vaccine vials or syringes into groups 

arranged by private and public vaccines. She then inputs the information into the EMR system by tabbing 

across and manually typing: vaccine name, manufacturer, expiration date, lot #, practice, location, 

administrator’s initials; private vs. public; and saves after all vaccines have been entered. After the data 

entry portion of the process, the nurse updates the hard copy patient chart with vaccines given and the 

date. Data entry of vaccine information always occurred prior to vaccine administration. In the final stages 

of the visit the nurse calls the patient into the room, administers the vaccines, cleans up, and conducts 

patient counseling. Figure C-19 depicts this overall workflow process for the administration process. 

69 
Access 

System
 

Chart Review 

Vaccine 

Preparation 

Vaccine 

Administration 

Hard copy 

updates 

Waste disposal 

Patient 

counseling 

Data Entry 

Log in to EMR 

Immunization
 

Event 

Recorded
 

Figure C-19. Overall Administration Process 

The average time to perform all steps in the process depicted in Figure C-19 was 229.24 seconds 

(SD=448.44). The longest step in the process on average was vaccine administration whereas the shortest 

step was Login. Table C-7 provides the average length of time for each step as well as the standard deviation 

associated with each of these steps. 

70 
 

 

 

 

 

 

 

 

 

 

Table C-7. Average time to perform key steps* in vaccine administration workflow 

Observations Mean 

2D scanning 14 27.97 19.67 

Chart Review 10 21.32 15.51 

Data Entry 12 27.25 16.84 

Linear input 12 50.85 33.52 

Login 14 5.62 7.57 

Vaccine Admin 3 455.19 505.59 

Patient counseling 1 287.43 

Waste Disposal 2 8.08 0.70 

Vaccine Prep 8 46.22 31.51 

*The “2D input” step in this chart refers to the specific time it took to perform the steps specific only to 2D scanning: picking up the 

scanner, scanning the barcode for lot # and expiration date, and pressing the ‘scan’ button when necessary. The “linear input” steps 

refer to the specific steps that would be performed in the traditional method instead of the 2D scanning process—so either 

manually typing or selecting from a drop box the lot number and expiration date. The “Data Entry” step refers to the steps that are 

performed for both processes-such as typing or drop down selection of manufacturer, input location, etc. The WFA consultant 

wanted to single out the steps that are directly comparable to measure the differences for the two processes: linear vs. 2D. 

Data Entry Process & Time Measurements 

During the pilot observation period, this practice used a different EMR than the current one and was 

configured to scan at both inventory and administration. Since this time, the practice changed their EMR 

system; the practice currently is unable to use with 2D barcode scanners to record inventory. 

The EMR has a separate screen that is automatically viewed by the end-user to enter vaccine information. 

The data entry fields in the EMR are entered manually (i.e., no drop-down functionality). Each data entry 

field is set-up to auto-populate words that are typed frequently. Observations of the process indicated that 

this was easy to follow and only required a couple of letters because of the auto-populate feature—as the 

nurse tabbed between fields quickly. In the case of linear barcoded vaccines, the lot number was manually 

entered—this was noticeably the longest step in the data entry process. When the 2D barcode scanner was 

used, the nurse simply scanned the 2D barcode when the lot number field was reached while tabbing across. 

This automatically populated the next field—the expiration date. Figure C-20 depicts the primary steps in 

the data entry process for linear and 2D barcoded vaccines. 

71 

Standard 

Deviation 
Twenty-five time measurements were taken at this practice for the two individuals observed. Eleven time 

measurements were taken for data entry using 2D barcode scanners and 14 measurements were recorded 

for linear barcoded vaccines which were manually entered. On average, it took 14.31 fewer seconds for the 

nurses to enter vaccine administration data using 2D barcode scanners (Mean = 29.15, SD = 19.36) than it 

did for them to manually enter data from vaccines with a linear barcode (Mean = 43.46, SD=29.08). 

Figure C-20. Vaccine administration data capture process 

Scan 2D Barcode 

on Vaccine 

Manually type 

Manufacturer 

Manually key in 

Expiration Date 

Manually type 

Lot # 

Manually type 

Manufacturer 

Manually type 

location 

Manually type 

location 

Linear 2D 

Manually key in 

Vaccine name 

Manually type 

Nurse initials 

Manually type 

VFC or Private 

Save 

Manually type 

nurse initials 

Manually type 

VFC or Private 

Save 

72 
“I like the fact that it is convenient, you 

just tap where the lot # goes, you scan it, 

and there is less typing so there are less 

errors. That is what I like about it. My 

experience is great.” 

 

 

 

 

 

 

 

 

Figure C-21. Time to record vaccine administration data by barcode type 

43.46 29.15 

0 

10 

20 

30 

40 

50 

60 

70 

80 

A V E R A G E T I M E T O R E C O R D 

C O M P A R I SO N O F R E C O R D I N G T I M E 

2 D VERSUS N ON -2 D B A R C O DE S 

P R A C T I C E 2 2 9 9 

Linear 2D 

Other observations 

This is a long-standing health care practice. The newest person on the staff (who was the one mainly 

observed) has been employed at the practice for nine years whereas two individuals who staff the front 

desk and the physician have been here for over 20 years. These individuals on some occasions have cared 

for up to four generations of patients; they know their patients 

well. It is a family-like atmosphere. 

Only one of the nurses in this practice uses the 2D barcode 

scanner on a regular basis—she is the only practitioner who 

regularly administers vaccines in this practice. She was very 

positive about the technology—it is clear that she really likes it 

and talks very positively about it to the others in the office. 

Feedback throughout the day by staff at this location indicated 

that there were no issues with the scanner operating. The nurse who primarily uses the 2D barcode scanner 

noted that she wished the barcode pulled more information than just the lot number and expiration date 

and remarked that even populating the manufacturer name it would save a lot of time. The flu vaccine was 

noted as the slowest to scan; the HepA, DTaP, and Menactra vaccines were noted as being “easy to scan.” 

The nurse noted that she liked that the barcode scans pick up the case sensitivity, because the Vendor C 

EMR requires the lot numbers to be case sensitive. 

73 
Practice Attributes 

NY 

Practice Type Pediatrics 

Practice Size (# of physicians) 5 or fewer 

EMR System Vendor A 

Estimated 2D vaccine administration volume 3613/High 

Estimated % public vaccines 30% 

Process observed Administration 

 

 

 

 

 

 

 

 

Practice ID: 2413 

Overall Workflow Processes 

Practice 2413 has eight nursing stations. Five of these stations are located in a room with the refrigeration 

and preparation area, the other three stations are located in a separate room. Each nurse has her own 

designated area with a computer, preparation space, and a 2D barcode scanner. Nurses prepare vaccines 

and enter the vaccine administration data at the nursing station to which they are specifically assigned. 

Generally, five nurses are on duty, therefore some of the stations are available and used by floating staff 

who call patients to set up appointments and give patient’s results or updates. 

All nurses staffed to work during the WFA visit were observed, therefore a total of five nurses were 

observed. The leadership at this office was very positive about the use of 2D barcode scanners, and all of 

the nurses adopted them into their normal routine. 

The vaccine administration process was initiated when a nurse saw a ‘blue dot’ next to the patient 

information list available in the EMR. This blue dot appears when the physician orders a vaccine while in the 

patient room and alerts the nurse that a vaccine is needed. Typically, the nurse would update the patient 

record in the EMR at this time and retrieve the vaccine(s) from the refrigerator. The nurses used the vaccine 

preparation area to group the vaccines by barcode type (2D or linear). The 2D barcodes were usually done 

first to save time. Although nurses determined whether they would enter vaccine information prior to or 

after administering a vaccine, they most frequently entered this data prior to administration After entering 

the vaccine data, the nurse prepared the vaccine(s) at their respective nursing station and entered the 

patient care area once the physician visit was complete. Once in the room, the nurse answered questions 

from the patient’s parents, verified the patient’s date of birth, and administered the vaccine(s). Following 

vaccine administration, the nurse disposed of any waste, and reviewed the respective VIS statements with 

the parents. 

74 
or 

Access 

System 

Wait for alert that 

the vaccine is 

ordered 

refrigerator f 

Walk to 

Vaccines 

Chart Review 

Vaccine 

Preparation 

Patient 

Counseling 

Vaccine 

Administration 

Authentication 

screen 

Save Record 

Waste Disposal 

Give VIS 

Call patient and 

escort to patient 

room 

Walk back to 

Nurses Station 

Data Entry 

Type any 

additional notes 

into Note Field 

Immunization
 

Event 

Recorded
 

Figure C-22. Overall vaccine administration process 

75 
 

 

 

 

 

 

 

The average time to perform all steps in the process depicted in Figure C-22 was 309 seconds (SD=282). The 

longest step in the process on average was Chart Review whereas the shortest step was Login. Table C-8 

provides the average length of time for each step as well as the standard deviation associated with each of 

these steps. 

Table C-8. Average time to perform key steps* in vaccine administration workflow 

Observations Mean 

2D input 16 29.15 13.02821 

Call Patient 3 78.21 35.97192 

Chart Review 12 79.65 120.3064 

Data Entry 28 15.40 23.55034 

Linear input 30 30.95 10.6209 

Login to EMR 51 8.16 5.651279 

Patient Counseling 23 46.44 85.37094 

Vaccine Admin 20 48.01 26.32805 

Vaccine Prep 37 66.42 66.58643 

VIS Statement 27 27.80 52.11368 

Walking 34 30.25 46.37036 

Waste Disposal 20 16.39 10.79679 

*The “2D input” step in this chart refers to the specific time it took to perform the steps specific only to 2D scanning: picking up the 

scanner, scanning the barcode for lot # and expiration date, and pressing the ‘scan’ button when necessary. The “linear input” steps 

refer to the specific steps that would be performed in the traditional method instead of the 2D scanning process—so either 

manually typing or selecting from a drop box the lot number and expiration date. The “Data Entry” step refers to the steps that are 

performed for both processes-such as typing or drop down selection of manufacturer, input location, etc. The WFA consultant 

wanted to single out the steps that are directly comparable to measure the differences for the two processes: linear vs. 2D. 

Data Entry Process & Time Measurements 

As previously noted, Vendor A is used in this practice. To enter vaccine administration data for vaccines with 

a linear barcode, the nurse logs into the EMR and manually enters all data elements. The 2D barcoding scan 

is initiated immediately when input screen is pulled up. For 2D barcoded vaccines, the nurse scans the lot 

number and expiration date and manually enters all remaining data elements. After the nurse enters all of 

the vaccine data, they enter information into an authentication screen and then save. 

76 

Standard 

Deviation 
 

Linear 2D 

Move cursor to 

Lot number field 

Scan 2D Barcode 

on Vaccine 

Type 

Manufacturer 

Type Vaccine 

Name 

Type Date 

Type Lot Number 

Type 

Manufacturer 

Type Expiration 

Date 

Type Date 

Save 

Save 

Figure C-23. Vaccine administration data capture process 

The WFA consultant took 40 time measurements of the vaccine administration data entry process at this 

practice for the 5 individuals observed. Twenty-seven time measurements were taken for data entry using 

2D barcode scanners and 13 measurements were recorded for linear barcoded vaccines. On average, it took 

2.88 fewer seconds for nurses to enter vaccine administration data using 2D barcode scanners (Mean = 

29.58, SD =10.21) than it did for them to manually enter data from vaccines with a linear barcode (Mean = 

32.46, SD= 9.8). 

77 
 

 

 

 

 

 

 

 

 

 

 

 

 

Figure C-24. Comparison of time to record vaccine data by barcode type 

32.46 29.58 

0 

10 

20 

30 

40 

50 

A V E R A G E T I M E T O R E C O R D 

C O M P A R I SO N O F R E C O R D I N G T I M E 

2 D VERSUS N ON -2 D B A R C O DE S 

P R A C T I C E 2 4 1 3 

Linear 2D 

Other observations 

The clinic is in a small town and the patients consistently visit the clinic for all types of appointments. The 

nursing manager mentioned that the Pediatric Department was often recognized throughout the local 

health system as being a ‘best in class’ area. Most of the nurses offered very positive feedback about the 

use of 2D barcode scanners. They felt that the scanners worked easily and that use of the scanners 

increased the accuracy of the lot numbers entered. A common theme was that nurses preferred to use the 

scanners because it eliminated the need to read very small lot numbers. One individual offered a specific 

example—noting that sometimes there are five zeroes in a row in a lot number and it can be easy to miss 

one when manually recording this data. 

The most common challenge noted by the nurses was the wait time associated with the scanner reading the 

barcode. Nurses noted that individual syringes were the least consistent to scan. Additionally, staff noted 

that not all lots of the same vaccine have the 2D barcodes; some shipments have them and some do not— 

indicating that the supply of 2D barcoded vaccines may still be inconsistent in the supply chain. 

There was no formal policy in place at this practice requiring staff to use the 2D barcode scanners. However, 

the leadership encouraged the use of the scanners on a regular basis. Nurses in this practice appeared to be 

very collaborative—they often helped each other mid-process. This is one possible reason why 2D barcode 

scanning was accepted among all of the nurses. 

A printed list of which vaccines had 2D barcodes was kept by one of the lead nurses, but was not distributed. 

This was mainly because it was dependent on shipments whether the vaccines had barcodes, so this list 

changes frequently. The nurse with the list sat close to the refrigerator so it was easy to check the list when 

retrieving the vaccines. The 2D barcode scanners only captured the Lot number and expiration date, so each 

78 
 

 

entry still manually entered any notes and the publication date of the VIS. There was a ‘cheat sheet’ of 

these expiration dates at each station. 

79 
 

Practice Attributes 

NY 

Practice Type Pediatrics 

Practice Size (# of physicians) 5 or fewer 

EMR System Vendor E 

Estimated 2D vaccine administration volume 1779- Medium 

Estimated % public vaccines 25% 

Process observed Administration 

 

 

 

 

 

 

 

Practice ID: 2429 

Overall Workflow Processes 

Practice 2429 has two nurse stations that include a desktop computer with 2D barcode scanners attached. 

These nurse stations are located in the refrigerator preparation room that is centrally located to the six 

patient rooms in this practice. Everyone working in this practice, which includes approximately 10 people 

(including front-desk staff), has access to the 2D barcode scanners. Although nurses are responsible for 

administering vaccines, other individuals in the practice enter data about vaccines administered into the 

EMR. 

Five nurses were observed at this location—this comprises all nurses who used the 2D barcode scanners and 

were administering vaccines during the WFA visit. At the beginning of the visit, and whenever a new nurse 

started working, the WFA consultant requested to be informed about when vaccines were to be 

administered. 

The vaccine administration process27 typically started when the front desk staff updated a sheet containing 

patient information which they subsequently brought to a window in the nurse’s room. When a new patient 

was available, the front desk person said “chart” to alert the nurses. At this time, the nurse took the patient 

information sheet to the nurse’s station and accessed the patient’s information in the EMR. While in the 

EMR, the nurse ordered the vaccine(s), reviewed the patient information to verify past records, and entered 

additional information as needed about the activities planned for the current day’s appointment (e.g., eye 

exam, blood work). Subsequently, the nurse retrieved the vaccine(s) from the refrigerator and entered 

vaccine data into the EMR. Data about vaccines was always entered prior to administering the vaccine in 

the practice. 

 

27Observations related to patient engagement (e.g., patient counseling and the vaccine administration) were not observed at this 

practice. The WFA consultant was allowed to view processes that took place in the vaccine preparation area and during data entry. 

80 
 

 

 

 

After entering the vaccine data, the nurse completed vaccine preparation activities (e.g., ready band aids, 

open alcohol swabs) and gathered the appropriate VIS sheets on the way on the way to the patient room. 

When in the exam room, the nurse administered the vaccine(s). Following vaccine administration, the nurse 

returns the patient record to front desk and logs back on to a computer in the nurse’s station to update the 

patient chart noting that the vaccines were given and updating patient information/vitals captured during 

the visit (e.g., height, weight, and blood work). 

81 
Access 

System
 

Front Desk 

update patient 

info and alert 

nurses when 

patient is ready 

Chart Review 

Login and Order 

vaccines 

Data Entry 

Vaccine 

Preparation 

Give hard copy 

records back to 

front desk 

Administer 

Vaccines 

Vaccine 

preparation 

VIS Sheets 

Waste Disposal 

Update patient 

chart 

Save 

Immunization
 

Event 

Recorded
 

Figure C-25. Overall process for vaccine administration 

The average time to perform all steps in the process depicted in Figure C-25 was 281.13 seconds 

(SD=248.79). The longest step in the process on average was patient counseling whereas the shortest step 

82 
 

 

 

 

 

 

 

 

 

was logging into the EMR. Table C-9 provides the average length of time for each step as well as the 

standard deviation associated with each of these steps. 

Table C-9. Average time to perform key steps* in vaccine administration workflow 

Observations Mean 

2D inputs 11 37.64 31.12 

Chart Review 7 31.62 37.16 

Data Entry 14 36.96 28.40 

Linear inputs 15 19.26 9.81 

Login to EMR 16 10.31 7.83 

Vaccine Admin 6 168.23 155.63 

Patient Counseling 1 547.56 

Waste Disposal 1 43.49 

Vaccine Prep 17 102.80 215.62 

Order shots 2 21.45 10.38 

Walking 2 49.28 32.07 

Handwritten Notes 2 10.47 5.60 

*The “2D input” step in this chart refers to the specific time it took to perform the steps specific only to 2D scanning: picking up the 

scanner, scanning the barcode for lot # and expiration date, and pressing the ‘scan’ button when necessary. The “linear input” steps 

refer to the specific steps that would be performed in the traditional method instead of the 2D scanning process—so either 

manually typing or selecting from a drop box the lot number and expiration date. The “Data Entry” step refers to the steps that are 

performed for both processes-such as typing or drop down selection of manufacturer, input location, etc. The WFA consultant 

wanted to single out the steps that are directly comparable to measure the differences for the two processes: linear vs. 2D. 

Data Entry Process & Time Measurements 

Staff entered all data about a vaccine into a central box located in the middle of the screen by selecting 

values from drop down boxes. For linear barcoded vaccines this involved selecting the vaccine’s lot 

number28, expiration date, manufacturer, VFC status (VFC or not), and vaccine administration location. The 

procedure for entering data about 2D barcoded vaccines differs slightly. Staff must press a ‘scan’ button 

before using the scanners for each vaccine administered. Scanning the 2D barcode enters the lot number 

and expiration date into the EMR. The staff member saves the vaccine record after entering data for each 

vaccines administered. 

28 Lot numbers are pre-entered into the system during the inventory process. 

83 

Standard 

Deviation 
‘ ’ 

 

 

 

 

 

 

Figure C-26. Vaccine administration data capture process 

Select Vaccine 

name 

Press Scan 

Button 

Select Vaccine 

name 

Select Lot 

Number 

Select 

Manufacturer 

Select expiration 

date 

Scan 2D barcode 

on vaccine 

Linear 2D 

Select admin 

location 

Select VFC or 

Private 

Save Save 

Select admin 

location 

Select 

Manufacturer 

Select VFC or 

Private 

The WFA consultant recorded 31 time measurements were taken of the vaccine administration data entry 

process at this practice for the 5 individuals observed. Sixteen time measurements were taken for data entry 

using 2D barcode scanners and 15 measurements were recorded for linear barcoded vaccines. On average, 

it took 9.74 seconds longer for nurses to enter vaccine administration data using 2D barcode scanners 

(Mean = 32.08, SD =28.08) than it did for them to manually enter data from vaccines with a linear barcode 

(Mean = 22.34, SD= 8.04). 

84 
 

 

 

 

 

 

 

 

 

 

Figure C-27. Time to record vaccine administration data by barcode type 

22.34 32.08 

0 

10 

20 

30 

40 

50 

60 

70 

A V E R A G E T I M E T O R E C O R D 

C O M P A R I S O N O F R E C O R D I N G T I M E 

2 D VE R SU S N ON -2 D B A R C O DE S 

P R A C T I C E 2 4 2 9 

Linear 2D 

Other observations 

This is a pediatric practice that sees patients for many types of medical visits other than vaccine 

administration. This is a very busy practice with several doctors (n=5) and up to 10 nurses present in the 

office at a time. During the WFA most patient appointments were for procedures other than vaccinations. 

There were five nurses observed during the WFA, however, most of these individuals reported that they do 

not use the 2D barcode scanners on a regular basis because 

they perceive it is easier to enter vaccine data manually, 

particularly when the lot numbers are already loaded as part 

of the inventory process. None of the nurses observed 

worked in this practice during the pilot project observation 

period. One individual, who was employed in this practice during the pilot project observation period, was 

on vacation during the WFA. Staff noted that she is the one that likes 2D barcode scanning the best and 

uses it the most frequently. 

Nurses noted that if scanning the 2D barcode resulted in all of the vaccine data populating into the EMR that 

this process would be very helpful and save a lot of time. The nurses also mentioned that a lot number is 

“randomly chosen” when using a 2D barcode scanner. This is likely due to an auto-complete configuration in 

the EMR that matches the scanned lot number with the first few digits of lot numbers loaded in at inventory 

(instead of matching the entire lot number). Nurses are concerned that scanning a 2D barcode may result in 

an incorrect entry and therefore double check the entry regularly—causing an extra step in the data entry 

process. 

Nurses appreciated the peel-off labels on the flu vaccine. One noted that it was helpful to adhere the peel 

off label to the billing sheet so it could be scanned later. More than one nurse remarked that it seemed 

“Flu season it was especially nice because it had 

a peel off label and could stick it to the billing 

sheet and scan it later. It seems easier because 

it is not rounded.” 

85 
easier to scan a 2D barcode from labels that they peeled off—mentioning difficulties scanning barcodes on a 

rounded surface. One nurse also mentioned that Pentacel seemed easier to scan than Varivax. 

86 
Practice Attributes 

MI 

Practice Type
 Public Health Department
 

Practice Size (# of physicians) 5 or fewer 

EMR System Vendor B 

Estimated 2D vaccine administration volume 6295 -High 

Estimated % public vaccines 50% 

Administration and 

Process observed 

Inventory 

 

 

 

 

 

 

 

 

 

 

 

Practice ID: 2452 

Overall Workflow Processes 

Practice 2452 is one of two practices visited in a local county health department in Michigan during the 

WFA; the second facility is Practice 2091. There are minor differences between the workflow processes at 

these two practices—the primary difference relates to vaccine administration, namely the location of exam 

rooms. At the practice described in the current case study, all exam rooms are located on the same hallway 

whereas in Practice 2091 the rooms are distributed more broadly throughout the facility. Otherwise the 

procedures used at these sister practices are very similar for recording vaccine administration and are 

exactly the same for recording vaccine inventory. The WFA consultant did observe both administration and 

inventory data entry procedures for this practice, however, given that inventory procedures are performed 

by exactly the same individuals observed for practice 2091 inventory findings are not repeated here. 

Administration Process 

Four nurses were observed through the vaccine administration process during this visit. Five nurses use the 

barcode scanners in this practice, however, only four were present during observations. The vaccine 

administration process begins with the nurse reviewing the patient information and charts. The nurse then 

retrieves the vaccine(s) from the refrigerator and returns to the nursing station to write down the names of 

the vaccines and lot numbers on the hard copy patient record. If the vaccine has a removable label, the 

nurse adheres the label to the hard copy patient record. Next, the nurse orders the vaccines via the EMR 

system. After the nurse places this order, she calls the patient into the exam room, reviews the patient’s 

information with the family, provides the relevant VIS statements, and performs patient counseling. Vaccine 

preparation then takes place, followed by vaccine administration, and waste disposal. The nurse then 

updates information into the EMR. Nurses, however, can elect to input in the vaccine data either before or 

after administration. The Nurse Manager noted that she prefers to input vaccine data before administration 

occurs, because this procedure affords the opportunity to ensure the correct vaccine is administered by 

double checking the lot numbers match. As a final step in this process, the nurse provides an updated 

(handwritten) immunization card to the patient’s parents for their records. 

Most of the nurses left the barcode scanner on the stand when recording vaccine administrations. Some, 

however, have the scanner physically taped to a desk because they found that the specific angle allows 

87 
 

 

 

 

them to scan the barcode much easier and want to keep the exact specifications for future scans. When 

administering a group of vaccines—some 2D barcoded and some linear barcoded—the nurses would 

typically enter data for the 2D barcoded vaccines first. Multiple instances occurred where the EMR system 

would freeze when recording 2D barcoded vaccines, when this occurred, nurses noted that all vaccine data 

entered prior to the system freezing was lost. 

88 
Access 

System
 

Review Charts 

Order vaccines 

in EMR 

Manually write 

Lot # on hard 

copy 

Remove labels 

from vaccines 

for hard copy 

records 

Give VIS 

statements 

Vaccine 

Preparation 

Save Record 

Vaccine 

Administration 

Waste Disposal 

Patient 

Counseling 

Update patient 

information 

Data Entry 

Handwritten 

updates to 

patient records 

Immunization
 

Event 

Recorded
 

Figure C-28. Overall vaccine administration process 

89 
 

 

 

 

 

 

 

 

 

 

 

 

o	 

o	 

o	 

o	 

o	 

The average time to perform all steps in the process depicted in Figure C-28 was 30.35 seconds (SD=23.99). 

The longest step in the process on average was Vaccine preparation whereas the shortest step was Logging 

into the EMR. Table C-10 provides the average length of time for each step as well as the standard deviation 

associated with each of these steps. 

Table C-10. Average time to perform key steps* in vaccine administration workflow 

Observations Mean 

2D input 4 14.61 4.86 

Chart Review 2 9.68 11.03 

Data Entry 2 8.61 8.54 

Linear input 15 9.50 3.91 

Login 15 7.20 10.17 

Vaccine Prep 1 23.82 

*The “2D input” step in this chart refers to the specific time it took to perform the steps specific only to 2D scanning: picking up the 

scanner, scanning the barcode for lot # and expiration date, and pressing the ‘scan’ button when necessary. The “linear input” steps 

refer to the specific steps that would be performed in the traditional method instead of the 2D scanning process—so either 

manually typing or selecting from a drop box the lot number and expiration date. The “Data Entry” step refers to the steps that are 

performed for both processes-such as typing or drop down selection of manufacturer, input location, etc. The WFA consultant 

wanted to single out the steps that are directly comparable to measure the differences for the two processes: linear vs. 2D. 

Inventory Process 

Please refer to description provided for Practice 2091. 

Data Entry Process & Time Measurements 

This practice uses one of the few EMRs that are 2D barcode enabled, as such upon scanning a 2D barcode 

the information encoded on the barcode (lot number, expiration date, and product identifier) are entered 

into the record along with several other data elements specific to the vaccine scanned. 

The typical process used for recording data about vaccine inventory includes: 

•	 Log in to EMR 

•	 Type an “R” into the Transaction Type field to indicate the vaccine is from ‘Receiving’ 

•	 For linear barcoded vaccines... 

Type in the practice name for the destination location and enter the lot number 

Select manufacturer from a drop down menu. 

Type the expiration date, amount of units and NDC 

•	 For 2D barcode scan... 

Press “scan” icon on the screen 

Scan 2D barcoded vaccine – populates the expiration date and lot number; if the date is 

ever entered incorrectly, the date must be selected from the calendar view to continue. 

She will then type in the amount of units and NDC. 

90 

Standard 

Deviation 
 

 

 

 

 

 

o	 

o	 

o	 

o	 

o	 

o	 

Linear 2D 

Scan 2D Barcode 

on Vaccine 

Select 

Manufacturer 

Manually key in 

Expiration Date 

Select Lot # 

Select 

Manufacturer 

Select location 

Select location 

The EMR links the vaccine information entered during inventory to pull vaccine 

information such as lot number, vaccine name, and NDC from input during the inventory 

scanning process. This is used to populate the drop down menus. If the lot number is 

already in the system, it will populate everything except for the units. 

To record data about vaccines administered the nurse... 

•	 Logs into EMR 

•	 For linear barcoded vaccines... 

Select lot number, expiration date, manufacturer, vaccine location from drop down menus 

that are pre-populated from the inventory process. 

•	 For 2D barcode scan... 

Presses the ‘scan’ button icon on the screen 

Scans a single 2D barcoded vaccine – which inputs the lot number and expiration date. The 

vaccine name is pulled from when the vaccines were previously ordered, and the 

manufacturer and location is selected with drop down menus that are pre-populated from 

the inventory input. 

Repeat above process for each 2D barcoded vaccine administered 

After data for all vaccines has been entered the information is saved 

Figure C-29. Vaccine Administration data capture process 

Please refer to Practice 2091 case for information regarding the process used for entering vaccine inventory 

data. 

91 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Figure C-30. Time to record vaccine administration data by barcode type 

10.53 14.90 

0 

5 

10 

15 

20 

25 

A V E R A G E T I M E T O R E C O R D 

C O M P A R I SO N O F R E C O R D I N G T I M E 

2 D VERSUS N ON -2 D B A R C O DE S 

P R A C T I C E 2 4 5 2 

Linear 2D 

Other Observations 

Two main challenges staff at this practice consistently reported include – (1) having to press “scan” before 

scanning the 2D barcode on each vaccine and (2) a system freezing issue when using 2D barcode scanners. 

In order for the EMR to read the 2D barcode, the nurse had to press the “scan” button before each vaccine 

was scanned. Staff indicated that this added a step to the data recording process and noted that it would 

greatly increase the time of the process even if the “scan” button only had to be pressed once (not before 

each vaccine). 

Perhaps the most pressing complaint from staff was that the scanners cause the system to freeze 

intermittently. If nothing was recognized by the scanner within a time frame (approximately 30 seconds), 

the system would freeze which requires the user to go into Task Manager (or hit CTRL+ALT+Delete), close 

the program, and then restart. During this process any previously entered vaccine data is lost. Inventory 

interview suggested that one out of four generally scans on the first try. This freezing issue occurs when 

recording data about vaccine inventory and vaccine administrations. Such issues may have been particularly 

frustrating for this location, since staff noted that they are able to enter information manually very quickly 

into the EMR—the WFA consultant observed data entry times as quick as 6 seconds for recording linear 

barcoded vaccines. 

“I can definitely see the potential in the barcodes, and I think it is an excellent idea in theory. I’m not 

sure if it is the scanners or our computer system, but the execution currently is not ideal. I can see 

this being something really great in the future once the kinks get worked out, the 2D barcode holds 

more information and the system stops freezing. The lot numbers are really small and hard to read, 

so I like that it takes away the risk of keying it in wrong.” 

92 
 

 

 

 

 

 

 

 

 

 

 

The staff in this practice were focused on making the 2D barcode scanners work consistently—they 

indicated that the 2D barcode scanners seem very “sensitive” and/or “picky” because the nurses have to 

hold it exactly right for the barcodes to pick up. Staff tried multiple approaches to increase the probability of 

the barcodes scanning on the first try. In an effort to ensure appropriate alignment of the 2D barcode 

scanners they have used the following techniques: duct taped the scanner to a desk so that it maintains a 

consistent orientation and located a scanner so it aligns with a solid surface instead of a desktop (the 

desktop has a wood grain pattern on it and the scanners try to pick up that surface). 

“I have it aimed towards my sharps container because it is solid. I find that when it is against this 

textured desk it is always flickering, the light is very wide, and I can scan when it is like that. I tried 

moving it, so it would be more conducive but I was having problems so I just found that it being up 

against something solid makes it happier as you might like to say.” 

“I have to put it down so it is half on the table and half on a stack of copier paper and then I run it up 

along the side of the wall. If I move the scanner just a little bit then I sometimes will have a harder 

time scanning it. A lot of people will do it up against here (the side of the computer) or up against 

the wall. Once you find your position, then it is great. If someone moves it then it is like “oh my 

god”—if we had a bigger area that we can just go “boom” and it scans, that would be fantastic. It is 

kind of a narrow margin.” 

Staff also made some observations about specific vaccines. For instance, they singled out Flu-T as the 2D 

barcoded vaccine that worked best with the scanner. In at least two interviews the following vaccines were 

identified as consistently having 2D barcodes: Pediarix, DTap, Flu, HepA. They also noted that removing the 

labels from syringes tended to make it easier to scan the 2D barcode. Practitioners felt that the difficulty 

scanning was related to the curvature of the syringes. 

93 
Practice Attributes 

WA 

Practice Type Family Practice 

Practice Size (# of physicians) 6 to 15 

EMR System Vendor A 

Estimated 2D vaccine administration volume 1613-Medium 

Estimated % public vaccines 100% 

Process observed Administration 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Practice ID: 2508 

Overall Workflow Process 

Practice 2508 is a multi-purpose practice (family practice, walk-in clinic, emergency care, and pediatrics 

were all in the same building). The practice comprises three floors, on which there are nurses seeing 

patients. Nurses working on the first floor were selected for observations since this is where the majority of 

vaccines are administered in the practice (i.e., many appointments for newborns). Nurses working on the 

second floor were also interviewed for the WFA, but were not observed during vaccine administration. 

There are four nurse’s stations located on the first floor—each station has a 2D barcode scanner and a 

desktop computer. The station is at the entry of the patient care area and there are eight patient rooms 

surrounding the nurse station area. There are five workstations on the second floor with barcode scanners, 

but only two stations had scanners. Responses to the initial requests for information about the practice 

during recruitment indicated that the facility used the scanners for inventory, however, while visiting the 

practice the WFA consultant was informed that that recording vaccine inventory with 2D barcode scanners 

was not part of the normal procedures. 

Vaccine administration procedures varied greatly between nurses at this practice. There were two aspects 

of the process that always occurred—(1) a physician always visited the patient before the nurse and 

subsequently informed the nurse about what vaccine(s) to administer and (2) the nurses always entered 

vaccine data after administration. Four different nurses were observed for the WFA. Two of the nurses 

used the 2D barcode scanners on a regular basis, one noted that she used the scanner sometimes, and the 

other said she never used the scanners but tried it since it was part of the WFA to observe. 

The vaccine administration process began with the nurse reviewing the patient chart and patient 

information. The nurse then orders the vaccine(s); nurses used a container to carry vaccines into the patient 

room and the vaccines are labeled and separated into different standardized portions of the container to 

help with confusion and entering data. After she prepares the vaccines, administers the vaccines in the 

patient area, she walks back to nurse’s station to pull up patient chart in the EMR, enters the vaccine 

information including the date with notes, and saves the record. Nurses are required to order the vaccines 

from inventory before entering data about them into the system—-some nurses order the vaccines before 

administration and some do it at the same time as inputting the information. 

94 
Review Charts/ 

Patient 

Information 

Login/Order 

vaccines in EMR 

Vaccine 

Preparation 

Review patient 

charts 

Vaccine 

Preparation 

Save Record 

Data Entry 

Write notes 

Vaccine 

Administration 

Immunization
 

Event 

Recorded
 

Figure C-31. Overall administration process 

The average time to perform all steps in the process depicted in Figure C-31 was 256.58 seconds 

(SD=178.61). The longest step in the process on average was vaccine administration whereas the shortest 

step was data entry. Table C-11 provides the average length of time for each step as well as the standard 

deviation associated with each of these steps. 

95 
 

 

 

 

 

2D 10 42.19 22.67395 

Chart Review 7 45.24 37.38789 

Data Entry 7 26.50 27.29573 

Linear 16 42.07 26.66739 

Login 29 70.16 190.5879 

Vaccine Admin 5 248.59 304.3082 

Patient counseling 6 84.14 156.2082 

*The “2D input” step in this chart refers to the specific time it took to perform the steps specific only to 2D scanning: picking up the 

scanner, scanning the barcode for lot # and expiration date, and pressing the ‘scan’ button when necessary. The “linear input” steps 

refer to the specific steps that would be performed in the traditional method instead of the 2D scanning process—so either 

manually typing or selecting from a drop box the lot number and expiration date. The “Data Entry” step refers to the steps that are 

performed for both processes-such as typing or drop down selection of manufacturer, input location, etc. The WFA consultant 

wanted to single out the steps that are directly comparable to measure the differences for the two processes: linear vs. 2D. 

Data Entry Process & Time Measurements 

As previously noted, this practice uses Vendor A EMR. The configuration of this practice’s EMR required the 

nurses to manually type all data about vaccines as there are no drop down menus included in this system. 

Data about vaccines was always entered after administration and in the case of linear barcoded vaccines 

consisted of manually typing the vaccine manufacturer, lot number (2D scans would capture), expiration 

date (2D scans would capture), the NDC number, input practice, and insurance information. When scanning 

in a 2D barcoded vaccine the only difference was that lot number and expiration date were scanned into the 

system nurses entered all other data manually. After entering all of the data, the nurse saved the record 

once all vaccines were entered. 

96 

Observations Mean 

Standard 

Deviation

Table C-11. Average time to perform key steps* in vaccine administration workflow 
 

Linear 2D 

Scan 2D Barcode 

on Vaccine 

Manually type 

Manufacturer 

Manually key in 

Manufacturer 

Manually key in 

Lot # 

Manually type 

Expiration date 

Manually type 

location 

Manually update 

insurance type 

Save 

Manually type 

NDC 

Manually update 

Insurance type 

Save 

Manually key in 

Manufacturer 

Manually key in 

Manufacturer 

Figure C-32. Vaccine Administration data capture process 

The WFA consultant recorded 63 time measurements of the vaccine administration data entry process at 

this practice for the 4 individuals observed. Thirty-one time measurements were taken for data entry using 

2D barcode scanners and 32 measurements were recorded for linear barcoded vaccines. On average, it took 

16.79 fewer seconds for nurses to enter vaccine administration data using 2D barcode scanners (Mean 

=27.76, SD =16.39) than it did for them to enter data from vaccines with a linear barcode (Mean = 44.55, 

SD=22.66). 

97 
 

 

 

 

 

 

 

 

Figure C-33. Time to record vaccine administration data by barcode type 

44.55 27.76 

0 

10 

20 

30 

40 

50 

60 

70 

80 

A V E R A G E T I M E T O R E C O R D 

C O M P A R I S O N O F R E C O R D I N G T I M E 

2 D VE R SU S N ON -2 D B A R C O DE S 

P R A C T I C E 2 5 0 8 

Linear 2D 

Other observations 

This practice does not have formal policies requiring the use of 2D barcode scanners to record vaccine data. 

Additionally, nurses expressed that they were not generally informed about the scanners. Some of the 

newer employees did not seem aware that it was an option to use 2D barcode scanners. 

One of the nurses noted that she used the 2D barcode scanner as a paper weight, because it only scans in 

the lot number and expiration date and does not scan in the National Drug Code (NDC) number. The NDC 

number is lengthy and nurses in this practice are required to type it into the system manually. Another nurse 

noted that she would get frustrated with process if the scanner did not read information right away and 

would switch to manually typing. 

It should be noted that patient flow was somewhat slower than normal during the days of the WFA visit. 

However, the WFA consultant was able to get permission to conduct mock scans and worked with the 

nurses to perform these to increase the number of observations. 

98 
 

Practice Attributes 

WA 

Practice Type Pediatrics 

Practice Size (# of physicians) 16 or more 

EMR System Vendor A 

Estimated 2D vaccine administration volume 938 - Medium 

Estimated % public vaccines 100% 

Process observed Administration 

 

 

 

 

 

 

 

 

 

 

 

Practice ID: 2510 

Overall Workflow Processes 

Practice 2510 has five nursing stations. These stations are adjacent to each other and situated across from 

the work rooms. Each nursing station is equipped with a 2D barcode scanner. There is also a scanner set up 

in the Nursing Manager’s office attached to her computer. Only four out of the six 2D barcode scanners 

worked at the time of observations—staff placed post-it notes on the scanners that are not operable. 

Five nurses use the 2D barcode scanners in this practice, three of these nurses were observed for the WFA. 

The three nurses were chosen based on schedule. The vaccine administration process29 in this practice 

started with the physician handing the nurse the patient information chart after visiting the patient and 

informing the nurse as to which vaccine(s) need to be administered. The nurse then walked from the nursing 

station (located outside of the exam room) to the refrigeration room/preparation station to prepare the 

vaccine(s). Subsequently, the nurse walked back to the exam room and administered the vaccine(s). 

Following the vaccine administration, the nurse logged into the EMR on the computer located at the nursing 

station, reviewed the patient’s information and entered data about the vaccines administered. Whether 

nurses elected to use the 2D barcode scanner on a regular basis depended on whether there was a working 

2D barcode scanner at the nursing station where they were working. Nurses always entered vaccine data 

after administration. 

29 Due to office policies, the WFA consultant was unable to observe activities in the patient area or able to be present during patient 

interactions. Observations were therefore limited to what took place at the nurses’ station. The time measured as ‘Vaccine Prep’ is 

assumed as the entire time in preparation area, and ‘Vaccine Administration’ for the entire time in the room with the patient. 

99 
Access 

System
 

Patent 

counseling 

Vaccine 

Administration 

Login to Order 

vaccines 

Vaccine 

Preparation 

Chart Review 

Data Entry 

Immunization
 

Event 

Recorded
 

Figure C-34. Overall vaccine administration process 

The average time to perform all steps in the process depicted in Figure C-34 was 185.91 seconds 

(SD=117.77). The longest step in the process on average was vaccine administration whereas the shortest 

step was logging in. Table C-12 provides the average length of time for each step as well as the standard 

deviation associated with each of these steps. 

100 
 

 

 

 

 

 

 

Table C-12. Average time to perform key steps* in vaccine administration workflow 

Observations Mean 

2D input 1 19.28 

Chart Review 8 24.03 34.08 

Data Entry 3 22.48 15.83 

Linear input 8 27.49 12.25 

Login 7 7.01 5.79 

Vaccine Admin 4 208.68 296.62 

Patient Counseling 1 3.58 

Waste Disposal 1 12.14 

Vaccine Prep 3 167.54 74.54 

Walking 3 8.48 2.44 

*The “2D input” step in this chart refers to the specific time it took to perform the steps specific only to 2D scanning: picking up the 

scanner, scanning the barcode for lot # and expiration date, and pressing the ‘scan’ button when necessary. The “linear input” steps 

refer to the specific steps that would be performed in the traditional method instead of the 2D scanning process—so either 

manually typing or selecting from a drop box the lot number and expiration date. The “Data Entry” step refers to the steps that are 

performed for both processes-such as typing or drop down selection of manufacturer, input location, etc. The WFA consultant 

wanted to single out the steps that are directly comparable to measure the differences for the two processes: linear vs. 2D. 

Data Entry Process & Time Measurements 

For vaccines with a linear barcode, the nurse enters the following data using drop down menus and then 

saves the record—vaccine name, input location, manufacturer, expiration date, and lot number. Minimal 

differences exist between entering data for linear barcoded vaccines and 2D barcoded vaccines. When 

entering data for 2D barcoded vaccines, the nurse picks up the barcode scanner after they select the 

manufacturer from the drop down menu. They then scan the barcode which populates the expiration date 

and lot number. All other data entry fields are selected from the respective drop down menus and the 

record is saved after each vaccine entry. 

101 

Standard 

Deviation 
 

Linear 2D 

Select Vaccine 

name 

Select Location 

Select 

Manufacturer 

Select Expiration 

Date 

Select Location 

Select Lot # 

Select 

Manufacturer 

Select Vaccine 

name 

Save 

Scan 2D Barcode 

Save 

Figure C-35. Vaccine administration data capture process 

The WFA consultant recorded 16 time measurements of the vaccine administration data entry process at 

this practice for the 3 individuals observed. Four time measurements were taken for data entry using 2D 

barcode scanners and 12 measurements were recorded for linear barcoded vaccines. On average, it took 

9.44 seconds less seconds for nurses to enter vaccine administration data using 2D barcode scanners (Mean 

=20.03, SD =3.22) than it did for them to enter data from vaccines with a linear barcode (Mean = 29.47, 

SD=9.57 ). 

102 
29.47 20.03 

0 

10 

20 

30 

40 

50 

A V E R A G E T I M E T O R E C O R D 

C O MP A R I SO N O F R E C O R D I N G T I ME 

2D VERSUS NON-2 D B A R C ODE S 

P R A C T I C E 2 5 1 0 

Linear 2D 

 

 

 

 

 

 

 

 

 

Figure C-36. Time to record vaccine administration data by barcode type 

Other observations 

Both nurses who the WFA consultant interviewed during the visited mentioned accuracy as the biggest 

benefit of using the 2D barcode scanners. However, some challenges to using the scanners were noted. As 

already mentioned, some of the scanners at this practice were inoperable. For those scanners that did 

work, at least one nurse noted that sometimes the scanners work, and sometimes they don’t. One nurse 

specifically stated, “Sometimes you are able to scan and sometimes the scanner doesn’t work and by the 

time you get it to scan you could have entered the numbers in, you know, in less time.“ 

Another challenge noted by one of the nurses related to multi-dose vials. The nurse mentioned that the 

polio and influenza (for children 3 and over) are multi-component vials and they cannot scan both vials into 

the EMR. As a result they have to type at least one of the numbers into the EMR. She noted they are unable 

to keep an empty vial on the desk for reference because the lot number may have changed. The nurse 

suggested that there should be a way to copy the barcode, or have labels on the boxes that can be used to 

scan for the multi-vials. 

It should be noted that the time spent performing observations was truncated at this practice due to 

logistical issues in gaining access to practice processes. 

103