Innovative Data Collection Methods in Smallpox Program Evaluation 

Evans, Brian M. 

Burke, Brian J. 

Levy, Paul 

Baxter, Rodney K. 

RTI-International 

3040 Cornwallis Rd. 

Research Triangle Park, NC 27709 

evans@rti.org

bjb@rti.org

levy@rti.org

rbaxter@rti.org

Overview 

Since the atrocities of September 11, 2001, and the anthrax exposures that occurred soon after, concern for the security of the

United States has increased with the probability of additional bioterrorist incidents producing potentially devastating 

consequences, including widespread death and disease. Although naturally occurring outbreaks of smallpox have been 

eradicated, the threat of smallpox as a biological weapon remains. 

In late 2002, as part of the national response to the threat of bioterrorist attacks related to the smallpox disease, Smallpox 

Response Teams were formed to provide critical services in the event of such an attack. The National Immunization Program 

(NIP) within the Centers for Disease Control and Prevention (CDC) asked for volunteers among health care workers (HCWs) 

and other critical personnel to receive the vaccine against smallpox and established a voluntary Smallpox Vaccine Program 

(SVP) for the health care workers. This CDC-NIP program was administered through 56 state and county health departments 

that were working with hospitals to identify and inoculate HCWs who constituted the Smallpox Response Teams. However, 

given the decentralized nature of the program, there was considerable variation in how it was administered across hospitals, 

health departments, and other state agencies. CDC-NIP developed the Evaluation of Non-Participants in the Smallpox 

Vaccination Program as an opportunity to evaluate the success of the SVP, to learn about factors related to how the 

vaccination program was administered, and to understand the reasons that HCWs chose not to receive the vaccination. RTI 

International was contracted by CDC-NIP to conduct this evaluation. Questionnaires were administered to both HCWs and 

Smallpox Vaccination Coordinators at each agency. This paper analyzes how data collected from two separate 

questionnaires aided in achieving the three goals of the study. 

Program Evaluation 

The respondent questionnaire was administered to HCWs who decided not to receive the vaccination, asking specifically 

about factors that had influenced their decision not to be vaccinated. However, these data provided only one point of view 

about the SVP, that of the HCW. To supplement the potentially subjective view of the SVP by the health care workers, RTI 

and CDC-NIP agreed on an innovative approach to obtain a potentially more objective perspective on the vaccination 

program and how it was administered. An additional questionnaire was developed for the Smallpox Vaccination Coordinators 

at each hospital and health department that participated in the program evaluation. Among other things, this questionnaire 

asked the coordinator to provide information on how the program was administered, the employer’s compensation policies, 

and barriers encountered during the SVP administration. 

Administering an additional questionnaire has allowed CDC-NIP to analyze data from these two different perspectives to 

gain a better understanding of how the SVP was administered and why some HCWs chose not to get vaccinated. The HCW 

questionnaire provides data about reasons for not receiving the vaccine, whereas the coordinator questionnaire data provide 

perspectives about SVP administration. These data from the coordinator questionnaire offer valuable insight into reasons for 

nonresponse in the SVP. Additionally, by obtaining data from the smallpox coordinators, aggregate-level analysis by hospital 

and health departments can be performed by linking these data to the interview information provided by the HCW to give an 

overall picture of how the SVP was administered and why so many HCWs decided not to be vaccinated. Patterns of 

consistencies or disconnects in opinions about the SVP between the smallpox coordinator and those employed at the hospital 

or health department about the promotion and administration of the SVP could also be identified. These data also provide 
insight into how a vaccination program of this magnitude could be carried out in the future, including how CDC might better 

prepare for future adult vaccination programs. 

Data Collection Results 

RTI administered the coordinator questionnaire to 113 smallpox coordinators, 110 of whom completed the questionnaire for 

a 97% response rate. In this paper we analyze questions about promotion of the SVP, compensation to HCWs, and potential 

barriers or reasons for not being vaccinated. Analysis of these questions in both the respondent and the coordinator 

questionnaires shows how the administration of the coordinator questionnaire increased our understanding of the reasons for 

nonparticipation in the Smallpox Vaccination Program. 

One of the purposes of evaluation was to identify reasons that HCWs declined to be vaccinated. Key pieces of information 

that might explain nonparticipation could be provided by the hospital and health department smallpox coordinators. RTI and 

CDC discussed this issue and agreed that it would be important to follow up and complete an interview with each smallpox 

coordinator to obtain specific information about his or her hospital or health department and how the Smallpox Vaccination 

Program was conducted at that particular location. Subsequently, RTI developed a questionnaire on topics such as the size of 

the agency, the process for starting the vaccination program, the number of staff vaccinated, advertising and promotion for 

the vaccination program, barriers encountered by the coordinator during the program, and the agency’s compensation policies 

for those who received the vaccination. This information proved to be extremely important in determining reasons for 

nonresponse in the Smallpox Vaccination Program. 

To better understand why many HCWs chose not to be vaccinated and how the program was actually conducted, CDC 

contracted with RTI to conduct an evaluation in five states (in the order in which they were recruited): Tennessee, Utah, 

Michigan, Nebraska, and California. This evaluation was designed to provide an opportunity to learn about factors related to 

the decisions not to receive the vaccination. Information from this effort will be used to enhance the efficacy of this program

and, possibly, subsequent similar programs directed toward adults and/or health care staff. As CDC’s prime contractor for 

implementing this evaluation, RTI was charged with several tasks: list acquisition, frame construction and sampling, data 

collection, data processing, questionnaire design, and reporting and analysis on the study’s findings. 

Discussion 

In the following tables, a random effects logistic model (specifically, xtlogit in Stata Release 9.0) was estimated using the 

respondent’s answer as the dependent variable and the coordinator’s response to the equivalent question as the independent 

variable and with hospitals/health departments as the random effect. A p-value less than 0.05 indicates significant 

concordance between respondents’ and coordinators’ answers adjusting for the clustering within hospitals/health 

departments. 

When agencies were asked to administer the Smallpox Vaccination Program, no instructions were provided on how to do so. 

We asked respondents whether they were personally invited by someone to receive the vaccination. If they were not 

personally invited, we provided a list of items from which they could choose all that applied to them and asked them how 

they were notified about the program (see Table 1). We gave coordinators a similar list and asked how the agency had 

notified employees about the program. 

Table 1. How staff were notified about the vaccination program 

% Yes % Agreement Significance 

Question N Coordinator Respondent Yes No Total p 

Asked/invited to be vaccinated 1,217 75.4 83.2 62.5 3.9 66.4 0.240 

Notified in a group setting 503 94.6 38.0 37.0 4.4 41.4 0.060 

Provided with sign up sheet 487 34.1 17.3 9.5 58.1 67.6 0.006 

General announcement, e-mail, or memo 489 91.6 60.3 56.7 4.7 61.4 0.040 

Coordinators and respondents clearly agreed that one way of notifying employees about the program was to invite them 

personally, with 75.4% of coordinators saying that they invited people, and 83.2% of respondents saying they were 

personally invited. The other options were much more disconnected: 94.6% of coordinators said that they notified employees 
of the program in a group setting, but only 38.0% of employees who were not personally invited said that they had been 

notified in a group setting. However, the agreement between respondents and coordinators was not significant for either of 

these first two announcement methods in the logit analysis. 

Sign-up sheets were used much less often than the other methods, with coordinators saying they were used 34.1% of the time, 

and respondents saying they were used only 17.3% of the time. However, there was 67.6% agreement that was significant 

between the two groups mainly because most of them said the sign-up sheets were not used (p = .006). For the final method, 

91.6% of coordinators said there was a general announcement to employees, but only 60.3% of employees said they were 

notified through a general announcement. The logit analysis was significant for this comparison as well (p = .040). 

These data are very important in understanding how the vaccination program was administered at agencies. If we had only 

the respondent data, we would have inferred that agencies did personally invite people but that they did not effectively use 

other means of announcement. With the coordinator data, we see that agencies did report announcing the program in many 

different ways, but they were not necessarily effective in reaching respondents. 

After learning how staff were notified about the program, we wanted to know how actively agencies actually promoted their 

vaccination programs: 73.4% of coordinators and respondents agreed that the agency either somewhat or very actively 

promoted the program (see Table 2). We found a high degree of agreement with these two items (p = 0.002), which indicates 

that agencies were actively trying to carry out the vaccination program. With the coordinator data affirming the activity of the 

agency in promoting the program, we can assume that the lack of participation in the vaccination program was not due to a 

lack of promotion at the agency level. There is one set of outliers in this table, where 161 respondents indicated that staff 

were “somewhat actively recruited, but coordinators said that recruiting was “Not at all active.” This could be attributed to 

the different perceptions of HCWs and coordinators. 

Table 2. How actively staff were recruited 

Respondent 

Somewhat 

Very actively actively Not actively Not at all Row total 

Coordinator N % N % N % N % N % 

Very actively 406 52.5 253 33.3 35 16.9 9 17.0 703 39.2 

Somewhat actively 242 31.3 285 37.5 69 33.3 18 34.0 614 34.2 

Not actively 61 7.9 61 8.0 13 6.3 3 5.7 138 7.7 

Not at all 64 8.3 161 21.2 90 43.5 23 43.4 338 18.9 

Column total 773 43.1 760 42.4 207 11.5 53 3.0 1,793 100.0 

p = 0.002 

In the coordinator questionnaire, we asked a question about what they considered the major barriers to the vaccination 

program. One of the hypotheses of CDC was that the safety of the vaccination was one of the reasons that so many health 

care workers were not getting vaccinated. We asked respondents how safe they thought the smallpox vaccine was, and 

coordinators how much of a barrier safety was to the vaccination program (see Table 3). In this case, if a respondent said the 

vaccine was very unsafe, and the coordinator said it was a major barrier, then there would be a positive correlation between 

the responses. 63.4% of coordinators said that safety was a major barrier to health care workers receiving the vaccine; 

however, many of the employees at those agencies felt the vaccine was fairly safe. (81.7% of respondents felt the vaccine was 

very safe or somewhat safe.) Expanding this a little, at agencies where the coordinator said safety was either a major barrier 

or a barrier, most of the employees said that the vaccine was safe. The disagreement between respondent and coordinators 

was significant in the logit analysis as well (p = 0.0000). We can make a couple of inferences about this disconnect between 

the responses. It could be that the coordinators were not as aware of employees’ concerns as they had thought. Coordinators 

may have answered the safety question from a personal perspective (i.e., they themselves might have thought the vaccine was 

not very safe, thus indicating it as a major barrier). The definition of “safety” could also have been interpreted differently by 

the two groups. The respondent data imply that safety may not have been as large a factor in the decision by employees not to 

receive the vaccination, whereas the coordinator data imply that they saw it as a major factor in respondents’ decisions to 

receive the vaccine. 
Table 3. How safe was the vaccination 

Respondent 

Very safe Somewhat safe Unsafe Very unsafe Row total 

Coordinator N % N % N % N % N %

A major barrier 180 54.1 682 63.0 186 73.5 51 78.5 1,099 63.4 

A barrier 94 28.2 264 24.4 48 19.0 

 

 

 

5 7.7 411 23.7 

A small barrier 53 15.9 127 11.7 18 7.1 9 13.9 207 11.9 

Not a barrier 6 0.9 10 0.6 1 0.4 0 0.0 17 1.0 

Column total 333 19.2 1,083 62.5 253 14.6 65 3.8 1,734 100.0 

p = 0.0000 

During the implementation of the evaluation, concerns were raised about how unions might potentially affect HCWs’ 

decisions to receive the vaccination. To understand what role unions actually played in this process, we asked respondents 

whether they were aware of union participation in the vaccination program, and we asked coordinators how much of a barrier 

they felt unions had posed to the vaccination program (see Table 4). Coordinators indicated that for the most part unions were 

not a barrier at all to the vaccination program, even if employees were aware of union participation. Respondent data show 

that 37.5% of respondents were aware of their agency’s union involvement, but the coordinator data show that 88.1% of 

coordinators did not consider it a barrier to participation in the program. Having the coordinator data report this information

is very important and shows that the unions did not have as much of an effect on health care workers’ decisions to participate 

as we may have thought they had. While the logit analysis is significant (p = 0.028), this is due mainly to the majority of the 

respondents’ reporting that they were not aware of the unions, whereas their coordinators indicated that unions were not a 

barrier (n = 758). 

Table 4. Awareness of union position 

Respondent 

Yes No Row total

Coordinator N % N % N %

A major barrier 

A barrier 

11 

31 

2.2 

6.3 

 9 

 13 

1.1 

1.6 

 20 

 44 

1.5 

3.3 

A small barrier 49 9.9 44 5.3 93 7.1 

Not a barrier 404 81.6 758 92.0 1,162 88.1 

Column total 495 37.5 824 62.5 1,319 100.0 

p = 0.028 

Shortly after the implementation of the vaccination program, the media reported three deaths of persons (CNN.com, 2003)

who had recently received the smallpox vaccine. There was nationwide media coverage of these deaths, prompting concern 

that the reports were negatively affecting health care workers’ decisions about whether to receive the vaccine. We asked 

respondents what kind of an effect the media had had on their decision to receive the vaccine, and we asked coordinators how 

much the media reports had been a barrier to the program. As indicated in Table 5, 42.2% of health care workers reported 

they had already made their decision about their participation before the media reports. In addition, another 45.5% indicated 

that the media reports had little or no effect on their decision to receive the vaccine. However, 41.8% of coordinators reported 

they thought the media reports were a major barrier to employees’ participating in the program, and 26.6% reported it was a 

barrier. This disagreement could again be attributed to the lack of communication between the coordinators and employees. 

Another possibility is that coordinators may have heard talk of the media reports among employees at the hospitals and health 

departments and thought that this had influenced their decisions. According to the respondents, the media reports were not a 

major factor in their decision to participate. However, the coordinator data indicate it may have been more of a factor than 

respondents reported. 
Table 5. News of cardiac effects publicized in media reports 

Respondent 

Made decision before 

No effect A little effect A large effect side effect known Row total 

Coordinator N % N % N % N % N %

A major barrier 165 44.2 117 41.5 106 59.9 213 35.1 601 41.8 

A barrier 98 26.3 90 31.9 37 20.9 158 26.0 383 26.6 

A small barrier 54 14.5 43 15.3 18 10.2 132 21.8 247 17.2 

Not a barrier 56 15.0 32 11.4 16 9.0 104 17.1 208 14.5 

Column total 373 25.9 282 19.6 177 12.3 607 42.2 1,439 100.0 

p = 0.107 

Compensation and liability were major concerns of health care workers during the Smallpox Vaccination Program, as 

indicated by the fact that the issue was taken to Congress and legislation was passed concerning these issues. We asked 

respondents whether they felt they would be adequately compensated if they suffered an adverse event from the vaccine (see 

Table 6). Coordinators were asked if their agency would compensate employees if they suffered an adverse event. Not 

surprisingly, 62.2% of the coordinators said they would offer compensation, but only 33.1% of overall respondents said they 

would be adequately compensated. Only 22.8% of respondents at the agencies where coordinators said they would offer 

compensation said that they would be adequately compensated. Overall, 50.2% of coordinators and respondents agreed in 

their responses (p = 0.013). This agreement is not surprising given all the attention that compensation received in the media 

and in government circles. 

Table 6. Compensation and liability 

% Yes % Agreement Logit 

Question N Coordinator Respondent Yes No Total p 

Staff compensation for adverse event 1,017 62.2 33.1 22.8 27.4 50.2 0.013 

Staff covered by liability insurance if 720 68.3 73.3 51.9 10.3 62.2 0.626

virus transmitted 

In addition, we asked respondents whether they felt they would be covered by liability insurance if the vaccinia virus were 

transmitted from them to a patient. Coordinators, in turn, were asked the same question; 68.3% of coordinators said their 

staff would be covered by liability insurance, and 73.3% of respondents also said they would be covered. However, there 

was only 51.9% agreement between the groups on the employer’s policy that they would be covered by liability insurance. 

Although the agreement level of the liability insurance question was higher than that of the compensation question, the logit 

analysis was not significant. 

Again we can draw a few inferences about these data. We could say that health care workers and coordinators knew more 

about their agency’s liability coverage than they did about its compensation policies. There is also a fairly low level of 

agreement in both, although it is higher in the liability question. This could indicate a lack of communication at one or 

multiple levels within the organization. The low level of agreement also shows that employees are not necessarily familiar 

with their agency’s policies. Both of these assumptions may have had an impact on health care workers’ decisions to 

participate in the vaccination program. 

Conclusion 

During the Evaluation of Non-Participants in the Smallpox Vaccination Program, we administered a questionnaire both to 

health care workers and agency smallpox coordinators. By conducting an interview with both populations, we collected 

comparable data with which to gauge how the Smallpox Vaccination Program was administered at different hospitals and 

health departments. Without data from both questionnaires, we would have only one point of view from which to draw 

conclusions about the vaccination program. Instead, data from both questionnaires can be utilized and compared to 

understand factors related to the effectiveness and administration of the program. For several items, health care workers and 

coordinators had some agreement in their responses. For example, data from the coordinator questionnaire indicated that 

agencies tried to promote the program in many different ways, something that we may not have been able to conclude with 
confidence from the respondent questionnaire only. By analyzing the level of agreement, we can make inferences about how 

the program was administered and try to answer the question of why so few health care workers received the smallpox 

vaccine. 

Administering a separate questionnaire was vital in helping us understand how the Smallpox Vaccination Program was 

administered and implemented. In addition, it provided us with another viewpoint, that of the smallpox coordinators. In the 

future, we hope to continue this research and compare other questions from the health care worker and coordinator 

questionnaires. Future analysis may help us better understand reasons that the program was less successful than anticipated, 

as well as factors that influenced the program’s success, which might be useful for other vaccination programs. Additional 

analysis may also lead to the understanding of why many HCWs did not receive the vaccine. The administration of two 

questionnaires will prove vital to this analysis and understanding of the program. In addition, future research on how to 

effectively and efficiently administer multiple questionnaires to subpopulations of the same sample will benefit the field, 

especially subpopulations that might have different views of the subject matter. Additional data can be obtained from them to 

help strengthen and support findings of the general sample. 

References 

http://www.cnn.com/2003/HEALTH/03/28/smallpox.vaccinations.ap