Grading of Recommendations, Assessment, Development, and Evaluation (GRADE) for Use of Smallpox 

Vaccine in Laboratory and Health-Care Personnel at Risk for Occupational Exposure to Orthopoxviruses: 

Recommendations of the Advisory Committee on Immunization Practices, 2015. 

Background

ACAM2000, a live vaccinia virus vaccine, is the only smallpox vaccine currently licensed and available in the 

United States (U.S.) for vaccination of persons at risk for orthopoxviral disease. ACAM2000 is produced in Vero 

cells and derived from a clonal isolate of Dryvax, the New York City Board of Health strain widely used during 

the smallpox eradication campaign [1-5]. Like Dryvax, ACAM2000 is administered percutaneously using a 

bifurcated needle, and comes with potential risks of serious adverse events [6-7]. Recommendations of the 

U.S. Advisory Committee on Immunization Practices (ACIP) regarding smallpox (vaccinia) vaccination, most 

recently revised in 2003, specify Dryvax as the designated smallpox vaccine in routine non-emergency 

vaccination programs, which primarily involve laboratory and health-care personnel, as well as select military 

personnel [8-9]. However, as the license for Dryvax was withdrawn in 2008, and remaining vaccine supplies 

were subsequently destroyed, the need to develop new ACIP recommendations based on ACAM2000 is 

paramount [10]. Thus, the ACIP Smallpox Vaccine Work Group has applied the GRADE framework to the 

available evidence evaluating the administration of ACAM2000 in laboratory and health-care personnel at risk 

for orthopoxviral disease due to occupational exposure. 

Policy Question

Should administration of ACAM2000 be recommended routinely for laboratory and health-care personnel at 

risk for occupational exposure to orthopoxviruses? 

PICO of Interest

The population, intervention, comparison, and outcomes (PICO) of interest was defined as follows:

Population (P): Laboratory and health-care personnel at risk for occupational exposure to orthopoxviruses

Intervention (I): Vaccination with ACAM2000

Comparison (C): Vaccination with Dryvax

Outcomes (O): ACIP workgroup members compiled an initial list of relevant outcomes to consider, which 

included both beneficial and harmful outcomes. A modified Delphi process was then utilized to rate the 

importance of each outcome. Members of the workgroup used these results to compile a final list of 

outcomes to be considered (Table 1).

https://www.cdc.gov/vaccines/acip/recs/grade/orthopoxvirus.pdf 4/26/2017
Smallpox WG GRADE Summary

2

Table 1. Results of survey and identified critical outcomes: ranked in decreasing order of importance

Outcome Mean Importance 

Rating 

(Range)

Standard Error 

of Importance 

Rating

Include in 

Evidence Table? 

1, 2

1. Death 8.3 (6-9) 0.29 Yes

2. Postvaccinial

encephalitis 7.5 (4-9) 0.35 Yes

3. Eczema vaccinatum 7.4 (4-9) 0.38 Yes

4. Myo/pericarditis 

resolved with sequelae 7.3 (4-9) 0.41 Yes

5. Progressive vaccinia 7.2 (4-9) 0.36 Yes

6. Cutaneous response 6.1 (2-9) 0.72 Yes

7. Generalized vaccinia 5.8 (1-9) 0.74 Yes

8. Inadvertent inoculation 5.6 (2-9) 0.71 Yes

9. Myo/pericarditis

resolve without sequelae 5.5 (1-9) 0.61 Yes

10. Neutralizing antibody 

response 5.4 (1-9) 0.72 Yes

11. Mild adverse events / 

injection site reactions 4.2 (1-9) 0.79 Yes

Table 1 Footnotes

1 Outcomes with importance ratings of 1-3 are generally considered not important and not included in the 

evidence tables; ratings of 4-6 are considered important but not critical for making a decision; ratings of 7-9 

are critical for making a decision and are included in the evidence tables. Evidence tables are generally limited 

to 7 outcomes, and therefore a combination of critical and/or critical but not important outcomes may be 

considered [11]. 2 The outcomes assessed by the workgroup included both critical and important outcomes. The workgroup felt 

that all of the rated outcomes should be assessed and considered when making their recommendation. 

Therefore, in order to keep the number of total outcomes assessed to 7, the workgroup decided to combine 

outcomes that are normally classified as serious adverse events (SAE) into a single category, which included 

the following SAE: death, postvaccinial encephalitis, eczema vaccinatum, progressive vaccinia, and generalized 

vaccinia. Additionally, mild adverse events (MAE) were combined to include the following: those adverse 

events which were not previously identified as a SAE, inadvertent inoculation, and myo/pericarditis resolved 

with or without sequelae. 
Smallpox WG GRADE Summary

3

Evidence Retrieval, Assessment and Synthesis: Included Studies

To identify published literature containing evidence relevant to our policy question, we conducted an 

electronic search of PubMed for studies written in English that were published through August 22, 2013. 

Search terms included “smallpox vaccine”, “ACAM2000” and “Dryvax”. There were 2424 records identified 

through PubMed database searching. Additional studies were searched for by scanning references of included 

studies, as well as, relevant reviews. Those randomized controlled trials (RCT) that provided a direct analysis of 

the intervention (ACAM2000) and comparison of interest (Dryvax) were selected. No observational studies 

were found which provided a direct analysis of the intervention and comparison. Two reviewers selected 

studies in two stages: review of titles and abstracts, followed by a review of full-text articles. Studies not 

directly relevant to the policy question were eliminated and included a variety of records and studies 

involving: animals, primary molecular investigation, vaccines not of direct interest, reviews, position papers, 

issue briefs, and meeting notes. Any discrepancies were resolved through discussion between the two 

reviewers. In all, five studies were included in the analysis [1-2, 12] (Table 2).

Table 2. Characteristics of Included Studies 

Author, Year (Study)1 Participants Intervention Reported outcomes-of interest

Frey et al., 2009

(Study H-400-002)

18-29 y/o 

naïve adults

ACAM1000 and 

ACAM2000

Cutaneous response, neutralizing 

antibody response, MAE, SAE

Artenstein et al., 2005

(Study H-400-005)

18-29 y/o 

naïve adults ACAM2000 Cutaneous response, neutralizing antibody response, MAE, SAE

Acambis, Inc., 2007

(Study H-400-003)

>28 y/o 

previously 

vaccinated 

adults

ACAM2000 Cutaneous response, neutralizing 

antibody response

Acambis, Inc., 2007

(Study H-400-009)

18-29 y/o 

naïve adults ACAM2000 Cutaneous response, neutralizing antibody response, MAE, SAE

Acambis, Inc., 2007

(Study H-400-012)

>31 y/o 

previously 

vaccinated 

adults

ACAM2000 Cutaneous response, neutralizing 

antibody response, MAE, SAE

Table 2 Footnote

1 The Centers for Disease Control and Prevention was listed as the sole funding source for all included studies.

Evidence Tables: Summary of Data

1) Outcomes of both benefits and harms were abstracted for each study. 

2) Benefit outcomes were assessed and reported in all 5 RCTs and included both cutaneous response 

(vaccination success) and neutralizing antibody response (based on 50% PRNT). 

3) Outcomes considered harms were assessed and reported in 4 out of 5 RCTs and included: SAE, MAE, 

myo/pericarditis resolved with sequelae, myo/pericarditis resolved without sequelae, and inadvertent 

inoculation. 
Smallpox WG GRADE Summary

4

Table 3. Smallpox vaccine: Benefits

Study Population / Treatment Group

Vaccinia-Naïve Subjects Previously Vaccinated Subjects

ACAM2000 Dryvax ACAM2000 Dryvax

Cutaneous Response (Vaccination Success)

No. of Evaluable 

Subjects

(# studies)

857

(3)

336

(3)

1238

(2) 

440

(2) 

Number of 

Vaccination 

Successes (%)

828

(96.6%)

334

(99.4%) 1041

(84.1%)

433

(98.4%)

Neutralizing Antibody Response

No. of Evaluable 

Subjects

(# studies) 

646

(3) 

269

(3)

784

(2) 

428

(2) 

Pooled Geometric 

Mean Titer (GMT)

Ratio1,2,

(95% CI of pooled 

GMT Ratio)

0.677

(0.625, .733) 

Table 3 Footnotes

1Pooled GMT Ratio was generated in Revman. Though GMT’s were reported for all studies included in the 

analysis, variance data for the evaluable population was only reported in one Phase 2 study (H-400-005); 

however it was unpublished, as it was not appropriate for the analysis the authors needed. Variance data for 

Phase 2 and Phase 3 studies was requested from the authors [13]. Data from Phase 3 studies (H-400-009, 

which looked at the naïve total study population, and H-400-012, which looked at the previously vaccinated 

total study population) was acquired. The neutralizing antibody titers on Day 30 reported for the total 

population (both naïve and previously vaccinated individuals) in the Phase 3 studies was reported as both 

GMT and Log10GMT (± standard deviation). The standard deviation of the reported GMT (for both naïve and 

previously vaccinated individuals) was calculated by taking the inverse log of the standard deviation reported 

for the Log10GMT. The resulting calculated standard deviation for the GMT (for both naive and previously 

vaccinated individuals), along with the sample size and mean (GMT), were used to calculate the 95% 

confidence interval (CI) and range. The calculated standard deviation from the evaluable populations within 

the Phase 3 studies was subsequently applied to the corresponding evaluable populations from Phase 1 and 2 

studies lacking variance data. 2Data for each study were converted to a log scale for computing pooled estimates. The pooled results were 

then converted back into their original metric by taking their anti-log. 
Smallpox WG GRADE Summary

5

Table 4. Smallpox vaccine: Harms

Study Population / Treatment Group

Vaccinia-Naïve Subjects Previously Vaccinated Subjects

ACAM2000

N1 = 954

n (%)

[# Studies]

Dryvax

N1 = 368

n (%)

[# Studies]

ACAM2000

N1 = 1371

n (%)

[# Studies]

Dryvax

N1 = 448

n (%)

[# Studies]

Experienced 

Serious Adverse 

Events2

0 (0%)

[3]

0 (0%)

[3]

0 (0%)

[1] 

1 (.22%)

[1] 

Myo/pericarditis 

Resolved with 

Sequelae3

1 (.10%)

[3]

1 (.27%)

[3]

0 (0%)

[1] 

0 (0%)

[1] 

Myo/pericarditis 

Resolved 

without 

Sequelae

6 (.63%)

[3]

2 (.54%)

[3]

0 (0%)

[1] 

0 (0%)

[1] 

Inadvertent 

Inoculation

0 (0%)

[3]

0 (0%)

[3]

0 (0%)

[1]

0 (0%)

[1]

Mild Adverse 

Events

945

(99.05%)

[3]

368

(100%)

[3]

1325

(96.64%)

[1]

443

(98.8%)

[1]

Table 4 Footnotes

1Number indicated represents the total number of subjects enrolled in studies where subjects were 

administered either ACAM2000 or Dryvax, subsequently monitored for the outcome(s) of interest, and results 

were reported. 2Though not included in the Acambis 2007 VRBPAC briefing document, one case of generalized vaccinia was 

reported in Study H-400-012 within the VRBPAC Background Document [1, 7]. This case was discovered in a 

previously vaccinated subject upon reporting to a scheduled study center visit on Day 10 post vaccination. The 

subject was admitted to a local hospital for observation, dermatological consult, treatment, and subsequently 

discharged from the hospital the following day. This event was determined to be study-vaccine related and 

resolved without sequelae on Day 13. 

3One case of myo/pericarditis was categorized by the Acambis report as remaining ongoing [1]. The individual 

was a subject within the Phase 3 study, and received Dryvax as a naïve vaccine. We have included that subject 

within the table as having myo/pericarditis resolved with sequelae.
Smallpox WG GRADE Summary

6

Evidence Tables: Type of Evidence

The evidence type for each outcome was determined based on the studies reviewed. Evidence type was 

initially ranked according to the following ACIP GRADE criteria [11]: 

1 = Randomized controlled trials (RCTs), or overwhelming evidence from observational studies.

2 = RCTs with important limitations, or exceptionally strong evidence from observational studies.

3 = Observational studies or RCTs with notable limitations

4 = Clinical experience and observations, observational studies with important limitations, or RCTs with several 

major limitations. 

The initial evidence type could subsequently be downgraded if any of the GRADE criteria was determined to 

be serious (-1) or very serious (-2). Evidence could also be upgraded if the strength of association was shown 

to be strong (+1 when the relative risk is approximately >2 or <0.5) or very strong (+2 when the relative risk is 

approximately >5 or <0.2) and there was no serious risk of bias [11]. 

Overall Evidence Type

Table 5. Type of Evidence: The effect of ACAM2000 on identified outcomes

Outcome Design

(# studies)

Risk of 

bias Inconsistency Indirectness Imprecision4 Other considerations1

Evidence 

Type

Benefits

Cutaneous 

Response

RCT 

(5)

No 

Serious

No Serious Serious2 No Serious None 2 

Neutralizing 

Antibody 

Response

RCT 

(5)

No 

Serious No Serious Serious2 No Serious None 2 

Harms

Serious Adverse 

Events

RCT 

(4)

No 

Serious No Serious No Serious Serious4 None 2 

Myo/pericarditis 

Resolved with 

Sequelae

RCT 

(4)

No 

Serious

No Serious No Serious No Serious None 1 

Myo/pericarditis 

Resolved without 

Sequelae

RCT 

(4)

No 

Serious No Serious Serious3 No Serious None 2 

Inadvertent 

Inoculation

RCT 

(4)

No 

Serious No Serious No Serious Serious4 None 2 

Mild Adverse 

Events

RCT 

(4)

No 

Serious No Serious No Serious No Serious None 1 

Overall evidence 

type across all 

critical

outcomes5

2 

Table 5 Footnotes

1Strength of association, dose-response, opposing plausible residual confounding or bias, publication bias. 

2Cutaneous response and neutralizing antibody response were surrogates for the outcome of primary interest: 

vaccine efficacy to prevent orthopoxviral disease

3The clinical significance of myo/pericarditis resolved without sequelae is unclear, therefore, myo/pericarditis 

resolved with sequelae was assessed to be the outcome of primary interest. 
Smallpox WG GRADE Summary

7

4 The total number of participants enrolled across all RCTs was <4000. Thus, these studies were not powered 

to detect serious adverse events (i.e. EV, PV, PVE, death) or inadvertent inoculation. Please See Table 1 in 

Supplementary Appendix 1 for information regarding sample size needed to detect twice the AE rate [14]. 5The lowest evidence quality from critical outcomes assessed

Judgments About the Recommendation Category

Factors to consider: 

Key Factors Comments

Evidence type for benefits and 

harms

• Overall evidence type 2 across all critical outcomes

• Evidence type 1 for myo/pericarditis resolved with sequelae and 

mild adverse events

• Evidence type 2 for cutaneous response, neutralizing antibody 

response, serious adverse events, myo/pericarditis resolved 

without sequelae and inadvertent inoculation 

• Cutaneous response and neutralizing antibody response were 

downgraded due to indirectness (assessed to be surrogates for 

outcome of primary interest) 

• Serious adverse events and inadvertent inoculation were 

downgraded for imprecision (studies were not powered to detect

these events) 

Balance between benefits and 

harms

• Vaccination provides benefits through protective antibody 

response, as well as cutaneous response (take rate) 

• Serious adverse events are uncommon when proper screening 

methods are applied

• Very few documented vaccine failures among vaccinated

laboratory personnel 

Value • Prevents a potentially serious disease with poor outcomes

• Inform decisions about smallpox vaccination in laboratory and 

health-care personnel at risk for occupational exposure to 

orthopoxviruses

Cost-effectiveness • Not evaluated

• Likely low risk of disease in persons receiving appropriate dose of 

smallpox vaccine (ACAM2000)

• Vaccine provided free of charge upon request 

Summary: The overall evidence type was determined to be type 2 across all critical outcomes. It was 

determined that the benefits of vaccinating with ACAM2000 are likely greater than potential harms. 

Additionally, there is high value placed on prevention of OPXV infections among laboratory and health-care

personnel at risk for occupational exposure to orthopoxviruses through routine recommendation of 

vaccination with ACAM2000. Though cost-effectiveness was not analyzed in detail, the impact from a social 

and personal standpoint cannot be overlooked given the potential for disfiguring scars from orthopoxvirus 
Smallpox WG GRADE Summary

8

infections. Furthermore, emotional and economic cost associated with treating an unvaccinated individual 

who is occupationally exposed to an orthopoxvirus could be high.

References

1. Acambis, Inc. (2007). ACAM2000 smallpox vaccine. Vaccines and Related Biological Products Advisory 

Committee (VRBPAC) Briefing Document. Apr 18 2007. 

2. Artenstein AW, Johnson C, Marbury TC, Morrison D, Blum PS, Kemp T, Nichols R, Balser JP, Currie M, & 

Monath TP. (2005) A novel, cell culture-derived smallpox vaccine in vaccinia naïve adults. Vaccine; 23(25): 

3301-3309.

3. Greenberg, R.N., & Kennedy, J.S. (2008). ACAM2000: a newly licensed cell-culture based live vaccinia 

smallpox vaccine. Expert Opin. Investig. Drugs; 17(4): 555-564. 

4. Handley, L., Buller, R.M., Frey, S.E., Bellone, C., & Parker, C. (2009). The new ACAM2000 vaccine and other 

therapies to control orthopoxvirus outbreaks and bioterror attacks. Expert Review Vaccines; 8(7): 841 – 

850. 

5. Monath, T.P., Caldwell, J.R., Mundt, W., Fusco, J., Johnson, C.S., Buller, M., Liu, J., Gardner, B.,Downing, G., 

Blum, P.S., Kemp, T., Nichols, R., & Weltzin, R. (2004). ACAM2000 clonal Vero cell culture vaccinia virus 

(New York City Board of Health strain): a second-generation smallpox vaccine for biological defense. Int. J. 

Infectious Diseases; Suppl 2: S31-S44.

6. Acambis, Inc. (2008). ACAM2000 smallpox (vaccinia) vaccine, live. Package insert. 

7. Rosenthal, S., Merchlinsky, M., Chowdhury, M. (2007). ACAM2000 (Live Vaccinia Virus Smallpox Vaccine). 

VRBPAC Background Document. May 17, 2007 VRBPAC Meeting. 

8. Centers for Disease Control and Prevention (CDC). (2001). Vaccinia (Smallpox) Vaccine: Recommendations 

of the Advisory Committee on Immunization Practices (ACIP), 2001. MMWR; 50(RR-10): 1-25. 

9. Centers for Disease Control and Prevention (CDC). (2003a). Recommendations for using smallpox vaccine 

in a pre-event vaccination program. Supplemental recommendations of the Advisory Committee on 

Immunization Practices (ACIP) and the Healthcare Infection Control Practices Advisory Committee 

(HICPAC). MMWR; 52 (RR-7): 1-16. 

10. Centers for Disease Control and Prevention (CDC). (2008). Notice to Readers: newly licensed smallpox 

vaccine to replace old smallpox vaccine. MMWR; 57(8): 207-208. 

11. Ahmed, F. (2013). U.S. Advisory Committee on Immunization Practices (ACIP) Handbook for Developing 

Evidence-based Recommendations. Version 1.2. Sep 10, 2013.
Smallpox WG GRADE Summary

9

12. Frey, S.E., Newman, F.K., Kennedy, J.S., Ennis, F., Abate, G., Hoft, D.F., & Monath, T.P. (2009) Comparison 

of the safety and immunogenicity of ACAM1000, ACAM2000, and Dryvax in healthy vaccinia-naïve adults. 

Vaccine; 27(10): 1637 - 44. 

13. Chowdhury, M.K. (2006). ACAM2000 STN 125158 Memorandum: Statistical Review and Evaluation. August 

16, 2006. Obtained by request from Dr. Michael Merchlinsky. 

14. Lane, J.M., Neff, J.M., Millar, J.D. (1970). Complications of smallpox vaccination, 1968: results of ten 

statewide surveys. Journal of Infectious Diseases; 122(4): 303-309. 
Smallpox WG GRADE Summary

10

Supplementary Appendix 1: Reported Rates of Serious Adverse Events

Table 1. Rates of Serious Adverse Events 

Rates of SAE in vaccinated 

population 

(# cases / million vaccinations)1

% Chance You Would NOT see 

SAE in ACAM2000 RCTs

Sample Size Needed to Detect Twice 

the AE Rate (Power 0.8)

Naïve

(n= 1207)

Previously 

Vaccinated

(n=1670)

Naïve Previously 

Vaccinated

Naïve Previously 

Vaccinated

Eczema 

vaccinatum 38.5 3 95.50% 99.50% 611,565 7,848,844

Progressive 

vaccinia 1.5 3 99.80% 99.50% 15,697,723 7,848,844

Postvaccinial 

encephalitis 12.3 2 98.50% 99.60% 1,914,325 11,773,284

Inadvertent 

Inoculation 529.2 42.1 52.80% 95.00% 44,459 559,267

Death 1.5 NA 99.80% NA 15,697,723 NA

1Rates of SAEs from Lane JM, Ruben FL, Neff JM, Millar JD. (1970). Complications of smallpox vaccination, 

1968: results of ten statewide surveys. Journal of Infectious Diseases. 122(4): 303-309.