1 

ACIP Evidence to Recommendation User’s Guide

October 1, 2020
2 

Contents

Introduction .................................................................................................................................................. 3

How to Use This Guide.................................................................................................................................. 3

The EtR Framework................................................................................................................................... 4

Organization of this Guide ........................................................................................................................ 5

Overarching Framework Completion Guidance ....................................................................................... 5

Specific Criteria ............................................................................................................................................. 6

Domain 1: The Problem ............................................................................................................................ 6

Criterion: Public Health Priority ............................................................................................................ 6

Domain 2: Benefits & Harms of the Options ............................................................................................ 6

Criterion 1: Magnitude of desirable anticipated effects....................................................................... 6

Criterion 2: Magnitude of undesirable anticipated effects .................................................................. 7

Criterion 3: Balance of desirable versus undesirable anticipated effects ............................................ 8

Criterion 4: Certainty of evidence for outcomes.................................................................................. 8

Domain 3: Values and Preferences........................................................................................................... 9

Criterion 1: Target population perception of value.............................................................................. 9

Criterion 2: Uncertainty around target population perception of value.............................................. 9

Domain 4: Acceptability..........................................................................................................................10

Criterion: Acceptability to key stakeholders.......................................................................................10

Domain 5: Resource Use.........................................................................................................................10

Criterion: Resource allocation ............................................................................................................10

Domain 6: Equity.....................................................................................................................................11

Criterion: Health equity ......................................................................................................................11

Domain 7: Feasibility...............................................................................................................................12

Criterion: Implementation feasibility..................................................................................................12

Conclusions and Additional Considerations............................................................................................12

Summary of the ACIP’s deliberations and final recommendations........................................................13

Additional Resources ..................................................................................................................................13

Appendix 1: ACIP Evidence to Recommendations Framework ..................................................................14

Appendix 2 When to Prepare GRADE Evidence Profile Algorithm .............................................................21

Appendix 3 Implementation Considerations Checklist...............................................................................22
3 

Introduction

In 2010, the Advisory Committee on Immunization Practice (ACIP) formally adopted the Grading of 

Recommendations Assessment, Development and Evaluation (GRADE) approach for assessing the 

evidence and developing evidence-based recommendations. Since then, an assessment of the certainty 

of available evidence using GRADE has been an integral part of ACIP recommendation development. In 

addition to assessing the certainty of evidence, guideline panels consider other domains when making a 

recommendation (e.g., the balance of benefits and harms, patient’s values and preferences, etc.). This 

document presents the information that must be considered when making a recommendation. Previous 

documents/publications provide information on assessing the certainty of the evidence (Ahmed 2011). 

During the process of recommendation formulation, panels consider a range of factors in addition to the 

certainty in the evidence. Elucidation of these factors and the judgments behind them facilitates 

transparency, consistency, and communication of recommendations to healthcare providers, partner 

organizations, and the public (Alonso-Cuello 2017a & b). To structure the discussion of these additional 

factors and to make the judgments transparent, an evidence to decision-making (EtD) framework is 

used. 

The EtD structures decision-making by:

• Informing panel members’ judgments about the pros and cons of each intervention under 

consideration;

• Ensuring the important factors that determine a decision (criteria) are considered;

• Providing a concise summary of the best available evidence to inform judgements about each 

criterion;

• Helping structure discussion and identify reasons for disagreements when they occur; and

• Making the basis for decisions transparent to guideline users or those affected by a policy 

decision.

The ACIP has continued to follow and build upon the methodological advances in the GRADE approach 

and, as a result, has developed a modified Evidence to Recommendation (EtR) framework tailored to the 

needs of ACIP (Appendix 1). The purpose of EtR framework is to help panels making recommendations 

move from evidence to decisions, and to provide transparency around the impact of additional factors 

on deliberations when considering a recommendation. 

How to Use This Guide

The following supplementary information is intended to guide ACIP work groups while they complete 

the EtR framework, as they address specific policy questions and develop policy options for ACIP’s 

consideration. 

ACIP’s EtR frameworks are prepared by ACIP work groups for presentation to ACIP. As such, the EtR

frameworks represent the information reviewed and considered by the ACIP work group in developing 

policy options for ACIP’s consideration. The ACIP’s final recommendation and any additional 

considerations of ACIP during its deliberations are captured in a separate table titled ‘Final deliberation 

and decision by ACIP’ at the bottom of the EtR framework (Appendix 1). 
4 

The ACIP’s EtR frameworks and accompanying GRADE evidence profiles are posted on the ACIP website 

(https://www.cdc.gov/vaccines/acip/recs/index.html), and are referenced with links in the MMWR 

Policy Note or Recommendations and Reports documents, upon publication in MMWR. 

The EtR Framework

The EtR frameworks are designed to include key background information, criteria for making a decision, 

and conclusions. 

The background section of the EtR is intended to provide details of the policy question that is the topic 

of the framework. In the EtR framework table, this section is opened by a statement of the policy 

question and the PICO (Population, Intervention, Comparator, Outcome), followed by a brief summary 

of information needed to understand the question and why a recommendation is needed. Specific

items that should be addressed include whether the recommendation would be for “off-label” use (i.e., 

an indication not specified in the FDA approved label) or a preferential recommendation.

The second part of the framework is composed of rows delineating the factors or criteria considered 

during recommendation development, which are grouped into “domains.” These questions are 

intended to frame the discussions and guide literature searches and collection of relevant data.

The following columns are included for each criterion:

• Judgments that the work group members must make in relation to each criterion are summarized in 

the first column of the framework as checkboxes. In most cases, it is anticipated that the framework 

will be developed with judgments for each criterion suggested by the work group and subject 

matter experts (SMEs) who have prepared the framework. The judgments should be a result of the 

work group reaching a consensus; however, minority opinions expressed during the discussion 

should be captured in the “additional considerations” section (see below).

• Evidence to inform each of those judgments is presented in the second column of the table. 

Evidence refers to facts used to inform the work group’s judgments that are derived from studies 

that used systematic and explicit methods.1

 If published evidence is available, a paragraph or 

bulleted list summarizing the important considerations is sufficient, with mention of the most 

critical references or links to more detailed summaries of the evidence such as the GRADE evidence 

profile1

. If no peer-reviewed body of evidence is available, this should be simply stated and any 

additional information used to inform the judgment indicated. The intent is to be transparent about 

the information that was used to make the judgment, not to imply the need for the development of 

evidence when it is not available.

• Additional information that informs or justifies each judgment may be provided in the final, righthand column of the table. Additional information can include other data, assumptions, and logic 

used to make a judgment. Work groups may make different judgments for one or more subgroups 

(such as patients who are older or who have more severe disease) in relation to some or all of the 

criteria. When relevant, they may also report additional details, such as dissenting views of work 

group members or the results of voting on judgments where there was disagreement. Minority 

opinions voiced during discussions should be presented to increase transparency around the 

deliberation process. In addition, interpretations of the evidence may be presented here.

1 If using GRADEpro Guideline Development Tool software (https://gradepro.org/), a GRADE evidence profile can 

be automatically inserted into the EtR framework.
5 

Organization of this Guide

This document provides additional information for each domain and criterion, including the criterion 

question as presented in the EtR framework, related questions that expand on the issues to consider for 

each criterion (adapted from the GRADE Handbook), and completion guidance. A Discussion section is 

also included to provide further information and clarification on issues specific to each 

criterion/question. This additional information is intended to assist work groups and the ACIP in 

providing the relevant information to populate the framework.

Overarching Framework Completion Guidance

Despite availability of a framework, individuals and groups will need to continue to make judgments at 

several points, especially when data are minimal or absent. One purpose of the framework is to make 

those judgments explicit. Therefore, when evidence is not available, this should be stated and a 

summary of the discussion included. It is also anticipated that there will be differences of opinion within 

the work group, and the framework should be used to reflect these differences, particularly with regard 

to the final recommendation. Since one goal of the framework is to enhance transparency and 

communication of deliberations, significant differences of opinion or discussion points should be 

captured within each domain when relevant.
6 

Specific Criteria

Domain 1: The Problem

Criterion: Public Health Priority

Criterion question: Is the problem of public health importance?

Related questions:

• Are the consequences of the problem serious (i.e. severe or important in terms of the potential 

benefits or savings)?

• Is the problem urgent?

• Are a large number of people affected by the problem?

• Is the problem related to emerging diseases, antimicrobial resistance, or epidemic potential?

• Are there disadvantaged groups or populations disproportionately/differentially affected by this 

problem?

Completion Guidance: Provide a short summary of the background and significant elements of the 

problem, including a description of epidemiology, clinical features, and sequelae of the 

disease/condition in terms of public health consequences. This may be excerpted or summary 

information from presentations to the full ACIP. Specific items to consider may include:

• Frequency of the disease/condition (e.g., incidence, prevalence, secular trends)

• Severity of the disease/condition (e.g., mortality, morbidity)

• Social impact of the disease/condition (e.g., hospitalization rate, sickness absenteeism, effects 

on high-risk groups and vulnerable populations, clinical features, perception of importance, and 

the existence of other preventive measures)

Any additional considerations regarding the public health priority may also be included.

Domain 2: Benefits & Harms of the Options

Criterion 1: Magnitude of desirable anticipated effects

Criterion question: How substantial are the desirable anticipated effects?

Related question: 

• How substantial is the anticipated effect for each main outcome for which there is a desirable 

effect?

Completion Guidance: Across the critical outcomes (as specified in the PICO question and elaborated in 

the evidence profiles), evaluate the magnitude of the anticipated effect. The anticipated direct and 

indirect (e.g., herd immunity) benefits of vaccination or other interventions, should then be listed in the 

framework in terms of specified outcomes with a description of how substantial the effects are. For 

effectiveness or efficacy, specific items that may be considered are immunogenicity, strain coverage, 

capacity to reduce the disease incidence, capacity to disrupt carriage, duration of protection, and/or 

serotype replacements.

The range of effectiveness/efficacy estimates should be included here, but specific information 

regarding the confidence in point estimates should be presented in the certainty of evidence section 

(Criterion 4) below as part of the GRADE evidence profile. Describe any differences in 
7 

effectiveness/efficacy estimates for special populations (e.g., immunocompromised, specific age groups, 

etc.) and disadvantaged groups or settings (i.e. baseline risk). Work groups should also consider the 

uncertainties (if any) related to benefit with regard to durability of effectiveness, and effect on serious 

health outcomes (e.g., death and hospitalization), which are not often measured pre-licensure because 

such studies would require a sample size too large to be feasible. The numbers of illnesses, 

hospitalizations and deaths that can be averted by use of the vaccine may be presented here but is not 

required.

In addition, information may be included in the “additional considerations” section to highlight where 

evidence may be lacking (e.g., subgroups where additional data are needed to assess the benefits).

Discussion

Additional considerations here can include quality of life, acute and chronic pain, and functional status, 

among those who receive the vaccine and those who do not, including those who suffer the condition 

that could be prevented or ameliorated by vaccination. Work groups are encouraged to perform a 

number needed to vaccinate (NNV) analysis. If a number needed to vaccinate (NNV) analysis is 

performed, any important uncertainty or assumptions should be included. There are no defined 

thresholds for an “acceptable” NNV. 

Guidance surrounding the use of immunogenicity data as a surrogate for efficacy or effectiveness can be 

found in the GRADE Handbook (Schünemann 2013). It is important to use the same language around 

judgments (e.g., “minimal,” “small,” “moderate,” “large”) as included in the framework. The framework 

requires that the work group check one box and rationale should be provided for the judgment selected 

by the work group. However, any differences in opinions on the judgement selected should be reflected 

in the additional Information section to ensure transparency.

Criterion 2: Magnitude of undesirable anticipated effects

Criterion question: How substantial are the undesirable anticipated effects?

Related question:

• How substantial is the anticipated effect for each main outcome for which there is an 

undesirable effect (taking into account the severity or importance of the adverse effects and the 

number of people affected)?

Completion Guidance: For each of the critical outcomes (as specified in the PICO question and 

elaborated in the evidence profiles), evaluate the magnitude of the undesirable anticipated effect. List 

the anticipated harms of vaccination, in terms of chosen outcomes and describe how substantial those 

effects are. Specific items to consider may include reactogenicity, adverse events, and interactions with 

other vaccines. Describe any differences in harms for disadvantaged groups or settings. As above, 

information may be included in the Additional Information section to highlight where evidence may be 

lacking.

Discussion

When answering this question, work groups should consider the size and strength of the pre-licensure 

safety database to sufficiently examine safety outcomes of interest. Groups can also discuss safety 

experience with vaccines from the same product class, and any uncertainties related to the safety of the 
8 

product based on limitations of study design, limitations of size of safety database for rare outcomes, 

and duration of follow up. Work groups should consider “risk of harm” (e.g., unknown long-term effects 

of new adjuvants, having only clinical trial data vs. long-term real-world experience).

Just as for the benefits section, it is important to use the same language around judgments (e.g., 

“minimal,” “small,” “moderate,” “large”) as included in the framework. Only one box should be 

checked, but differences in opinions should be reflected in the Additional Information section to ensure 

transparency.

Similar to the previous criterion, work groups should consider the existing safety information for both 

the product under consideration, as well as similar products (e.g., vaccines in the same class). Any 

uncertainties related to safety should be clearly articulated and discussed.

Criterion 3: Balance of desirable versus undesirable anticipated effects

Criterion question: Do the desirable effects outweigh the undesirable effects?

Related question:

• What is the balance between the desirable effects relative to the undesirable effects?

Completion Guidance: The work group should describe the deliberations on this topic, ensuring that any 

minority opinions are included. If specific subgroups have a different benefit to harm balance than the 

general population, describe those considerations here (especially if those groups are not included in 

the recommendation on the basis of this altered balance).

Discussion

When the comparator is “no vaccination,” this section is focused on comparing the balance of risks and 

benefits of the same vaccine. Comparison of the adverse events profile of this vaccine to other 

vaccine(s) available to prevent the same disease can be addressed in the implementation section (e.g., 

communication with providers and patients to ensure recipients are counseled about severity of adverse 

events) or in the ‘Additional Considerations’ section. Since it may be difficult to compare benefits and 

risks, as measures are inherently different, judgment must be exercised.

Criterion 4: Certainty of evidence for outcomes

Criterion question: What is the overall certainty of this evidence for the critical outcomes?

Related question:

• What is the overall certainty of this evidence of effects, across all of the outcomes that are 

critical to making a decision?

Completion Guidance: For most situations, the overall certainty of the evidence is the lowest certainty of 

the critical outcomes. See the ACIP GRADE Handbook regarding the process of making detailed 

judgments about the quality of evidence or certainty in estimates of effects and preparation of evidence 

profiles. The GRADE evidence profiles and associated documents should be referenced here. If there is 

uncertainty about whether to prepare evidence profiles, please follow the “When to GRADE” algorithm 

(Appendix 2) and consult the ACIP Secretariat. If evidence profiles are not prepared, a rationale and 

summary of any alternative methods should be provided.
9 

Discussion

It is expected that the overall certainty of the evidence will be informed by the GRADE evidence profiles

used to inform Criteria 2 and 3 (benefits and harms). If evidence profiles are not prepared, a rationale 

and summary of any alternative methods should be provided.

Domain 3: Values and Preferences

Criterion 1: Target population perception of value

Criterion question: Does the target population feel that the desirable effects are large relative to 

undesirable effects?

Related questions: 

• How does the target population (i.e., those affected or potentially affected by the disease) view 

the balance of desirable effects versus undesirable effects?

• Would patients and caregivers feel that the benefits outweigh the harms and burden?

• Does the population appreciate and value the vaccination?

Completion Guidance: Provide information regarding the perspectives and perceptions of potential 

recipients about the disease and the vaccine. Describe the source of target population values and 

preferences for critical outcomes (e.g., targeted research, questionnaires). Discussion and evidence 

regarding the possible value to the population due to herd immunity may also be included, where 

applicable.

Discussion

Work groups may want to consider the following topics:

• How much less people value outcomes that occur in the future versus outcomes that occur now

• Potential recipient attitudes to undesirable effects (i.e. how risk averse they are)

• Risk perception – whether some segment of the population is inclined to view itself as not at risk 

for the disease

If the evidence is limited, work group deliberations can be used. If evaluation of values is desirable and 

there is sufficient time to conduct research, there are existing survey mechanisms that may be used 

when funding and space are available. Work groups interested in this should consult with the ACIP 

Secretariat.

Criterion 2: Uncertainty around target population perception of value

Criterion question: Is there important uncertainty about, or variability in, how much people value the 

main outcomes?

Related questions:

• How much do individuals value each of the outcomes in relation to the other outcomes (i.e., 

what is the relative importance of the outcomes)?

• Is there evidence to support those value judgements, or is there evidence that the variability in 

those values is large enough to lead to different decisions?

Completion Guidance: It is not anticipated that there will be a need to routinely gather evidence if none 

is currently available; rather, any paucity of information should be indicated here. However, if the 
10

variability is estimated to be so significant that this factor alone could lead to a different decision, this 

should be indicated and consideration given to obtaining additional evidence. The more likely it is that 

differences in values would lead to different decisions, the less likely it is that there will be a consensus 

that an option is a priority from both an individual and public health perspective and the more 

important it is to obtain evidence of the values of those potentially affected by the disease.

Domain 4: Acceptability

Criterion: Acceptability to key stakeholders

Criterion question: Is the option acceptable to key stakeholders?

Related question: 

• Are there key stakeholders that would not accept the distribution of benefits, harms, and costs?

• Are there key stakeholders that would not accept the costs or undesirable effects in the short 

term for the desirable effects (benefits) in the future?

Completion Guidance: This will often represent the opinion of the work group. Input from ACIP 

members may also be obtained through commentary and discussion at meetings and incorporated into 

the final framework. Stakeholders may include professional societies, liaison organizations, providers, 

pharmaceutical companies, advocacy groups, and the general public. Acceptability may vary across 

stakeholder groups and any such variability, and the rationale for it, should be captured here. Similar to 

other domains, the work group consensus opinion on whether it would be acceptable to the majority of 

stakeholders will be the basis for the final judgment; there is not a specified “threshold” to determine 

the answer.

Discussion

Work groups should carefully consider and define which specific stakeholders (e.g., providers in 

healthcare settings, providers in community settings, providers in public health settings, healthcare 

delivery systems, and the public) are considered in their discussions. Critical stakeholders for each 

circumstance may differ and therefore there is no pre-specified list of stakeholders that should always 

be considered, rather work groups should determine those groups that are relevant for their given 

situation. Liaison members on work groups can often provide perspective for their organizations that 

may be useful in deliberations.

Domain 5: Resource Use

Criterion: Resource allocation 

Criterion question: Is the option a reasonable and efficient allocation of resources?

Related questions: 

• What is the cost-effectiveness of the vaccination?

• How does the cost-effectiveness of the vaccination vary in any sensitivity analyses?

• How does the cost-effectiveness change in response to changes in context, assumptions, model 

structure, across different studies, etc.?

Completion Guidance: The objective of this section is to describe the available evidence from costeffectiveness analyses (CEAs) as well as any major factors that could affect the cost-effectiveness profile 
11

of the vaccine. Identify the cost-effectiveness results, including the base-case and a sensitivity range, 

from any available cost-effectiveness studies. When possible, indicate which assumptions cause the 

greatest change in results by examining the sensitivity analyses of a study. If there are two or more 

studies, identify any major differences between the studies. Identify any other important factors that 

may affect the cost-effectiveness profile of the vaccination. For example, issues identified in other 

domains of the EtR framework regarding the overall certainty of the evidence for critical outcomes may 

also be important factors that affect vaccination cost-effectiveness.

Discussion

Work groups should assemble economic evidence on the study question and the intervention strategies 

being evaluated. It may be helpful for the health economic evidence to be assembled in consultation 

with the ACIP Economics Lead and/or the NCIRD Lead Economist. The evidence collected can include 

analyses conducted by independent researchers, the vaccine industry and CDC economists. The work 

group lead and the ACIP Secretariat will determine if an internal CDC CEA should be conducted. This will 

generally be needed for new vaccines and new recommendations with major programmatic economic 

impact.

The findings from the health economic analyses should be reviewed and presented in accordance with 

ACIP guidance on economic studies 

(https://www.cdc.gov/vaccines/acip/committee/guidance/economic-studies.html). The work group 

should consider the quality of the CEAs as described by the Second Panel on Cost-Effectiveness in Health 

and Medicine. 4

 This may include review of the model design and inputs to determine if the estimates 

are reasonable in terms of disease transmission, intervention effects, vaccine costs, implementation 

costs, disease outcome costs, and community vs. patient preferences. It may be important to 

understand and articulate whether there is variability in local epidemiology, infrastructure, or costs that 

influence impact of the intervention.

Domain 6: Equity

Criterion: Health equity

Criterion question: What would be the impact on health equity?

Related questions:

• Are there any groups or settings that might be disadvantaged in relation to the problem or options 

that are considered?

• Are there plausible reasons for anticipating differences in the relative effectiveness of the option for 

disadvantaged groups or settings?

• Are there different baseline conditions across groups or settings that affect the absolute 

effectiveness of the option or the importance of the problem for disadvantaged groups or settings?

• Are there important considerations that should be made when implementing the intervention 

(option) in order to ensure that inequities are reduced, if possible, and that they are not increased?

Completion Guidance: Clinical and public health guidelines need to explicitly consider health equity

(Welch et al, 2017). Health inequities are differences in health considered unfair or unjust and could 

have been avoided. This domain facilitates transparent and explicit consideration of the impact of the 

intervention when compared with the alternative option on the target population, specifically to identify 

if any persons would be disadvantaged as a result of the intervention. The identification of factors that 
12

may lead to health inequities may lead to modified recommendations that apply to the target 

population or separate recommendations tailored to disadvantaged groups. Work groups may conduct a 

high-level search to identify any publications reporting on issues of health equity or inequity on the topic 

under consideration. Relevant publications may be qualitative or quantitative. If no evidence is 

identified, then the work group should provide additional considerations addressing the related 

questions to the best of their ability. 

Domain 7: Feasibility

Criterion: Implementation feasibility

Criterion question: Is the option feasible to implement?

Related questions: 

• Is the intervention sustainable?

• Are there important barriers that are likely to limit the feasibility of implementing the intervention or 

require consideration when implementing it?

• Is access to the vaccine an important concern?

• Would the vaccine recommendation have any impact on health equity?

• Are there important considerations when implementing the intervention in order to ensure that 

inequities are reduced, if possible, and that they are not increased?

Completion Guidance: The Implementation Checklist tool (Appendix 3) summarizes potential barriers to 

implementation and can be used to guide these discussions. Impact on health equity is a component 

that factors into several areas of the tool, but any outstanding concerns can be explicitly described. 

Information regarding barriers that would be difficult to overcome should be provided. Implementation 

issues are not expected to drive the recommendation, but it is possible that implementation 

considerations may change the type of recommendation, influence the wording of the recommendation 

or only inform the guidance that accompanies the recommendation. If there are specific factors that 

influence the ultimate decision regarding the recommendation or its wording, the rationale would be 

important to include here. As with other areas of the framework, if data are lacking this should be 

stated. Factors that may impact guidance accompanying the recommendation may be briefly listed 

here, with additional information included in the “Additional Considerations” section of the framework.

Conclusions and Additional Considerations

Finally, the conclusions that the work group reaches and any additional considerations that the work 

group would like to present regarding the policy question or recommendation are presented in the last 

four rows of the EtR framework (Appendix 1).

The type of recommendation and actual text of the recommendation are based on the judgments made 

for all of the criteria and relate directly to the summary “Balance of Consequences” judgement 

presented in the first row of the conclusions section. The specific wording of each of the three 

recommendation types will stand alone. The three recommendation options include: ACIP recommends 

the intervention; ACIP recommends the intervention for individuals based on shared clinical decisionmaking; and, ACIP does not recommend the intervention (Even though the Intervention may be used 

within FDA licensed indications). 
13

The final row of the EtR framework table provides space to emphasize any additional considerations that 

are pertinent to the recommendation, including suggestions for overcoming implementation barriers, 

proposed monitoring and evaluation needs, and/or areas requiring additional research to inform future 

decisions.

Summary of the ACIP’s deliberations and final recommendations

A summary of the ACIP discussions regarding the work group recommendations and the final wording 

approved should be included in the table ‘Final deliberation and decision by the ACIP.’ This should 

include a brief description of the rationale supporting any ACIP modification of, or disagreement with, 

the work group recommendation. This section will be completed by the ACIP work group lead and the 

ACIP work group chair collaboratively.

Additional Resources

1. Alonso-Coello P, Schünemann HJ, Moberg J, Brignardello-Petersen R, Akl EA, Davoli M, Treweek

S, Mustafa RA, Rada G, Rosenbaum S, Morelli A. GRADE Evidence to Decision (EtD) frameworks: 

a systematic and transparent approach to making well informed healthcare choices. 1: 

Introduction. bmj. 2016 Jun 28;353:i2016.

2. Alonso-Coello P, Oxman AD, Moberg J, Brignardello-Petersen R, Akl EA, Davoli M, Treweek S, 

Mustafa RA, Vandvik PO, Meerpohl J, Guyatt GH. GRADE Evidence to Decision (EtD) frameworks: 

a systematic and transparent approach to making well informed healthcare choices. 2: Clinical 

practice guidelines. bmj. 2016 Jun 30;353:i2089.

3. Ahmed, F., Temte, J.L., Campos-Outcalt, D., Schünemann, H.J. and ACIP Evidence Based 

Recommendations Work Group (EBRWG, 2011. Methods for developing evidence-based 

recommendations by the Advisory Committee on Immunization Practices (ACIP) of the US 

Centers for Disease Control and Prevention (CDC). Vaccine, 29(49), pp.9171-9176.

4. Schünemann H, Brożek J, Guyatt G, Oxman A. GRADE handbook for grading quality of evidence 

and strength of recommendations. Updated October 2013. The GRADE Working Group, 2013. 

Available from guidelinedevelopment.org/handbook. 2013.

5. Welch, V. A., Akl, E. A., Guyatt, G., Pottie, K., Eslava-Schmalbach, J., Ansari, M. T., ... & 

Hultcrantz, M. (2017). GRADE equity guidelines 1: considering health equity in GRADE guideline 

development: introduction and rationale. Journal of clinical epidemiology, 90, 59-67.
14

Appendix 1: ACIP Evidence to Recommendations Framework

Question: Overarching policy question to be answered by the guideline panel (ACIP) using the Evidence to Recommendations (EtR) framework. 

The question should be precise and identify the specific intervention, comparison, and outcome, as well as the target population and the setting 

(specific subpopulations) in PICO format.

Population: Target population for vaccine (e.g., age range, sex, immune status, pregnancy) 

Intervention: Vaccination (if applicable, dosage and schedule) 

Comparison(s): No Vaccination/Standard of care/An existing vaccine/Other prevention option 

Outcome: Outcome(s) associated with vaccination (e.g., prevention outcomes or adverse effects)

Background: The addressed PICO question should be described in detail, and important background information for understanding the question 

and why a recommendation or decision is needed should be briefly provided. If a recommendation is preferential or represents off-label use, 

this should be indicated. 

Include sample language: Additional background information supporting the ACIP recommendations on the use of xxx vaccine can be found in 

the relevant publication of the recommendation referenced on the ACIP website. 

WORK GROUP JUDGMENTS EVIDENCE ADDITIONAL INFORMATION

PROBLEM

Is the problem of public health importance?

○ 

○ 

○ 

○ 

○ 

○ 

No

Probably no

Probably yes

Yes

Varies

Don't know

Provide available scientific evidence on 

burden of disease, preferably within the

target population for the

recommendation.

If no published evidence is available,

provide expert judgment on the public 

health priority considerations.

Identify any additional public health

priority considerations, including 

consideration of disparities.
15

WORK GROUP JUDGMENTS EVIDENCE ADDITIONAL INFORMATION

BENEFITS & HARMS

How substantial are the desirable 

anticipated effects?

○

○ 

○ 

○ 

○ 

○ 

○

○

○

○

○

○

○

○ 

○ 

○ 

○ 

○ 

Minimal

Small

Moderate

Large

Varies

Don't know

Describe the magnitude of the beneficial

effects of vaccination on individual 

(vaccine effectiveness, duration of 

protection) and population (herd 

immunity) levels. 

Take into consideration:

Is the baseline benefit similar across 

subgroups (by age, gender, pregnancy or 

lactation status, occupation [i.e., healthcare 

workers], immune status, race, SES, and 

other groups)?

Are there indirect effects that should be 

considered (e.g., herd immunity)?

How substantial are the undesirable 

anticipated effects?

Minimal

Small

Moderate

Large

Varies

Don't know

Are there undesirable effects of the 

vaccine, either on the individual (e.g., 

adverse events following immunization) 

or population (e.g., age-shift of disease, 

serotype replacement) levels? 

Take into consideration: 

Is the baseline risk for harm similar across 

subgroups (see above)? 

Should there be separate recommendations 

for subgroups based on harms? 

Do the desirable effects outweigh the 

undesirable effects?

Favors interventionl

Favors comparison

Favors both

Favors neither

Varies

Don't know

Describe the balance of benefits of the 

vaccine with possible harms (individual and 

population level). 
16

○

○

○

○

○

○

○

○

○

○

○

○

○

○

○

○

WORK GROUP JUDGMENTS EVIDENCE ADDITIONAL INFORMATION

What is the overall certainty of this 

evidence for the critical outcomes? 

Effectiveness of the intervention

No studies found

4 (very low)

 3 (low)

2 (moderate)

1 (high)

Safety of the intervention

No studies found

4 (very low)

 3 (low)

2 (moderate)

1 (high)

Please refer to GRADE evidence profiles for 

detailed assessment of the certainty of the 

evidence. For more information, please see 

the ACIP Handbook for Developing EvidenceBased Recommendations.

If GRADE was not used to evaluate the 

certainty of evidence, please provide 

justification and the method and outcome of 

any other tools used to evaluate the body of 

evidence relevant to the critical outcomes.

VALUES

Does the target population feel that the 

desirable effects are large relative to 

undesirable effects?

No

Probably no

Probably yes

Yes

Varies

Don't know

Provide any available evidence on target 

population values & preferences related to 

vaccination and comparative health benefits 

and risks. Describe the source of these 

estimates.

Are values and preferences for relevant 

outcomes measured? Are the benefits, 

harms and costs of vaccination valued 

differently by different subgroups? 

 

If the target group doesn’t value the 

intervention, or attributes little value to the 

harms and benefits, consider whether 

potential education measures are needed. 
17

WORK GROUP JUDGMENTS EVIDENCE ADDITIONAL INFORMATION

Is there important uncertainty about or 

variability in how much people value the 

main outcomes?

○ 

○ 

○ 

○ 

○ 

○ 

○ 

○ 

○ 

○ 

○ 

○

○

○

○

○

○

Important uncertainty or variability

Probably important uncertainty or variability

Probabl not important uncertainty or variability

No important uncertainty or variability

No known undesireable outcomes

Please provide available data used to 

determine the relative importance that the 

target population attributes to the desirable 

and the undesirable outcomes related to the 

intervention as well as the comparison.

Describe the source of variability, if any.

Are there methods for determining values 

satisfactory for this recommendation? 

If not, systematic assessment of values and 

preferences of target group may be 

considered. 

ACCEPTABILITY

Is the intervention acceptable to key 

stakeholders?

No

Probably no

Probably yes

Yes

Varies

Don't know

Provide assessment of whether intervention 

would be acceptable to stakeholders 

(ethically, programmatically, financially, etc.) 

RESOURCE USE

Is the intervention a reasonable and 

efficient allocation of resources?

No

Probably no

Probably yes

Yes

Varies

Don't know

Provide summary of cost-effectiveness 

analyses (CEAs) of the vaccine in the target 

population. Include base case results and a 

sensitivity range. Include any other notable 

findings, for example, specific policyrelevant scenarios. 

Overall findings: Summarize the findings 

from available CEAs, including major 

differences in baseline assumptions.

Uncertainty: Does the analysis capture the 

full range of uncertainty? For example, are 

the findings from the uncertainty of 

evidence analysis, identified earlier in this 

document (the EtR Framework), 

appropriately represented in the methods of 

the CEAs?

Multiple assessments: Are there multiple 

CEAs? If so, what are the major differences 

in methods and results?
18

WORK GROUP JUDGMENTS EVIDENCE ADDITIONAL INFORMATION

EQUITY

What would be the impact on health 

equity?

○

○

○

○

○

○

○

○ 

○ 

○ 

○ 

○ 

○ 

Reduced

Probably reduced

Probably no impact

Probably increased

Increased

Varies

Don't know

Summarize the findings from a review of the 

literature addressing issues of health 

inequities or groups who may be 

disadvantaged.

Consider from the evidence or guideline 

panel:

• Are there any groups or settings that 

might be disadvantaged in relation to 

the problem or options that are 

considered?

• Are there plausible reasons for 

anticipating differences in the relative 

effectiveness of the option for 

disadvantaged groups or settings?

• Are there different baseline conditions 

across groups or settings that affect the 

absolute effectiveness of the option or 

the importance of the problem for 

disadvantaged groups or settings?

• Are there important considerations 

that should be made when 

implementing the intervention 

(option) in order to ensure that 

inequities are reduced, if possible, 

and that they are not increased?

FEASIBILITY

Is the intervention feasible to implement?

No

Probably no

Probably yes

Yes

Varies

Don't know

Are there any barriers to implementation? Please refer to the Implementation 

Considerations checklist.
19

Balance of 

consequences

Undesirable 

consequences 

clearly outweigh 

desirable 

consequences

in most settings

Undesirable 

consequences 

probably outweigh 

desirable 

consequences

in most settings

The balance 

between 

desirable and 

undesirable 

consequences 

is closely balanced 

or uncertain

Desirable 

consequences 

probably outweigh 

undesirable 

consequences

in most settings

Desirable 

consequences 

clearly outweigh 

undesirable 

consequences

in most settings

There is 

insufficient 

evidence to 

determine the 

balance of 

consequences

○ ○ ○ ○ ○ ○

Is there sufficient information to move forward with a recommendation?

Yes ○ No ○ 

○ ○ ○

Policy options for 

ACIP consideration

ACIP does not recommend the 

intervention* 

*Intervention may be used within 

FDA licensed indications

ACIP recommends the intervention 

for individuals based on shared 

clinical decision-making

ACIP recommends the intervention

Draft 

recommendation 

(text)

Please provide the draft recommendations proposed to ACIP. 

Additional 

considerations

(optional)

Please outline any significant additional considerations (e.g., aspects related to implementation, monitoring and 

evaluation, research priorities, etc.). 

Final deliberation and decision by the ACIP
20

Final ACIP 

recommendation

○ ○

ACIP does not recommend the 

intervention* 

*Intervention may be used within FDA 

licensed indications

ACIP recommends the intervention for 

individuals based on shared clinical 

decision-making 

○

ACIP recommends the intervention

Additional ACIP 

considerations

Wording as accepted in the guide
21

Appendix 2 When to Prepare GRADE Evidence Profile Algorithm
22

Appendix 3 Implementation Considerations Checklist