1 

Guidance for Health Economics Studies Presented to the Advisory Committee on 

Immunization Practices (ACIP), 2019 Update 

Date: 10/11/2019 

Project lead Project leadandcontact person contact person contact person: 

Andrew J. Leidnera

CDC contributors: CDC contributors: 

Harrell W. Chessonb

Martin I. Meltzerc 

Mark L. Messonnierd 

Contributors Contributors ontributorsfrom outside CDC from outside CDC from outside CDC: 

Grace M. Leee 

Lisa A. Prosserf

a. Health Economist and ACIP Economics Lead 

Immunization Services Division 

National Center for Immunization and Respiratory Diseases 

Centers for Disease Control and Prevention 

b.National Center for HIV/AIDS, Viral Hepatitis, STD, and TB Prevention, CDC 

c.National Center for Emerging and Zoonotic Infectious Diseases, CDC 

d.Center for Surveillance, Epidemiology and Laboratory Services, CDC 

e.Stanford University, School of Medicine and the Advisory Committee on Immunization Practices 

f.University of Michigan 

Acknowledgements/Disclaimer: The findings and conclusions in this report are those of the authors and do 

not necessarily represent the views of the Centers for Disease Control and Prevention. 
2 

Guidance for Health Economics Studies Presented to the Advisory Committee on 

Immunization Practices (ACIP), 2019 Update 

Acronyms and terminology 

For additional reference, the following acronyms and terminology appear in the guidance. 

CDC Centers for Disease Control and Prevention

ACIP Advisory Committee on Immunization Practices

ACIP Work Group(s) Advisory Committee on Immunization Practices Work Group(s)

Work Group Chair Member of ACIP who is the chair of a Work Group

CDC Work Group Lead CDC staff person who is the primary liaison for an ACIP Work Group

ACIP Economics Lead Designated economist at CDC who is primary liaison to ACIP

Rationale for this guidance 

The Advisory Committee on Immunization Practices (ACIP) develops recommendations on the use of vaccines 

for the US civilian population. The ACIP charter specifically mentions economic evidence as one of the

considerations for ACIP deliberations on the use of vaccines [1]. In addition, the importance of health economic 

data is highlighted in the “Evidence to Recommendations Framework” recently adopted by the ACIP [2]. The 

framework states that consideration should be given to whether or not a vaccination intervention is a 

“reasonable and efficient allocation of resources” and highlights the importance of considering any available 

cost-effectiveness research. 

This document (the Guidance) is intended to help ensure a high level of quality for economic evidence 

presentations and to ensure that these presentations are understandable and transparent to members of the 

ACIP Work Groups, the ACIP, and any individuals in attendance at an ACIP general meeting. This version of the 

Guidance is an update to the original Guidance that was published in 2007 [3]. This update is intended to 

improve the processes for submission and review of health economic evidence, based on lessons learned since 

the 2007 implementation of the original Guidance. Further, this update is intended to address changes in the 

field of cost-effectiveness over the past decade [4]. 

During deliberations, members of the ACIP Work Groups and members of the ACIP consider a substantial 

amount of scientific evidence, including medical, epidemiological, and economic data. The Guidance has been 

developed to help ensure economic presentations follow a consistent format and adhere to the technical

standards established in the fields of economic analysis and decision sciences. 

In addition to using this Guidance to help ensure the quality, consistency, and transparency of economic 

evidence presented at meetings of the ACIP Work Groups and the ACIP, ACIP Work Group leaders and CDC Work 

Group Leads are recommended to involve CDC economists from the inception of an ACIP Work Group or the 

inception of an economics-related project being considered by the ACIP Work Group. The involvement of a CDC 

economist can be initiated by contacting the ACIP Secretariat, who can coordinate these efforts. 

Objectives 

The objective of this Guidance is to provide standards on the description and presentation of the methods and 

results in economic analyses that are presented to an ACIP Work Group or the ACIP. To ensure that the 

standards have been achieved, this Guidance mandates a technical review of any economic materials that are 
3 

being considered for presentation to an ACIP Work Group or the ACIP. This Guidance does not provide an

exhaustive list of technical standards that cover every aspect of any potential economic analyses that might be 

presented, as such an exhaustive document would be impractical to create and use efficiently. For any topics or 

issues that are not explicitly described in the guidance, the ACIP Economics Lead, CDC Work Group Lead, and 

ACIP Secretary have discretion to make decisions or provide additional guidance as needed. 

This Guidance, and more generally the economic review process, aims to ensure that economic evidence: 

(1) adheres to broadly accepted technical standards in health economic analyses, (2) is transparent and 

understandable, and (3) is presented to the ACIP in a timely manner, such that important vaccination 

deliberations can occur as soon as any relevant information is available. 

Overview Overviewof guidance uidance uidance 

Materials 

The economic review process must be completed prior to the presentation of health economic evidence at a 

meeting of an ACIP Work Group or the ACIP. Economic evidence and presentation materials should be 

submitted to the CDC Work Group Lead with a copy sent to the ACIP Secretary. Submission of the following two 

items is required to begin the economic review process:

1) A report that presents the methodology and results: The report must provide a detailed description of the 

methods and results, similar to a manuscript that would be submitted to a peer-reviewed journal. There is 

no need for a detailed introduction section. The discussion section typically will be much shorter than that 

found in a published manuscript. Additional details regarding the content of this report are provided below. 

2) A set of slides intended for presentation at an ACIP Work Group meeting. Additional details regarding the 

content of the slide set are provided below. Also, attached to this Guidance is a set of template slides that 

should be used as a model to guide the content and organization of the presentation. Presentations at the 

ACIP general meeting and the ACIP Work Groups tend to be 10-15 minutes and 25-30 minutes, respectively.

Submission of the model (e.g., the file(s) from Microsoft Excel, TreeAge, or other software that contain(s) the 

model’s inputs, calculations, and outcomes) is encouraged when possible. At the discretion of the ACIP

Economics Lead and CDC Work Group Lead, the submission of the model may be required in some cases to help 

facilitate a more timely review process. 

Review procedures 

The report, slides, and any other materials intended to be presented at the ACIP general meeting must first be 

presented to an appropriate ACIP Work Group. These materials must be submitted to the relevant CDC Work 

Group Lead and the ACIP Economics Lead at least 8 weeks before anticipated presentation at an ACIP Work 

Group meeting. Under extraordinary circumstances, the timeline for submission of materials for review can be 

modified at the discretion of the ACIP Economics Lead and CDC Work Group Lead. The steps in a typical 

economic review process are presented in Figure 1. 
4 

 

Figure 1. Diagram of a typical economic study review process with presentations to ACIP Work Group and the 

ACIP. 

The review process can vary depending on a number of factors. We encourage studies to be submitted no later than 8 weeks 

prior to a targeted ACIP Work Group presentation date. Some reviews may require more time depending on the type of 

model and the level of detail provided in the report and presentation materials. Prior to submitting an economic study, early 

engagement with the appropriate CDC Work Group Lead can help to facilitate the review process. Not all reviewed 

submissions are presented to an ACIP Work Group or to the ACIP.

For economic evidence to be presented, all materials must complete the economic review process. Following 

the preliminary review by the ACIP Economics Lead, the Work Group Chair and CDC Work Group Lead will work 
5 

with the ACIP Economics Lead to identify individuals to conduct an anonymous technical review. Reviewers who 

are not affiliated with CDC may be used at the discretion of the ACIP Economics Lead. Reviewers may consult 

with relevant subject-matter experts if additional expertise would be beneficial to the review. Reviewers will 

return critical comments and any questions in writing to the ACIP Economics Lead. These comments will be 

forwarded to the researchers who submitted the economic analysis. This process should allow time for at least 

one round of comments by the reviewers and, if needed, revisions or responses by the authors prior to the ACIP 

Work Group presentation. The Work Group Chair, CDC Work Group Lead, and ACIP Economics Lead will 

determine if revisions and responses are sufficient to allow presentation to the ACIP Work Group. The Work 

Group Chair and CDC Work Group Lead, with input from the members of the Work Group, will decide if the

information will be presented at the ACIP meeting. As part of procedural due-diligence and transparency, the 

review comments will be provided by the ACIP Economics Lead to the ACIP Secretary and the appropriate CDC 

Work Group Lead. Review comments may be shared with ACIP Work Group members at the discretion of the 

CDC Work Group Lead. The specifics of the review procedures can be modified at the discretion of the ACIP 

Economics Lead, in consideration of the CDC Work Group Lead and reviewers, as applicable. Completion of the 

economic review process does not necessarily guarantee the opportunity to have an analysis presented at a 

meeting of an ACIP Work Group or the ACIP. In addition, completion of the economic review does not confer 

any explicit or implied endorsement of the model. 

Guidance for reports Guidance for reports 

The documents describing the methods and results of an economic analysis intended to be presented to an ACIP 

Work Group or the ACIP will contain the following sections and elements. Additional guidance on specific 

elements that should be included are described in Appendix B. Additional guidance on information and elements 

to include in a revision are described in Appendix C. 

(1) Affiliations 

All authors should clearly state any relevant affiliations. 

(2) Statements of conflicts of interest 

A separate section listing any potential conflicts of interest shall be included for each author, including any 

current or past sources of funding with a look-back period of at least 3 years (e.g., “Author A has received 

research funding from Company B, which is a manufacturer of vaccines.”). A look-back period longer than 3 

years may be applicable if a reasonable observer might conclude a potential conflict of interest exists. If there 

are no potential conflicts of interest, a statement to that effect must be included (e.g., “Author C: No known 

conflicts of interest.”). 

Methods 

(3) Study question 

The study question must be explicitly stated in detail (e.g., “In this study, we present results of a costeffectiveness analysis of routinely vaccinating age group X against Disease Y, using Vaccine Z, using three doses 

given once per year over three consecutive years compared to a strategy of not providing Vaccine Z to age 

group X.”). 
6 

(4) Perspective 

The study perspective(s) must be explicitly stated (e.g., “This study used the societal perspective.”) and defined 

so that the audience knows exactly what costs and outcomes are included. The study should typically be

conducted from the healthcare sector perspective, societal perspective, or ideally both of these perspectives. 

More comprehensive descriptions of these perspectives can be found in other publications [4]. Briefly,

healthcare sector perspective typically includes formal healthcare sector (medical) costs borne by third-party 

payers or paid for out-of-pocket by patients. The societal perspective typically includes all relevant medical 

costs regardless of payer, time costs of patients in seeking and receiving care, time costs of informal caregivers, 

transportation costs, effects on future productivity and consumption, and other effects occurring outside the 

healthcare sector. Both perspectives typically use quality-adjusted life-years (QALYs) as the primary health 

outcome measure [4]. While QALYs are commonly used, they may not always be the most easily understood 

outcome for the audience and they are often measured with substantial uncertainty. For these reasons, 

additional outcomes, such as cases averted, hospitalizations averted, and deaths averted, may be included as 

well. A complete list of the costs and outcomes included in a study is important to include in all studies, but 

perhaps especially for studies conducted from the societal perspective, where differences across studies in the 

types of costs and outcomes included under the societal perspective have been observed [5]. Alternative 

perspectives may be included as well, but clear and relevant justification must be provided to support their 

consideration. 

(5) Intervention strategies 

The vaccination strategy or strategies that are the focus of the study must be evaluated against a comparator (or 

baseline) strategy. The comparator strategy may include one or more vaccination strategies or may include 

other interventions, such as medical treatment. In some cases, the comparator strategy may include a “no 

vaccination” strategy, which would include any standard care for treating a condition in the absence of 

vaccination. 

(6) Time frame and analytic horizon 

The time frame of the intervention and analytic horizon of the analysis for each strategy must be clearly 

identified and must include appropriate justification (e.g., “We estimated the economic value of a catch-up 

vaccination program that lasts 5 years, with benefits and costs calculated over the lifetime of a modeled cohort. 

The time-frame of the intervention was assumed to be five years because that was the proposed duration of the 

catch-up program. The analytic horizon of the cohort’s lifetime was selected because available data suggests the 

vaccine-induced immunity protects individuals against infection, morbidity, and mortality related disease for the 

lifetime of the patient.”). 

(7) Economic model 

The analytic method used must be specified (e.g., cost-benefit, cost-effectiveness, or cost-utility analysis). The 

summary measure must be identified and defined. The basic model used in the analysis must be described in the 

text without use of mathematical notation. As appropriate, a clearly labeled figure (or flow chart, or schematic 

diagram) that depicts the important components and relationships of the model should also accompany the 

descriptive text. Authors may consider enhancing the written description with an equation consisting of 

descriptively named variables. If deemed necessary by the authors, an equation, or set of equations, using 

mathematical notation can be provided in a technical appendix. 
7 

(8) Inclusion of epidemiologic models 

When an epidemiologic model (e.g., a dynamic transmission model) is an integral part of an economic analysis, 

the authors must include a description of the model. This description should include a clearly labeled figure (or 

flow chart, or schematic diagram) that illustrates the model. The description of the model and any annotation in 

the figure must be done without the use of mathematical notation. Model diagrams should be presented in such 

a way that they are readily understood without extensive reading of the main text (i.e., figures must “stand 

alone”). For the description of an epidemiologic model, it would be insufficient to provide only a reference to 

another source (e.g., reference a previously published study). A model description using mathematical notation 

can be included in a technical appendix. 

(9) Health outcomes 

Health outcomes must be clearly identified (e.g., deaths, hospitalizations, outpatient visits, QALYs, number 

needed to vaccinate). Health outcomes must be relevant to the perspective and policy question. 

(10) Inputs 

The values and sources must be presented for all input values for all models used in the analysis (i.e., economic 

models and epidemiologic models), including probabilities, costs, health utility weights (or QALY decrements), 

time steps, and others. Values should include a base case value as well as ranges used in sensitivity analyses. If 

multiple sources are available for a particular input value, then the authors should provide a rationale for the 

value they choose to use in their model or how the values from multiple sources were combined. Values that are 

based on assumptions, expert opinion, or calculated (e.g., a residual probability) should be identified and should 

include a clear description of any relevant calculations and any approaches used to elicit expert opinion. 

a) Probabilities: Probabilities used in the analysis should be realistic and based on high-quality empirical 

evidence whenever possible. 

b) Costs: The study shall differentiate between direct medical, direct non-medical, and indirect (i.e., 

productivity losses). The currency year of cost data must be stated and all methods used to adjust cost 

values to a particular currency year must be clearly described. Authors must make sure that included 

costs are relevant to the stated perspective. 

Typically, the best way to report the required information is to provide a table listing all input values and 

sources. These tables typically include sufficient footnotes to enable a reader to readily understand the table 

without extensive reading of the main text (i.e., tables must “stand alone”). Preliminary discussions with the CDC 

Work Group Lead and the ACIP Economics Lead can be organized to help identify the best available evidence for 

epidemiologic and economic inputs. 

(11) Discounting 

All future costs and benefits included in the summary measure of a cost-effectiveness analysis shall be 

discounted to present values, including future health outcomes (e.g., future QALYs must be discounted to 

present). The discount rate used must be explicitly stated and justified. 

(12) Summary measures 

Results that answer the study question must be identified and presented. This summary economic measure(s) 

must be appropriate for the perspective used in the study and the policy question addressed by the study. 
8 

Authors may also present other results calculated during the analysis, such as total number of cases averted or 

number needed to vaccinate to prevent a case. 

(13) Sensitivity analyses 

The general goal of the sensitivity analyses should be to demonstrate how conclusions might change with 

changes in input values. Authors are strongly encouraged to conduct sensitivity analyses that allow them to 

identify which inputs or modeling choices are most influential in determining the overall results. The ranges and 

sources of any values used in the sensitivity analyses must be clearly presented. Authors should report input 

values used in sensitivity analyses in the same table in which they report input values for the base case (see 

item 10). 

Univariate sensitivity analyses (altering only one value at a time, and keeping all other values fixed at original 

values) are usually necessary but not adequate on their own. Authors should present some form of multivariate 

sensitivity analysis or an explanation as to why such analysis is not presented. Multivariate sensitivity analyses 

may be scenario analyses as well as probabilistic sensitivity analyses. Scenario analyses may be used to address 

specific scenarios of interest to the ACIP Work Group or to the ACIP, including specific combinations of input 

values and target populations. Whenever available, authors should use sensitivity analysis ranges that are based 

on clinically relevant or policy relevant cutoffs rather than those based on arbitrary changes in input values (e.g. 

plus or minus 20% of initially used values should be avoided if better alternatives can be identified). 

Assessment of input values with sensitivity analyses is especially important for any inputs for which there are 

limited data available. Often this is relevant for economic analyses that are presented to the ACIP because the 

ACIP frequently considers new vaccines or new populations to receive vaccination for which limited data has 

been collected or published. In these cases especially, extensive sensitivity analyses are helpful in determining 

the extent to which input values affect model outcomes and conclusions. 

(14) Independent replication 

In general, a report must include sufficient information about the model to allow a researcher, with relevant 

skills, to independently replicate the study. Challenges in replication of economic analyses have been

documented, even among studies that appear to do a good job reporting critical aspects of the model [6]. In 

order to meet the standard of independent replication, authors may need to provide additional details in a 

technical appendix. 

Results 

(15) Tables and graphs 

All tables and graphs used to present results must be readily understood without extensive reading of the main 

text (i.e., graphs and tables should “stand alone”). Authors may need to include footnotes that will help a reader 

understand each table and graph. 

Figures and graphs should be drawn using the standard guidelines for graphical representation of data. Detailed 

guidelines can be found in texts such as Tufte [7]. Examples of such guidelines include: 

a) Pie charts are almost always unacceptable. 

b) Horizontal and vertical grid lines are usually not needed. 
9 

c) For line graphs, typically no more than four lines (variables) drawn per graph. 

d) For column and bar graphs, typically no more than four bars (variables) included per graph. Avoid 

using “stacked” bars or columns (e.g., a column that has several elements that add to 100%). 

e) To allow easy printing and copying, graphs and figures should be drawn so they are clear and easy-toread, even in gray-scale.

(16) Sensitivity analyses and influential variables

Authors should present their sensitivity analyses in a clearly identified results section, complete with relevant 

tables and graphs. Whenever possible, authors should present a list of the most influential inputs, input groups, 

or modeling choices, as identified through sensitivity analysis (see item 13). 

Discussion 

(17) Overall 

The discussion section should be sufficient to address items 18 – 20 below, but does not need to be as long as 

the discussion section in a typical peer-reviewed manuscript. 

(18) Limitations 

Study limitations must be discussed. Limitations shall include a critical discussion of any epidemiologic model 

and an assessment of the quality of input data (see items 8 and 10). 

(19) Relation to other relevant studies 

Findings must be discussed in relation to other, similar studies if any are available. 

(20) How results may change 

There must be an explicit discussion, drawing from the sensitivity analyses, of how model results would change 

if key assumptions or values were to change. 

(21) No policy implications 

Unless otherwise requested by the Work Group Chair, the report should NOT include a discussion of the policy 

implications of the results. Regarding cost-effectiveness analyses in particular, the discussion of policy 

implications should avoid any references to explicit or implicit cost-effectiveness thresholds. Because it is the 

responsibility of the ACIP and CDC to make policy interpretations, any discussion of policy implications will be 

deleted if included in the materials used for ACIP deliberations, unless specifically requested by the Work Group 

Chair. 

(22) Details not addressed in this Guidance 

For further guidance regarding analytic methods and presentation of results, researchers may consult the 

following standard texts: 

(a) Prevention Effectiveness: A Guide to Decision Analysis and Economic Evaluation, 2d. Ed. Anne Haddix, 

Steven Teutsch, and Phaedra Corso, editors. (New York: Oxford University Press, 2003) [8], and 

(b) Cost-Effectiveness in Health and Medicine. 2nd Edition. Peter J Neumann, Gillian D Sanders, Louise B 

Russell, Joanna E Siegel, and Theodore G Ganiats, editors. (New York: Oxford University Press, 2016) [4]. 
10 

(c) Methods for the Economic Evaluation of Health Care Programmes, 4th Edition. Michael F Drummond, 

Mark J Sculpher, Karl Claxton, Greg L Stoddart, and George W Torrance. (New York: Oxford University 

Press, 2015) [9]. 

(23) Additional information 

A number of other resources are also available from CDC and the health economics research community: 

(a) Link to the Community Guide Economic Evaluation abstraction form where all of the important 

components of an economic analysis are presented: 

https://www.thecommunityguide.org/sites/default/files/assets/EconAbstraction_v5.pdf 

(b) Many standards for conducting economic evaluations and other resources are also on the CDC Public 

Health Economics and Tools web page: https://www.cdc.gov/stltpublichealth/pheconomics/

(c) The Guide to Community Preventive Services book, Chapter 11: “Understanding and Using the 

Economic Evidence” [10] 

(d) Allen S Detsky and Andreas Laupacis. Relevance of cost-effectiveness analysis to clinicians and policy 

makers. JAMA. 2007; 298(2):221-4. [11] 

Guidance for presentation materials Guidance for presentation materials 

Principles for presentations 

After the technical review is complete, the CDC Work Group Lead and Work Group Chair may decide to have a 

presentation to an ACIP Work Group only, have a presentation to an ACIP Work Group and the ACIP, or to have 

no presentation. If a presentation is planned for an ACIP general meeting, then a presentation to an ACIP Work 

Group must occur before the ACIP general meeting presentation. This procedure may be modified at the 

discretion of the ACIP Economics Lead, CDC Work Group Lead, and Work Group Chair. 

For presentations at an ACIP general meeting, many of the ACIP members may not have had the opportunity to 

read the report with all the methodological and results described in detail. In addition, because the ACIP 

meetings are open to the public, a large number of audience members also may not have had the opportunity to 

read the report. Therefore, the presenter of the health economics study should realize that most of the 

audience at the ACIP general meeting will not be aware of the study details, and that the audience members 

have diverse backgrounds. Generally, presenters describing single health economics study or summarizing 

multiple health economics studies will also face a time constraint, typically 10-15 minutes for presentation with 

an additional 5-10 minutes for questions. 

Presentations at the ACIP general meeting are typically made in a large auditorium and presenters should 

prepare slides with that in mind. The number of slides, slide style, colors, font size and style, and the number of 

words used on a slide all affect whether the audience can readily read and comprehend the presentation. 

Presenters should attempt to balance the need to provide detail within the limited time available for 

presentation and the audience’s need to easily understand the most important aspects of the study. 

Presentations at the ACIP general meeting typically focus on describing the results of a single economic model. 

However, some presentations may also compare and summarize findings from two or more economic models. 

Usually presentations that summarize two or more economic models are given by CDC staff. In any case, all 
11 

economic material that is presented or being planned for presentation to the ACIP or an ACIP Work Group 

should adhere to this Guidance. A recent study from 2019 assessed preferences regarding the presentation of 

economic results at ACIP meetings [12]. This study surveyed professionals familiar with ACIP, including current 

and past ACIP members, members of ACIP Work Groups, and CDC Work Group liaisons. From all respondents in 

the sample, the authors found that the three most important attributes of health economics presentations 

were: (1) summary results or cost-effectiveness ratio, (2) model overview and structural assumptions, and (3) 

relationship of the results to other relevant studies. Sensitivity analyses and methods were also found to be 

important attributes from health economics presentations, particularly among respondents who had previous 

experiences with health economic models. 

Template slides 

The attached set of template slides provides an outline of the type of slides and suggested layout of each type of 

slide. The subject matter of the slides in the template is designed to present the most important aspects related 

to the methods, results and limitations of the study. Printed copies of presentation slides are usually made 

available at ACIP meetings. Ideally, and especially if a presentation is scheduled for an ACIP meeting, all graphs 

and tables should be legible when printed in a format of six slides per page. 

 
12 

Major changes from the previous version of the Guid Major changes from the previous version of the Guidance hanges from the previous version of the Guidance 

1. Revised and edited text throughout document. 

2. Updated the references throughout document. 

3. Expanded “Rationale” section to incorporate recently adopted Evidence to Recommendation 

framework. 

4. Revised “Objectives” section to clarify objectives of economic review process. 

5. Added details and clarified several steps in the review process. 

a. Clarified review process: A preliminary review by the ACIP Economics Lead precedes the 

technical review by peer reviewers. Review and ACIP Work Group comments may be 

incorporated into any final presentation to the ACIP. 

b. Clarified timeline: Presentations at an ACIP general meeting must occur after a presentation to 

the appropriate ACIP Work Group. Reports and slides should be submitted at least 8 weeks prior 

to the anticipated ACIP Work Group presentation. 

6. Added details of look-back period to “Statements of conflicts of interest” section. 

7. Expanded descriptions in the “Perspectives” section. 

8. Required inclusion of model diagram in “Economic model” section. 

9. Specified in “No policy implications” section that explicit or implicit cost-effectiveness thresholds should 

be avoided. 

10. Expanded discussion of the objectives and rationale for sensitivity analyses in “Sensitivity analyses”

section. 

11. Added an appendix of Frequently Asked Questions (FAQs), which addresses several areas in more detail. 

12. Added an appendix of elements that are helpful to include in submissions. 

13. Added an appendix of protocols for model revisions, following any changes to assumptions or the 

structural components of a model that occur during the course of the economic review. 

 
13 

Appendix A: Frequently Asked Questions (FAQ) for Appendix A: Frequently Asked Questions (FAQ) for Frequently Asked Questions (FAQ) for Health Economi Health Economics Studies Presented to the Advisory cs Studies Presented to the Advisory 

Committee on Committee on Immunization Practices (ACIP) Immunization Practices (ACIP) Immunization Practices (ACIP) 

1. How long should the report be? 

There is no restriction or requirement for the length of the report as long as sufficient details and methods are 

presented such that the analyses could be replicated by a sufficiently trained individual. All included content should 

be directly relevant to the research objective and policy question being assessed. To facilitate prompt reviews, 

reports may be organized into a main text, approximately the length of a peer-review journal manuscript, and a 

technical appendix, which has no suggested length. 

2. How long does the economic review take to complete?

The entire review process is designed to take about 8 weeks, which allows for approximately 2 weeks for the 

preliminary review, followed by 4 weeks for the technical review, followed by 2 weeks for revision and resubmission, followed by the ACIP Work Group presentation. In some cases, particularly when there are multiple 

iterations of review and revision, the process can take longer than 8 weeks. Every effort is made to ensure reviews 

are thorough and timely. One of the objectives of the review process is to deliver timely information to the ACIP. 

When preparing materials to be submitted to the review process, there are several considerations and elements of 

the report that can help to ensure a prompt review will occur: 

• All documents submitted for review should be free of typos and editorial inconsistencies. 

• Tables and figures should add value to the report and contribute directly to the understanding of the 

summary measure. 

• Additional results that do not contribute directly to understanding the summary measure may be included 

in tables or figures in a technical appendix. 

• A table (or tables) of intermediate outcomes, such as total (or population-level) case counts, vaccine doses 

distributed, costs, and health outcomes 

• A table (or tables) of specific scenario analyses, such as a complete set of results from the comparator 

scenario (e.g., a “no vaccination” scenario or a comparison vaccine scenario), which could be used to 

assess underlying disease incidence assumptions. 

• The model itself, which includes any relevant model software files or model code (such as a Microsoft Excel 

document, a TreeAge file, or others) that contains all the inputs, calculations, and results in the study. 

The economic review process follows the format of a review for a peer-reviewed journal. In general, the objective of 

the review is to ensure a clear and accurate presentation of the methods, including model structure, inputs and all 

assumptions, results, and to ensure the study follows standard procedures. 

3. Do all studies that are submitted for an economic review eventually get presented at a meeting of the 

ACIP or an ACIP Work Group? 

No, not all studies are able to complete the economic review process. Even in cases where materials have 

completed the economic review process, there are many reasons, unrelated to the review, why a particular study 

may not be presented. One common reason is the effort of meeting organizers to be as efficient as possible with the 

use of meeting time, when there are multiple models are available for consideration. 

4. Is there a separate review process for models or economic studies that have already been reviewed and 

approved for presentation to ACIP or an ACIP Work Group? 

The process is the same but to facilitate the review and approval of a previously reviewed model, the following 

items should be submitted: 
14 

(1) Documentation that a previous review was conducted and the model was approved for presentation, 

which includes a copy of all previous review reports as well as documentation, such as an email, that the 

original analysis had been approved to be presented to an ACIP Work Group or to the ACIP; 

(2) A list of all changes made to the model since the previous review was conducted, which includes 

changes made to input parameters as well as any associated changes in results or outcomes (See also: 

Appendix C); and 

(3) A clearly stated rationale for why each change was made. 

These three items should be submitted in addition to the report and presentation materials that are required for all 

submissions to the economic review process. In some cases, the timeline of previously reviewed models can be 

modified at the discretion of the ACIP Economics Lead or CDC Work Group Lead. Submission of these items does not 

guarantee that a modified timeline may be used, so all materials should be submitted 8 weeks prior to any 

scheduled ACIP Work Group presentation. 

5. Do presentations that contain short summaries (1-2 slides) of the results from already published 

economic assessment(s) need to go through the CDC economic review process? 

Generally, all economic material included in presentations needs to go through the economic review process. 

However, exceptions may be allowed at the discretion of the ACIP Economics Lead and CDC Work Group Lead. 

6. Do presentations to an ACIP Work Group (not planned to be presented to the ACIP) need to be 

submitted to the economic review process? 

All presentations need to be submitted and given approval to be presented. However, exceptions may be made at 

the discretion of the ACIP Economics Lead and CDC Work Group Lead. When such exceptions are granted, the 

presentations must include a slide with the following disclaimer statement: “This was not reviewed by the CDC 

economic review process.” 

7. What should be used as the comparator scenario? (I.e., Vaccination vs. no vaccination. Or, vaccine A vs. 

vaccine B) 

The selection of the comparator scenario should closely follow the stated policy question of interest to the ACIP 

Work Group or the ACIP. In general, the comparator scenario will include any standard preventions or treatments 

that are currently practiced, but not including the intervention (or vaccine) that is being evaluated. In some cases, 

the stated policy questions may evolve while the economic analyses are being conducted, so frequent 

communication between the CDC Work Group Lead, ACIP Economics Lead, and the economic research team can

help to ensure that the economic analysis addresses the policy question of greatest interest to the ACIP Work Group 

and the ACIP. 

8. If two or more economic analyses are being conducted on a particular vaccine policy decision, is there

any guidance for how these analyses should be presented? 

This is up to the discretion of the Work Group Chair, CDC Work Group Lead, and ACIP Economics Lead. In general, 

the Work Group Chair and CDC Work Group Lead use their best judgement and try to set an agenda to present, or 

focus presentation time, on whatever information the Work Group Chair and CDC Work Group Lead determine to 

be the most useful to the ACIP Work Group and the ACIP during their deliberations. In the past, a comparison-style 

presentation has been used when two independent models for herpes zoster vaccinations were developed and 

assessed the same policy question [13]. At other times in the past, a summary-style presentation has been used to 

describe three models on human papillomavirus vaccinations [14, 15]. Whatever the format, however, it is 

important that all health economic studies that are to be presented or summarized have gone through the CDC 

economic review process. 
15 

9. What types of outcome(s) should be used in the summary measures? 

The most useful outcome measure depends on the specific policy question being assessed. This can be determined 

by working with the CDC Work Group Lead, Work Group Chair, and ACIP Economics Lead. In past presentations, the 

cost per QALY gained has been frequently used as a summary measure. In some cases, reporting number needed to 

vaccinate (NNV) or cost-effectiveness ratios (CERs) that utilize different health outcomes may be helpful. NNV to 

prevent a case of a particular infection-related sequela can be intuitive to medically-trained decision-makers, but 

may not capture all outcomes related to decision making and may not capture all aspects of a disease system, such 

as adverse events or in some cases benefits that occur due to indirect, or herd immunity, effects. 

10. How should number needed to vaccinate (NNV) be calculated? 

At least two methods exist to calculate NNV: (1) A model-based approach, where NNV = the size of the vaccinated 

cohort / the predicted number of events, such as cases or deaths, that are prevented in the vaccinated cohort over 

its lifetime [16] and (2) a formula-based approach, where NNV = 1 / (annual incidence of event in the unvaccinated 

× vaccine effectiveness) [17]. Either approach would be acceptable, however the model-based approach may 

preferred in some cases because it can more easily account for additional attributes of the vaccine and vaccinepreventable disease, such as the time-frame of risks for unvaccinated individuals [18] and herd-immunity, provided 

herd-immunity is captured by the model. 

11. What other types of outcome(s) should be included in the report and presentation materials? 

The most useful intermediate outcomes may depend on the specific policy question being assessed and can be 

determined by working with the CDC Work Group Lead, Work Group Chair, and ACIP Economics Lead. For example, 

in addition to reporting a summary measure, such as the cost per QALY gained for a given intervention strategy vs. 

a comparison strategy, analysts could also include intermediate outcomes for each strategy, such as total 

vaccination costs, medical costs, cases, and QALY. Reporting the total values for all scenarios that are investigated 

can allow for the calculation of CERs across different scenarios if these CERs are not already included in the results. 

12. What should be the adherence rate(s) for multi-dose vaccines? 

The adherence rate assumptions should allow for a valid comparison of the strategies being evaluated. For 

example, when comparing Vaccine A to Vaccine B, accounting for adherence might be particularly important when 

Vaccine A requires 1 dose and Vaccine B requires 2 doses, but less important when both vaccines require the same 

number of doses. 

When allowing for the possibility of less than 100% adherence, the applied adherence rates should be based on the 

best available data. When no data are available, assumptions based on expert opinion or on adherence rates from 

a comparable vaccine may be required. 

13. Are there specific types of sensitivity analyses that are preferred, such as univariate, multivariate, or 

probabilistic? 

Typically, most health economic analyses presented to the ACIP include univariate sensitivity analyses and some 

form of multivariate sensitivity analyses. Multivariate sensitivity analyses may include scenario analyses as well as 

probabilistic sensitivity analyses. The specific type(s) of sensitivity analyses used should be sufficient to capture the 

full range of uncertainty in the analysis and to understand how the values of inputs, input groups, and modeling 

choices affect the estimated summary measures. Conducting univariate sensitivity analyses alone is usually not 

adequate to capture the full range of uncertainty. Sensitivity analyses should investigate all relevant factors (e.g., 

vaccine price, vaccine effectiveness, waning immunity, herd effects, indirect costs, and others). 

14. Should CEAs include the potential costs of outbreaks and outbreak responses? 

Outbreak costs may be considered for certain policy decisions. The authors of the study and the reviewers should be 

able to judge whether these costs are relevant to the research question and policy decision. Certain vaccine-
16 

preventable diseases do not experience outbreaks in the traditional sense, such as HPV infection. In a general 

population setting, costs of certain disease outbreaks, while substantial for a given outbreak, are unlikely to impact 

results that focus on vaccinating a general population due to the low probability of an outbreak. These costs are 

more important in an outbreak response setting, such as was addressed with the January 2018 recommendation to 

vaccinate persons at increased risk for measles and mumps with the MMR vaccine[19]. For some models and policy 

decisions, the inclusion of outbreak costs may not yield additional benefits in terms of model accuracy. 

15. What perspective should the study use? 

The study perspective should be appropriate for the policy decision being assessed, the nature of the vaccine 

program, and the nature of outcomes associated with the vaccine-preventable disease. There are a number of 

potentially relevant perspectives including the healthcare sector perspective, healthcare sector with time cost 

perspective, and the societal perspective [4]. Results may be presented for more than one perspective. Any 

perspectives that are used should be clearly stated and justified. A clear statement of the perspective should list all 

the costs and health outcomes that were included in the analysis. 

16. What kinds of costs and outcomes should be included in CEAs? 

Past analyses have typically included, at a minimum, some measure of direct medical costs, health outcomes (e.g., 

QALYs, cases, hospitalizations, or deaths), and vaccine program costs. In addition, indirect costs or productivity 

losses may be considered for certain models and it may also be helpful to present results with and without 

productivity losses or indirect costs. As an example, presentation slides reporting results from an economic 

evaluation of meningitis B vaccinations among college students [20] incorporated productivity losses in different 

ways, depending on the health outcome that was used. As with the study perspective that is used, all costs and 

outcomes that are considered in a model should be clearly stated and justified. 

17. Is there any additional guidance for studies that provide an economic assessment of travel vaccines? 

Unless otherwise directed by the ACIP Economics Lead, CDC Work Group Lead, or Work Group Chair, the Guidance 

applies to any study whether it is an assessment of travel vaccines or vaccines for the general population. 

 
17 

Appendix B Appendix B ppendix B: List of elements ( elements ( elements (tables orfigures figures figures)that are helpful to include in submissions that are helpful to include in submissions that are helpful to include in submissions 

Many of the items described in this appendix would be considered standard components of reporting economic 

evaluation and cost-effectiveness research. This list is non-exclusive and so many other elements and results 

presentations are possible and reasonable, given the type of analyses being conducted. Some items in this list 

may not be practical to present in a report as a single table. For example, it might make more sense to report 

intermediate outcomes for an intervention scenario(s) and a comparator scenario(s) in two (or more) tables. 

Presenting multiple tables for parameters, intermediate outcomes, base case results and/or sensitivity analyses 

would be reasonable under a number of circumstances. This list is designed so that when reviewers receive 

submissions, the submissions contain sufficient information to conduct a reasonable and efficient review. 

(1) Parameters 

A complete list of all parameters utilized in the model must be presented in the main text or in an appendix. For 

each parameter, all base case values and all ranges used in sensitivity analyses must be presented. If

probabilistic sensitivity analyses are conducted, then all parameters used in the probabilistic sensitivity analyses 

must be associated with an appropriate and reproducible probability distribution. The parameter table(s) should 

also include all relevant citation or source information for each parameter. Any modifications or calculations 

done to values from the source material must be documented in the table footnotes or in the text. 

Examples of presentation of model parameters in other published studies: 

• See Supplemental Tables 1 to 3 for parameter values and ranges and Supplemental Table 6 for 

distributions used in probabilistic sensitivity analyses from Prosser et al. (2019) [21] 

• See Tables 1 and 2 and Appendix Tables 1 to 34 for parameter values and ranges from Chesson et al. 

(2018) [22] 

• See Table 1 for parameter values, ranges, and distributions used in probabilistic sensitivity analyses from 

Le and Rothberg (2015) [23] 

(2) Model diagram 

A visual representation of the model can be helpful for understanding the model structure and the relationships 

between parameters. 

Examples of model diagrams in other published studies: 

• See Appendix Figure 1 and 2 from Le and Rothberg (2015) [23] 

• See Figure 1 from Prosser et al. (2019) [21] 

(3) Intermediate outcomes 

The presentation of intermediate outcomes (such as vaccination costs, medical costs, cases, deaths, and QALYs) 

for the base case analyses has been found to be helpful in assessing models as well as in comparing the results 

across any other models that might be available. When possible, intermediate outcomes should be stratified by 

disease type. For example, the total number of QALYs gained and averted medical costs due to the prevention of 

symptomatic cases, hospitalized cases, fatal cases, and any other disease states that might be relevant to the 

vaccine preventable disease(s) under consideration. Ideally, these intermediate outcomes would be reported for 

a “status quo” (or “no vaccination”) scenario as well as for an “intervention” (or “vaccination”) scenario so that 
18 

any base case cost-effectiveness ratios that are reported in base-case results can be calculated directly from this 

table. 

Examples of presentation of intermediate outcomes in other published studies: 

• See Supplemental Figure 5 from Prosser et al. (2019) [21] 

• See Table 2 from Wateska et al. (2019) [24] 

(4) Base case results 

The base case results should include presentation of total and incremental costs and benefits (or effectiveness). 

As with intermediate outcomes, in some cases base case results stratified by scenario and health outcomes can 

be useful. 

Examples: 

• See Table in the main text for presenting total and incremental results stratified by scenario from 

Prosser et al. (2019) [21] 

• See Table 3 for results stratified by scenario and health outcomes from Chesson et al. (2011) [25] 

(5) Sensitivity analyses and scenario analyses 

In many cases, standard presentation of results from one-way and multi-way sensitivity analyses may be useful. 

Examples of standard presentation would include tornado diagrams, tabular presentations of incremental costeffectiveness ratios from one-way sensitivity analyses or from scenario analyses, and cost-effectiveness 

acceptability curves. 

Examples of presentation of sensitivity analyses in other published studies: 

• See Figures 2 for cost-effectiveness acceptability curves, Figure 3 for multi-way analyses, and 

Supplement Tables 7 for multi-way analyses, Supplement Table 8 for one-way analyses, and Supplement 

Table 9 for multi-way analyses, all from Prosser et al. (2019) [21] 

• See Figure 2 for tornado diagram and Figure 3 for multi-way analysis from Le and Rothberg (2015) [23] 

• See Figure 1 for cost-effectiveness acceptability curves from Wateska et al. (2019) [24] 

• See Figure 1 for tornado diagram, Figure 2 for two-way analyses, and Figure 3 for cost-effectiveness 

acceptability curves from Le and Rothberg (2018) [26] 

 
19 

Appendix C: Protocols for revisions Appendix C: Protocols for revisions 

During the economic review process, authors have the ability to revise their work and submit updated results 

that incorporate changes to model structure and/or assumptions. When submitting any revised work, it is 

essential to clearly document all changes made to the model with justifications and any updated source

material. This documentation may occur in a cover letter or in a clearly labeled section of the revised report. In 

addition, if during a single revision there are multiple changes made to the model or any parameter values used 

in the model, these changes must be described in an iterative, step-by-step format that includes updated results 

alongside each incremental change. This is necessary to completely understand the impact of any new change to 

the model in the context of the other changes that may have also occurred. If sufficient documentation for these 

changes is not provided, then the review process may be interrupted and may need to be re-started. 

As an example, suppose during the review process, the study authors realize they need to update the costs of a 

vaccine. At the same time, suppose the authors identify a minor error in their modeling program related to a 

disease transmission equation. Updating the costs and resolving the error both result in small changes to the 

results of the study. The two changes and the impact of these changes on the results must be documented, 

individually and when combined, during the resubmission of the model. A very simple example of the kind of 

documentation that would be required is in Table C1. 

Table C1. An example of documentation for multiple changes to a model during revision 

Model descriptions:

(1) Original 

submission 

(submitted: 

2/1/2019) 

(2) Original 

submission plus 

vaccine costs updated 

to $200/dose (from 

$190/dose) 

(3) Original 

submission plus 

corrected error 

in model code 

(4) Revised submission 

(submitted: 2/15/2019), 

with vaccine costs updated 

to $200/dose and corrected 

error in model code 

Base case ICER $/QALY 

for Vaccination 

(vs. no vaccination) 

75,905 89,460 56,373 72,916 

 

 
20 

References References 

1. Advisory Committee on Immunization Practices. ACIP Charter. 2018; Available from: 

https://www.cdc.gov/vaccines/acip/committee/charter.html. 

2. Advisory Committee on Immunization Practices. ACIP Evidence to Recommendations Framework. 2018; 

Available from: https://www.cdc.gov/vaccines/acip/recs/grade/downloads/ACIP-evidence-rec-frame-508.pdf. 

3. Lieu, T., M.I. Meltzer, and M.L. Messonnier, Guidance for health economics studies presented to the 

Advisory Committee on Immunization Practices (ACIP). Atlanta: Centers for Disease Control and Prevention, 

2007. 

4. Neumann, P.J., et al., Cost-effectiveness in Health and Medicine. 2016: Oxford University Press. 

5. Drost, R.M., et al., Conceptualizations of the societal perspective within economic evaluations: A 

systematic review. International Journal of Technology Assessment in Health Care, 2017. 33(2): p. 251-260. 

6. McManus, E., et al., The barriers and facilitators to model replication within health economics. Value in 

Health, 2019. 22(9): p. 1018-1025. 

7. Tufte, E., The Visual Display of Quantitative Information, 2nd Edition. 2001, Graphics Press: Cheshire, 

Connecticut. 

8. Haddix, A.C., S.M. Teutsch, and P.S. Corso, Prevention Effectiveness: A Guide to Decision Analysis and 

Economic Evaluation. 2003: Oxford University Press. 

9. Drummond, M.F., et al., Methods for the Economic Evaluation of Health Care Programmes, 4th Edition. 

2015: Oxford University Press. 

10. Stone, G., et al., Understanding and using the economic evidence, in The Guide to Community Preventive 

Services. 2005, Oxford University Press. p. 449-63. 

11. Detsky, A.S. and A. Laupacis, Relevance of cost-effectiveness analysis to clinicians and policy makers.

Journal of the American Medical Association, 2007. 298(2): p. 221-224. 

12. Richardson, J.S., M.L. Messonnier, and L.A. Prosser, Preferences for health economics presentations 

among vaccine policymakers and researchers. Vaccine, 2018. 36(43): p. 6416-6423. 

13. Leidner, A.J. Overview of two economic models that assess the cost-effectiveness of herpes zoster 

vaccinations. In Meeting of the Advisory Committee on Immunization Practices, October. 2017. Atlanta GA: CDC. 

14. Chesson, H. Overview of cost-effectiveness of 9-valent HPV vaccination. In Meeting of the Advisory 

Committee on Immunization Practices, February. 2015. Atlanta, Georgia: CDC. 

15. Leidner, A.J. Overview of three economic analyses of pneumococcal vaccinations at age 65. In Meeting of 

the Advisory Committee on Immunization Practices, February 2019. 2019. Atlanta GA: CDC. 

16. Brisson, M., et al., Estimating the number needed to vaccinate to prevent diseases and death related to 

human papillomavirus infection. Canadian Medical Association Journal, 2007. 177(5): p. 464-468. 

17. Hashim, A., et al., How and why researchers use the number needed to vaccinate to inform decision 

making--a systematic review. Vaccine, 2015. 33(6): p. 753-8. 

18. McLaughlin, J.M., et al., Rethinking number-needed-to-vaccinate for pneumococcal conjugate vaccines in 

older adults: current and future implications. Vaccine, 2017. 35(40): p. 5360-5365. 

19. Marin, M., et al., Recommendation of the Advisory Committee on Immunization Practices for use of a 

third dose of mumps virus–containing vaccine in persons at increased risk for mumps during an outbreak.

Morbidity and Mortality Weekly Report, 2018. 67(1): p. 33. 

20. Ortega-Sanchez, I. Economic evaluation of vaccinating US adolescents and college students against 

Mening B, preliminary. In Meeting of the Advisory Committee on Immunization Practices, June. 2015. Atlanta, 

GA: CDC. 

21. Prosser, L.A., et al., A Cost-Effectiveness Analysis of Vaccination for Prevention of Herpes Zoster and 

Related Complications: Input for National Recommendations. Annals of Internal Medicine, 2019. 170(6): p. 380-

388. 

22. Chesson, H.W., et al., Cost-effectiveness of nonavalent HPV vaccination among males aged 22 through 

26 years in the United States. Vaccine, 2018. 36(29): p. 4362-4368. 

23. Le, P. and M.B. Rothberg, Cost-Effectiveness of herpes zoster vaccine for persons aged 50 years. Annals 

of Internal Medicine, 2015. 163(7): p. 489-97. 
21 

24. Wateska, A.R., et al., Cost-effectiveness of adult pneumococcal vaccination policies in underserved 

minorities aged 50–64 years compared to the US general population. Vaccine, 2019. 37(14): p. 2026-2033. 

25. Chesson, H.W., et al., The cost-effectiveness of male HPV vaccination in the United States. Vaccine, 2011. 

29(46): p. 8443-8450. 

26. Le, P. and M.B. Rothberg, Cost-effectiveness of the adjuvanted herpes zoster subunit vaccine in older 

adults. JAMA: Internal Medicine, 2018. 178(2): p. 248-258.