ACIP Guidance for Health Economics Studies 

Guidance for Health Economics Studies Presented to the Advisory Committee on 

Immunization Practices (ACIP) 

Prepared by 

The ACIP Ad Hoc WG on Economic Analyses 

(Members listed alphabetically) 

Tracy Lieu, M.D., M.P.H. 

Professor and Director, Center for Child Health Care Studies Department of Ambulatory Care 

and Prevention 

Harvard Pilgrim Health Care and Harvard Medical School 

ACIP Member July 2004-June 2008 

Martin I. Meltzer, M.S., Ph.D. 

Senior Health Economist & Distinguished Consultant 

CDC/NCPDCID/DEISS 

Mark L. Messonnier, M.S., Ph.D. 

Lead Economist 

CDC/NCIRD/ISD 

11/13/2007 Page 1 of 9 
ACIP Guidance for Health Economics Studies 

Guidance for Health Economics Studies Presented to the Advisory Committee on 

Immunization Practices (ACIP) 

Rationale for this guidance 

The Charter of the Advisory Committee on Immunization Practices (ACIP) states that, when 

considering recommendations for use of a vaccine, ACIP members’ deliberations should include 

consideration of vaccine efficacy as well as cost:beneft and risk:benefit analyses. In recent years 

there has been a trend toward an increasing number of economic studies of vaccines presented as 

evidence for the Committee to consider. Precedent has been established by journals such as the 

British Medical Journal and Vaccine, in that they have adopted standards by which editors and 

reviewers should consider the quality of submitted economic studies. The Academy of Managed 

Care Pharmacy also has adopted guidelines for economic data submitted in support of formulary 

consideration. This Guidance has been developed to help ensure the quality of economic data 

presented to the ACIP and its work groups (WG). 

To ensure the value and consistency of economic data presented to the ACIP, WG chairs are 

advised to involve CDC economists from the inception of the WG. Participation of CDC 

economists can be facilitated by the National Center for Immunization and Respiratory Diseases 

(NCIRD) lead economist, who can coordinate such involvement. 

Objective of this guidance 

It would be impossible to provide a set of standards for methods, inputs and values for inputs that 

would cover all the diverse vaccine-related issues that the ACIP considers. The objective of this 

guidance is to provide a framework for the description and presentation of methods used to 

examine the economics of a vaccine-related issue, such that the ACIP readily can understand the 

science of the analyses. To ensure that such standards have been achieved, this guidance 

specifically incorporates technical review of any economic materials that are to be presented to 

the ACIP. 

GUIDANCE 

Materials to be presented 

All researchers (internal or external to CDC) wishing to present a health economics study (either 

new or revised/updated) to the ACIP (or one of its WGs) shall present the following material: 

1) Document or report providing methods and results The document should provide a 

detailed description of the methods and results. The level of detail should be similar to that 

provided in a manuscript that would be submitted to a peer-reviewed journal. However, the 

document should not be bound by journal restrictions on length of the methods and results 

sections. There is no need for a detailed introduction section. The discussion section typically 

would be shorter than that found in a published manuscript. Additional details regarding the 

content of this document are provided below. 

11/13/2007 Page 2 of 9 
ACIP Guidance for Health Economics Studies 

2) Slide sets and other presentation materials (e.g., handouts) The principles and template 

for slides intended for presentation to the ACIP which outline the methods and results, are 

provided in the “Guidance: format for presentation of slides” section. Additional details 

regarding slide content also are provided (See attached set of “template slides.”) 

Timeline for materials to be made available and review process 

The report, slides, and other presentation material must be submitted to the relevant ACIP WG 

chair and CDC lead staff person no later than 8 weeks before the ACIP general meeting or WG 

meeting at which the analysis is proposed to be presented. Under extraordinary circumstance, an 

appeal can be made to the WG chair, the CDC lead staff person, and the NCIRD lead economist 

to submit the report fewer than 8 weeks before the ACIP meeting. In case of such an appeal, the 

lead economist will document the justification and granting of exceptions in collaboration with 

the WG chair and the CDC lead staff person. 

The NCIRD lead economist (or designee) will work with the WG chair and CDC lead staff 

person to identify reviewers for an anonymous peer review. Non-CDC reviewers may be used if 

a particular area of expertise is not available among CDC economists. Reviewers will consult 

with relevant CDC subject-matter experts and return comments and questions in writing to the 

NCIRD lead economist (or designee) at least 4 weeks in advance of the formal presentation. 

These comments will be forwarded to the WG chair and CDC lead staff person who will forward 

them to the appropriate WG members and to the researchers submitting the economic analysis. 

This process will allow time for at least one round of comments and revisions or responses prior 

to the formal presentation. The relevant WG chair, the CDC lead staff person, and the NCIRD 

lead economist (or designee) will determine if revisions and responses are sufficient to allow 

presentation. If differences persist, the WG chair and the CDC lead staff person, in conjunction 

with all members of the WG, will decide if the information will be presented to the ACIP. 

Reviewer comments also will be provided to the WG and the ACIP to explain the differences 

that may exist. 

11/13/2007 Page 3 of 9 
 

 

ACIP Guidance for Health Economics Studies 

GUIDANCE: Content of document 

A document describing the methods and results of a health economics study for consideration by 

the ACIP will contain the following sections and elements: 

1) Affiliations 

All authors shall include their affiliations. 

2) Statements of conflict or potential conflicts of interest 

A separate section listing any potential conflicts of interest shall be included for each author. If 

there are no potential conflicts of interest, a statement to that effect must be included (e.g., 

Author A: No conflicts of interest). 

METHODS 

3) Methods: the study question 

The study question must be explicitly stated in detail (e.g., In this study, we present results of a 

cost-effectiveness analysis of routinely vaccinating age group XX against Disease Y, using 

vaccine Z, using three doses given once per year over three consecutive years.) 

4) Methods: the perspective 

The study must be conducted from the societal perspective unless strong justification is provided 

for doing otherwise. 

Other perspectives may be included when relevant justification is provided for their 

consideration. 

The study perspective(s) must be explicitly stated (e.g., “The perspective used in this study is 

societal.”) 

5) Methods: intervention strategies 

Intervention strategies must be clearly identified and completely described. A baseline strategy 

must be included. 

Authors must describe all relevant interventions. If the analysis does not include all possible 

interventions, the authors shall provide the rationale for including or excluding them. For 

example, the authors of an analysis of vaccination of adults against pertussis would discuss the 

rationale for inclusion or exclusion of a selective strategy of vaccinating care providers of young 

infants. 

6) Methods: time frame and analytic horizon 

The time frame and analytic horizon for each strategy must be defined clearly, complete with 

appropriate justification (e.g., “We analyzed benefits and costs over a ten year period. The 

analytic horizon was selected because available data suggest that after XX years, vaccine11/13/2007 Page 4 of 9 
ACIP Guidance for Health Economics Studies 

induced immunity may wane to the point where a large proportion of vaccinees would require a 

booster dose to maintain adequate levels of immunity.”). 

7) Methods: the economic model 

a) The analytic method used must be specified (e.g., cost-benefit, cost-effectiveness, or costutility analysis). The summary measure must be defined/ identified. 

b) The basic model used in the analysis must be explained in the text without use of 

mathematical notation. Authors may consider enhancing the written description with a word 

“equation”. If deemed necessary by the authors, an equation, or set of equations, using 

mathematical notation also can be provided in a technical appendix. 

8) Methods: health outcomes of interest 

Health outcomes must be clearly identified (e.g., deaths, hospitalizations, outpatient visits, 

quality adjusted life years). 

Authors must ensure that health outcomes are relevant to the perspective. Although almost any 

health outcome may be relevant for a societal perspective, some health outcomes may be 

irrelevant for any additional perspectives. 

9) Methods: inclusion of epidemiologic models 

When an epidemiologic model is an integral part of a health economics study to be presented to 

the ACIP, the authors shall include a description of the model. The authors must include a 

schematic diagram illustrating the model. Annotation in such a diagram must be done without 

use of mathematical notation. Authors are directed to ensure that such schematic diagrams can be 

readily understood without extensive reading of the main text (i.e., can “stand alone”). 

The authors must state the time step used in the epidemiologic model. “Time step” refers to the 

time associated with the probabilities used in the model. For example, if the authors use 

probabilities of death per unit population per year, then the de facto time step of such a model 

would be one year. Probabilities, and time step, can be in almost any unit of time. The total 

number of time steps must match the time frame of the economic model unless a specific 

explanation is given. 

Authors should note that it is insufficient merely to reference another source (e.g., reference a 

previously published paper). 

10) Methods: Inputs: values and sources 

a) Probabilities: Values and sources must be presented. Authors must make sure that 

probabilities used in the analysis are “reasonable.” 

b) Costs: Values and sources must be presented. The study shall differentiate between direct 

medical, direct non-medical, indirect (i.e., productivity), and intangible costs (included when 

relevant). The year of cost data also must be stated. Authors must make sure that included costs 

are relevant to the stated perspective. 

11/13/2007 Page 5 of 9 
ACIP Guidance for Health Economics Studies 

c) Other inputs: Values and sources of any other inputs (e.g., Quality Adjusted Life Years) must 

be presented. 

Each value used in the model must have at least one clearly identified source. Values that are 

assumed, or calculated (e.g., a residual probability), or the results of expert opinion shall be 

identified as such. 

Authors are advised that the greatest clarity is often achieved by presenting all input values and 

sources in tabular format. Such tables must include sufficient footnotes to enable a reader to 

readily understand the table without extensive reading of the main text (i.e., such tables must 

“stand alone”). 

11) Methods: discounting 

All future costs and benefits shall be discounted to present value. This includes future health 

outcomes (e.g., future lives saved must be discounted to present). 

The discount rate used must be specified, and authors must ensure that the discount rate is 

relevant to the stated perspective. 

12) Methods: sensitivity analyses 

The general goal of sensitivity analyses is to demonstrate how conclusions might change with 

changes in input values. 

Sensitivity analyses, including appropriate threshold analyses, shall be conducted and reported 

on cost and incidence variables. The type of sensitivity analysis conducted must be described. 

Authors are strongly encouraged to conduct sensitivity analyses that allow them to identify 

which variables within the model are most influential in determining the overall results. 

The ranges and sources of the values used in the sensitivity analyses must be clearly reported. 

Authors shall report values used in sensitivity analyses in the same table in which they report 

values of all inputs (see Point 10). 

Univariate sensitivity analyses (altering only one value at a time, and keeping all other values 

fixed at original values) are unlikely to be considered adequate. Authors shall present some form 

of multivariate sensitivity analysis or an explanation as to why such analysis is not presented. 

Authors must use sensitivity analysis ranges that are based on clinically relevant or policy 

relevant cutoffs rather than those based on arbitrary changes in input values (e.g., plus or minus 

10% of initially used values). 

As a form of sensitivity analysis, in addition to presenting results that have been discounted to 

present values (see Point 11), undiscounted costs and benefits may be presented. 

13) Methods: independent replication 

11/13/2007 Page 6 of 9 
ACIP Guidance for Health Economics Studies 

In general, information must be presented that would allow a researcher, with sufficient interest 

and relevant skills, to independently replicate the study.. 

In order to meet such a standard, where necessary, authors may provide additional details in a 

technical appendix. 

RESULTS 

14) Results: summary measures 

Results that answer the study question must be identified. The summary economic measure(s) 

that answers the study question must be presented. 

The summary measure must be appropriate for the perspective used in the study. 

If deemed necessary, authors also may wish to present other results calculated during the 

analysis, such as total number of cases averted. 

15) Results: tables and graphs 

All tables and graphs used to present results must be readily understood without extensive 

reading of the main text (i.e., such graphs and tables can “stand alone”). Authors must include 

footnotes that will help a reader understand each table and graph. 

Tables and graphs must add value to the report. Graphs or tables that contain results not central 

to the summary measure (see Point 16) may be included in a technical appendix. 

Graphs shall be drawn using the standard guidelines for graphical representation of data. 

Detailed guidelines can be found in texts such as Tufte (Tufte ER. The Visual Display of 

Quantitative Information. Graphics Press, Cheshire Connecticut, 1983: pp. 197). Examples of 

such guidelines include: 

1) Pie charts are almost always unacceptable. 

2) Horizontal and vertical grid lines usually are not needed. 

3) For line charts, typically only 4 lines (variables) drawn per chart. 

4) For column and bar charts, typically only 4 bars (variables) included per chart. Avoid 

using “stacked” bars or columns (e.g., a column that has several elements that add to 

100%). 

5) To allow easy printing and copying, graphs and figures should be drawn in grayscale. Color in graphs and figures should be used sparingly, if at all. 

16) Results: sensitivity analyses and influential variables 

Authors shall present their sensitivity analyses in a clearly identified section, complete with 

relevant tables and graphs. 

Authors shall, whenever possible, present a list of “most influential” variables as identified 

through sensitivity analysis (see Point 12). 

DISCUSSION 

11/13/2007 Page 7 of 9 
 

 

 

ACIP Guidance for Health Economics Studies 

17) Discussion: overall 

The discussion section should be more limited than that typically found in a peer-reviewed 

manuscript. 

18) Discussion: limitations 

Study limitations must be discussed. Limitations shall include accuracy of any epidemiologic 

model and input data (see Points 9 and 10). 

Any implicit assumptions, such as an adequate supply of vaccine, should be mentioned. 

19) Discussion: relation to other relevant studies 

Results must be discussed in relation to other similar studies if such studies are available. 

20) Discussion: how results may change 

There must be an explicit discussion, typically drawing from the results of the sensitivity 

analyses, of how results would change if key assumptions or values were to change. 

21) Discussion: no policy implications 

Unless otherwise requested by the relevant WG chair, the report must NOT include a discussion 

of the policy implications of the results and limitations. Because it is CDC’s and ACIP's 

responsibility to make policy interpretations, such discussion will be deleted if included in the 

document distributed for ACIP discussion, unless the WG chair specifically requests it be 

included. 

22) Details not addressed in this guidance 

For further guidance regarding methods and results, researchers may follow the 

recommendations described in the following standard texts (or equivalent): 

1) Prevention Effectiveness: A Guide to Decision Analysis and Economic Evaluation, 2d. Ed. 

Anne Haddix, Steven Teutsch, and Phaedra Corso, editors. (New York: Oxford University 

Press, 2003) 

2) Cost-Effectiveness in Health and Medicine. Joanna Siegel, Louise Russell, Milton Weinstein, 

and Marthe Gold, editors. (New York: Oxford University Press, 1996). 

23) Additional information 

1) Link to the Community Guide Economic Evaluation abstraction form where all of the 

important components of an EE analysis are presented. The form has a quality rating of EEs at 

the end. www.thecommunityguide.org/methods/econ-abs-form.pdf 

2) Many standards for conducting economic evaluations are also on the CDC on-line Economic 

Evaluation course. www.cdc.gov/owcd/EET/Preface/Preface.html 

3) The Guide to Community Preventive Services book, chapter 11 is titled “Interpreting and 

Using Economic Evidence”. 

11/13/2007 Page 8 of 9 
ACIP Guidance for Health Economics Studies 

4) Detsky AS, Laupacis A. Relevance of cost-effectiveness analysis to clinicians and policy 

makers. JAMA. 2007;298:221-4. 

GUIDANCE: format for presentation of slides 

Principles for presenting a health economics study to ACIP: 

After the document has been peer-reviewed internally, and the WG chair and CDC lead staff 

person have reviewed and discussed the contents of the document and the relevant analysis, they 

will decide whether to proceed with inviting a presentation to the ACIP or WG. 

It is realistic to assume that many ACIP members will not have had the opportunity to read the 

document. Further, the ACIP general meetings (but not WG meetings) are open to the public. 

Thus, the presenter of the health economics study should realize that most of the audience at the 

presentation will not be aware of the study, and that the audience members have diverse 

backgrounds. Further, especially for presentations at an ACIP general meeting, it is likely that a 

relatively short amount of time will be allotted for the presentation, typically 20-30 minutes 

including time for questions. A presentation made in the closed setting of a WG teleconference 

or meeting may be longer and more detailed. 

Template slides: 

The attached set of template slides provides an outline of the type of slides and suggested layout 

of each type of slide. The subject matter of the slides in the template is designed so that a 

presenter can be assured of presenting the main facts as they relate to the methods, results and 

limitations of the study. 

11/13/2007 Page 9 of 9