# EXECUTIVE SUMMARY
According to the World Health Organization (WHO), guidelines are "documents that contain recommendations about health interventions, whether they be clinical, public health, or policy recommendations." Recommendations provide information about what policy makers, health care providers, or patients should do. Recommendations imply choices between different interventions that have an impact on health and that have ramifications for the use of resources."
The Centers for Disease Control and Prevention (CDC) is a leader in developing public health guidelines.
Ideally, CDC guidelines are developed by a group of multidisciplinary stakeholders, based on evidence from systematic reviews and expert judgment, and include an assessment of benefits and harms. CDC guidelines should be clear, valid, transparent, reliable, accurate, and applicable. Although the term "recommendations" may be used more narrowly to identify specific actions and the term "guidelines" may more broadly refer to the umbrella under which multiple recommendations for action are provided, we use the terms guidelines and recommendations interchangeably in this document. Some guidelines are developed by federally-chartered advisory committees like the Advisory Committee on Immunization Practices (ACIP) and the Healthcare Infection Control Practices Advisory Committee (HICPAC) and are subsequently accepted and issued by CDC. Other guidelines are developed under the initiative of CDC programs in collaboration with experts in the field. The most frequent CDC publication venue is the Morbidity and Mortality Weekly Report (MMWR), but other outlets mechanisms such as peerreviewed journals and agency publications are also used.
Although CDC guidelines have garnered wide acceptance among stakeholders and partners, the rationale and development process is not always clear to the user. Because of the breadth of public health topics and audiences, adopting a single approach to developing guidelines at CDC poses a significant challenge.
As a result of this challenge, we have refrained from recommending a single methodological approach for development and reporting of all CDC guidelines (although guidelines authors are encouraged to familiarize themselves with methods already used at CDC). Yet CDC guidelines should meet certain standards. This primer provides development and reporting standards to improve the transparency, validity, and reliability, of CDC guidelines and recommendations. Development standards include:
- Identify the public health problem on a given topic, need for new or revised guidelines, and required resources for development - Involve knowledgeable, impartial, and representative participants in the process - Minimize and disclose competing interests among participants - Involve and consult affected organizations
- Include a written charge to the development work group
- Obtain the evidence preferably using systematic reviews
- Use evidence-based frameworks and decision-making rules
- Involve an explicit scientific quality and policy control process
The above standards, along with reporting standards, are provided and covered in detail in Section 9. By following development and reporting standards, guidelines developers can improve the use of evidence, minimize bias, and enhance quality and consistency in reporting. Although occasionally some of these standards cannot be followed, most CDC guidelines will at least fulfill the intent of the standards. The reader should be able to understand the methods used to develop the guidelines, how the evidence was selected and assessed, and how the evidence led to the recommendations. Meeting these standards will also support CDC programs in developing guidelines that can be trusted by the public health community and the public. By improving the rigor and transparency of methods used to develop CDC's guidelines, criteria-based guideline clearinghouses will more readily adopt and publish CDC guidance, enhancing visibility and use.

#CDC Guidelines and Recommendations Work Group July 2012 INTRODUCTION
"Guidelines are documents that contain recommendations about health interventions whether they be clinical, public health, or policy recommendations." Recommendations provide information about what policy makers, health care providers, or patients should do.
Recommendations imply choices between different interventions that have an impact on health and that have ramifications for the use of resources."  The Centers for Disease Control and Prevention (CDC) is a leader in developing public health guidelines. Ideally, CDC guidelines are developed by a group of multidisciplinary stakeholders, are based on evidence from systematic reviews and expert judgment, and include an assessment of benefits and harms. CDC guidelines should be clear, valid, transparent, reliable, and applicable.

# Purpose
Guidelines developed at CDC cover a wide range of disciplines and needs. Guidelines may go from providing recommendations for interventions to increase physical activity to the treatment of pandemic influenza. Often, CDC guidelines address multiple stakeholders and provide recommendations for individuals, groups, and communities. Audiences may include health department staff, coordinating partner services, physicians looking for information on new treatments, labs looking for screening tests, and individuals looking for prevention programs and services. Recommendations in CDC guidelines may cover surveillance practices, program implementation, or even policy interventions. Nuances exist associated with the selection of a public health topic. For example, a particular public health topic selected for guideline development may require the use of federally-chartered advisory committees. The selection of a public health topic will affect the type and extent of the evidence of effectiveness. Or, addressing a public health topic may include policy-based interventions that could be socially or politically sensitive in nature.
Diverse audiences and the nuances of public health topics pose challenges to recommending a common approach for developing and reporting guidelines. Content and format may vary with the needs of each group of users. Additionally, diverse audiences may have access to guidelines of other organizations containing recommendations that may be inconsistent with CDC recommendations. This diversity in audiences, topics, and communication formats poses particular challenges for ensuring the clarity and acceptance of CDC guidelines. Because of all these considerations, a standardized approach to guideline development is not recommended in this primer. Instead, the primer provides a series of critical elements and standards that can be used in CDC guideline development.

# Audience
This primer is for CDC staff, contract review teams, CDC federal advisory committees, and CDC partners and stakeholders. Guideline developers sponsored or co-sponsored by CDC will find the primer particularly useful, as will anyone involved in the clearance or publication of guidelines.
Use of minimum standards in guideline development and reporting will improve guideline quality and facilitate the clearance and publication processes.

# Scope
Because of the difficulty in recommending a standardized approach for CDC guideline development, this primer is not a how-to manual. Instead, it covers elements common to the development of all guidelines, shows how standards can be integrated into the development process, and provides critical ways to improve CDC guidelines. This primer supplements the information outlined in CDC Guidelines: Improving the Quality (1996).  2.

# GUIDELINES, POLICIES, AND OTHER CONSIDERATIONS
Developing guidelines entails making many front-end decisions that, in the end, will save time and money. Early considerations about the need, evidence, and current methods for guideline development can assist developers in clarifying terms, concepts, and principles early on in the development process. We suggest guideline developers consider the following questions before beginning development of guidelines:
- What are public health guidelines?
- What are the differences between guidelines and policies?
- Is there a need for new guidelines?
- What type of evidence is available?
- Who develops guidelines?
Answers to these questions will guide decision-making at the onset of guideline development.
This section will remind developers of organizations that can be sources of technical knowledge, methods, procedures, tool kits, and software.

# Guidelines, Recommendations, and Guidance
Although the term "recommendations" may be used more narrowly to identify specific actions and the term "guidelines" may be used more broadly to refer to the umbrella under which multiple recommendations are provided, we use the terms guidelines and recommendations interchangeably in this document. Typically, public health guidelines are developed by a group of multidisciplinary stakeholders, use evidence from systematic reviews and expert judgment, and include an assessment of benefits and harms.  Guidelines should be clear, valid, transparent, reliable, and applicable. These terms are defined in Section 10.
Although the term "guidance" is sometimes used to indicate a "less-direct type of advice," guidelines authors should be aware that the term "guidance document" is used by regulatory agencies for specific types of documents and that the Office of Management and Budget (OMB) has issued policies and procedures for developing guidance documents.  Therefore, be aware that the use of the term "guidance" could lead to confusion with regulatory guidance documents, and consider using other terms.

# Types of Guidelines
Guidelines cover the full range of public health areas represented at CDC. Guidelines may be developed across the 10 essential public health services.  For example, guidelines might be developed for monitoring health status, mobilizing community partnerships, selecting and implementing programs and policies, and evaluating health services. The target audience can be wide and varied, including, for example, clinicians, public health professionals, or policy makers.
The evidence base for guidelines may be robust or sparse. Sometimes CDC must develop guidelines when evidence is lacking. In such cases, guidelines can be developed based on expert judgment and historical knowledge about the intervention in similar or different settings.
Expert judgment may be based on experience and observations and on extrapolation of evidence from different settings or similar interventions.
When guidelines on a specific topic are first developed, they may be called "initial" guidelines.
Initial guidelines may be limited in scope. For example, a guideline might focus on recognition and initial assessment and less on differential diagnosis, management, or treatment issues after diagnosis. The guideline to recognize and assess early Alzheimer's disease produced by the Agency for Health Care Research and Quality (AHRQ) is an example of an initial guideline.  "Comprehensive" or final guidelines (also called "full guidelines" by WHO)  are broader in scope and are developed by a wide range of stakeholders, usually with participation from several agencies, academia, nongovernmental organizations, and communities. Such guidelines provide full consideration of a disease, condition, environmental emergency, or natural disaster.
Comprehensive guidelines usually include many aspects related to the disease or condition including surveillance, detection, control, treatment, and prevention. An example of comprehensive guidelines is the Guidelines for Foodborne Disease Outbreak Response, published by the Council to Improve Foodborne Outbreak Response.  "Interim" guidelines are those developed in response to emergencies or to rapid increases in cases of a disease or condition. These guidelines are labeled "interim" to clarify that such guidelines were developed using less thorough processes or were based on tentative or emerging data. WHO refers to such guidelines as "rapid" guidelines and their development period may vary between a few weeks and a couple of months.  An example of an interim guideline is the CDC guideline for the use of face masks and respirators during an influenza pandemic.  Initial and interim guidelines are often updated when new evidence is available.
"Supplemental" guidelines update topics considered in previous guidelines or cover new topics.
These guidelines-which may be initial, interim, or comprehensive-supplement previous guidelines with new or revised information. Examples of supplemental guidelines are the CDC updated guidelines for management of occupational exposures to HIV.

# Guidelines and Policies
Guidelines, unlike some types of policies, are not mandatory. In health care and public health, guidelines are not meant to enforce but rather to recommend programs or practices based on the best evidence available. Often, however, CDC and others' guidelines become "the standards of practice," unintentionally acquiring the force of policy. The adoption of a set of guidelines can affect an entire organization. Whatever the organization (for-profits, nonprofits, public health departments, or federal government agencies), guidelines provide information for decisionmaking. Implementation of guidelines might improve organization effectiveness and efficiency.
Policies, on the other hand, are laws, regulations, procedures, administrative actions, incentives, disincentives (e.g. taxes, fines, fees, or other pecuniary or non-pecuniary penalties) or voluntary practices of government and other institutions. Many policies are enforceable and their implementation is frequently reflected in resource allocations. Policies also can be maps of actions that guide an organization or group in decision-making. Policies frequently address the impacts of decision-making on population health, the society, the economy, and the environment.
Mandatory policies are similar to executive orders or decrees.
 Occasionally, policy making follows the issuing of guidelines. When this occurs, the policies function like road maps and are essential to implementing the guidelines. Increasingly, policies are being used as tools to improve public health efforts. Although guidelines typically support programmatic interventions or health practices, guidelines can also aid the use of policy to improve public health efforts (e.g., guidelines for policies on nutrition standards for food in schools; guidelines for policy on alcohol-impaired driving such as ignition interlocks, expanded use of sobriety checkpoints, and minimum drinking age).

# Establishing the Need for New Guidelines
Guideline developers should make sure that new guidelines are needed before guideline planning is started. Establishing the need for new guidelines entails assessing the public health need for systematic advice and identifying the existence of current or past guidelines on the same topic covering the same population and settings. This process also entails determining the existence of guidelines that need to be updated because of new research findings. The following questions may help guideline developers assess the need for new guidelines:
- Are guidelines needed to sufficiently address the public health threat under consideration?
- Why are guidelines needed now?
- Are there new interventions that require new guidelines?
- Have there been changes in the disease or pattern of the disease?
- Have there been changes in the guideline audience?
- Is there a new role for the CDC?
- Is there sufficient new evidence available to develop the guidelines?
- Are there current guidelines on the same topic published by other authorities that might be used instead?
- If other guidelines exist, do they apply to the same target cohort, settings, and demographics?
- Are there sufficient funding and resources available to support the planning, development, and dissemination of these guidelines?

# Use of Evidence in Guidelines
Guidelines should be developed using the best available evidence of effectiveness of preselected outcomes. "Evidence" is knowledge gained from scientific research that is interpreted in the context of public health practitioner experience, conditions of intervention implementation, and experience of the population targeted.  When research evidence is available, systematic reviews can inform guideline development. Different types of research evidence exist for inclusion in systematic reviews including evidence from randomized controlled trials, observational studies (e.g., quasi-experimental and time-series studies), and qualitative studies.
Keep in mind, however, that guidelines extend beyond systematic reviews in important ways.
Although systematic reviews provide information about the scientific evidence behind an intervention ("this works"), guideline developers combine this information with expert judgment, information about harms and benefits, economic efficiency, and values and preferences to inform public health or clinical practice in guideline form ("this should be done"). The quality and strength of the evidence varies across areas of public health. Sometimes, practitioners in the field are needed to provide key pieces of information that can't be culled from the empirical literature. A practitioner perspective can be critical to understanding how interventions are carried out and which populations are targeted.

# Organizations That Develop Guidelines
Many organizations produce guidelines, develop databases of existing guidelines, or provide methods, tools, and training for developing guidelines. Some organizations, such as the AHRQ National Guidelines Clearinghouse, act as repositories of national guidelines. The clearinghouse has registered 2,373 clinical practice guidelines to date, produced by 285 organizations.  Other organizations, like the Cochrane Collaboration and Campbell Collaboration, also act as repositories of their own accredited systematic reviews but also offer methods and electronic tools for conducting systematic reviews by independent groups in academia or government.  The at the University of York (CRD) develop guidelines internally or under contract.  The
Canadian Medical Association provides a database of guidelines with a summary listing methods and development processes.  In the United States, the U.S. Preventive Services Task Force (USPSTF) and The Community Preventive Services Task Force (CPSTF) develop recommendations under the umbrella authorities known as the Guide to Clinical Preventive Services (Clinical Guide) and Guide to Community Preventive Services (Community Guide),
respectively.  The Institute of Medicine (IOM) has published standards for developing clinical practice guidelines.  CDC produces guidelines developed at the program level. Such guidelines are often developed in consultation with field experts and are typically published in the MMWR.  CDC also works with advisory committees that develop guidelines like the ACIP and HICPAC. 

# GUIDELINE DEVELOPMENT PLANNING
Guideline planning is critical to setting the stage for methods, processes, and procedures used during the development phase. Early planning will make the development process more fluid and the end product more transparent. Planning activities include: a) identifying the audience, b) identifying contributors and their roles, c) deciding on evidence and outcomes, d) developing the guideline logic framework, e) planning for guideline dissemination, implementation, and evaluation, and g) planning for guideline updates.

# Identifying the Audience
Public health guidelines often have diverse audiences (e.g., practitioners, policy makers, health care businesses, government agencies). Umbrella guidelines may try to balance information needs for these audiences. Audiences can be identified by addressing the following questions:
- For whom are the guidelines or recommendations being developed?
- To whom are the guidelines and recommendations being addressed?
- To what extent are these guidelines going to assist them?
Companion or support materials may further distill and interpret guideline content for specific audiences. For this reason, developers should distinguish between the actual guidelines and the support materials that will reference the guidelines. Supporting materials might include training materials, fact sheets, applications, pocket cards, algorithms, etc. Supporting materials differ from actual guidelines and require various development and dissemination methods. Developers therefore need to decide beforehand whether supporting materials will be part of the guideline development process or they will be developed after the primary guideline document is complete.

# Identifying Contributors and Their Roles
Major efforts to develop guidelines typically involve at least two distinct groups: a guideline steering committee (GSC) and a technical development group (TDG). Other groups might be needed, such as a dissemination group. Typically, the GSC will be comprised of CDC personnel.
The function of the GSC is overseeing the guideline development process. GSC functions have been adapted from those proposed by NICE and might include:  - Select members of the TDG (see below)
- Draft standard operating procedures for the TDG
- Define the goals and elements of the proposed guidelines
- Approve the methods to synthesize the evidence and develop the recommendations - Determine the sources of evidence
- Conduct a systematic search for evidence
- Select and review the evidence
- Summarize the evidence and specific outcomes
- Assess the quality of the evidence
- Present to the GSC the summary of the evidence
- Provide the GSC with a preliminary assessment of the recommendations' strength
- Provide the GSC with a preliminary draft of recommendations to the GSC
- Draft the guidelines with members of the GSC
- Discuss and incorporate suggestions from external reviewers
- Write the final version of the guidelines If the guidelines are being developed with the help of other authorities and task forces, members of these groups might participate in the GSC and provide input during the development process.
Stakeholders may also participate in the development of the guidelines through the GSC.
Stakeholders are groups whose activities could be affected by implementation of the guidelines or who have legitimate reasons for providing input. Inclusion of stakeholders in developing guidelines will ensure that a) the right issues are identified and b) there is early buy-in and a better chance for early adoption of the guidelines. Inclusion of liaison groups will ensure that guidelines address the needs of the collaborating agencies and can increase adoption among communities, patients, and individuals in at-risk groups. Liaison groups may include representatives from affected populations, patients, or at-risk groups. Frequently, CDC collaborates with international organizations or Ministers of Health in developing joint guidelines and recommendations. For joint guideline documents, CDC developers should work with collaborators to encourage the use of evidence-based frameworks in developing guidelines.
In developing a panel with outside members, whether they are stakeholders, liaisons, subject matter experts or members of international organizations, one might unknowingly construct a committee that is subject to the regulations of the Federal Advisory Committee Act (FACA).
FACA regulations define how federal advisory committees legally operate and ensure that advice is objective and accessible to the public. FACA committees have fixed membership, organizational structure, and comprise members other than full-or part-time employees. FACA outlines requirements for developing a charter and membership, conducting public meetings, issuing statements, using federal register announcements, keeping minutes, and documenting decisions.  CDC guideline developers are encouraged to review regulations pertinent to the development and use of federal advisory committees.

# Protecting Scientific Integrity by Minimizing the Influence of Personal Interests
Users of guidelines and recommendations need to feel confident that those participating in the development process were not unduly influenced by personal interests. Minimizing competing interests among members of steering committees and technical groups improves guideline acceptability, credibility, and scientific rigor. These interests might be financial, intellectual or professional.  For example, competing financial interests might include research support, stock holdings, or employment at organizations affected by the guidelines. Intellectual and professional interests might include authorship of studies or provision of expert opinion publicly or in testimony related to the guidelines topic.
Guidelines developers should assess whether participants' interests might influence their consideration of the scientific evidence or other factors that can influence the recommendations under consideration. CDC guideline developers should make every effort to either eliminate or manage financial, intellectual, or professional interests that compete with the goals of producing an evidence-based guideline. Developers should disclose in the guideline document the presence of financial, intellectual or professional interests. For example, a participant with a competing interest might be excluded from participating in the development of the final recommendation statement. The GSC might also choose to assign participants with competing interests a limited role, perhaps by excluding them from developing or approving recommendations. Participants should be chosen to balance interests in those cases where competing interests cannot be eliminated because it is desirable to have a range of opinions represented. For example, if a member of a systematic review team is the author of a study included in the review, the Cochrane Collaboration requires the inclusion of other team members who were not involved in the study in question. If employees of a pharmaceutical company or medical device manufacturer are members of a team conducting a review on a product produced by that company or manufacturer, extra effort should be made to make the review team multidisciplinary. The majority of the team members should have no employment or financial relationship to that particular company. Those with a direct financial interest in an intervention may consider abstaining from becoming members of the review team conducting a systematic review of that intervention.  Special rules about conflicts of interest pertain to federal employees and "special government

# Deciding on Evidence and Outcomes
Core issues in determining the types of evidence to use in the development of recommendations hinges upon several considerations. Below are some considerations adapted from those suggested by NICE :
- What is the most appropriate type of evidence to answer the question?
- How can the most relevant evidence (published and unpublished) be identified?
- How can the quality and applicability of evidence be assessed?
- How can evidence from different kinds of research be synthesized?
- How can quantitative and qualitative data be combined?
Evidence-based approaches vary according to the sources used in gathering the evidence.
There are many sources of evidence. The following is a listing of some of the most commonly used sources, combined or standing alone:
- Systematic reviews of research studies
- Meta-analyses of research studies
- Econometric analyses
- Decision analyses
- Review of published systematic reviews
- Review of published meta-analyses
- Nonsystematic review of research studies

# Expert judgments
Once the TDG decides on approaches to obtain the evidence, it needs to identify the outcomes of interest. The identification of outcomes is topic specific. Outcomes can be prioritized, rated by the TDG, and presented to the GSC for review.  Logic frameworks can be used to depict the links between the health problem and the interventions and between the interventions and outcomes of interest.

# Developing the Logic Framework
Logic frameworks (also known as conceptual or analytical frameworks) guide the assessment of an intervention's effectiveness for which guidelines are being developed. approach.  Frameworks assist in addressing the following questions:
- What interventions or professional practices are to be included in the guideline?
- Which health outcomes (ultimate or intermediate) are expected effects of the intervention?
- What types of evidence of effectiveness of the interventions are being considered?
- How will the quality, quantity, and relevance of the evidence of effectiveness be assessed?
- How will the strength of evidence of effectiveness and the strength of recommendation be related?
- What assumptions about causal relationships of interventions and health outcomes will be accepted without direct scientific proof?
- What form of exposition (narrative or graphic-flow chart, influence diagram, decision tree, or clinical algorithm) is most appropriate for describing the logic framework?
The logic framework also helps guideline developers compose a persuasive supporting rationale for the interventions included in the guidelines. The logic framework links the interventions under consideration with the estimated effect, benefits, and harms. We recommend the inclusion of the logic framework in graphic format in the published version of the document. For examples of logic frameworks included in published recommendations, see the publications of the Community Guide. 

# Planning for Dissemination, Implementation, and Evaluation
At this point in the development process, the GSC needs to decide whether the dissemination and translation of the guidelines will be done using "in-house" personnel or contractors. Those responsible for disseminating and translating the guidelines need to develop a plan that includes the following:
- Target audiences
- Venues (hard copies or electronic)
- Peer-reviewed journals targeted
- Professional organizations targeted
- Development of focus groups (quantity, audiences, etc.)
- Different formats according to audience needs
- Activities and respective leads
- Possible training needs
- Timetable
- Resource needs and costs
- Use of plain language and audience-appropriate visuals that match the text
- Need for translation of information into languages other than English
If the GSC decides that electronic dissemination is the best option, the dissemination and translation group needs to determine the facilitators and barriers for the dissemination and identify the public's needs for evidence-based scientific information on the Internet. If the guidelines will be posted online, some organizations recommend developing online tools to evaluate the website.
Implementation and impact evaluation of guidelines are not covered in detail in this primer.
However, and because implementation and impact evaluation may influence the development of the guidelines, we encourage developers to consider during the planning phase the main issues that may arise after the release of the guidelines. A systematic review of the effectiveness of various interventions designed to assist in the implementation of research findings conducted by Bero et al in 1998 found that reminders alone or combined with audit and feedback, local consensus processes, and marketing and interactive training were best suited for consistently effective interventions. Audit and feedback alone, local leaders (such as practitioners), local consensus processes and patient-mediated interventions aimed at changing the behavior of health care providers were best suited for interventions of variable effectiveness. Educational materials and educational meetings were best suited for interventions with little or not effect.  The use of specific strategies to implement research-based guidelines and recommendations seems to be necessary to ensure that practices change. The most successful are intensive efforts to alter practice. Bero et al contend the choice of intervention should be guided by the evidence on the effectiveness of dissemination and implementation strategies, the characteristics of the message, the recognition of external barriers to change, and the preparedness of practitioners to change.  Another activity in the entire cycle of guideline planning, production, dissemination and implementation is the evaluation of impact. Impact evaluation methods are not explored in detail in this primer. However, like in the case of dissemination and implementation, developers need to keep in mind that the initial planning for impact evaluation may affect the development of the guidelines. Impact research in developing countries shows that several factors increase the impact of practice guidelines like involvement of the end-users in developing, launching, and introducing the guidelines; multiple training modalities; providing feedback to prescribers on their prescription practices in relation to guidelines; and effective monitoring and supervision.  The main elements in evaluating the impact of guidelines and recommendations follow:
- Assessing awareness
- Assessing uptake
- Assessing health impact
- Assessing other impact (e.g., economic, social, environmental)

# Planning for Updates
We recommend the planning phase address dates and formats for updating the guidelines. As for dates, guideline developers may choose to indicate a specific time frame for retiring the guidelines either because new knowledge becomes available or technological advances render the existing guidelines obsolete. Guideline developers may also elect to indicate a time frame for updating the guidelines, either as a one-time event or periodically. As for format, guideline developers may choose to indicate that guidelines will be updated in their entirety or on specific sections. In that case, guideline developers may opt for publishing updates in summary and appendix form. Indicating retiring dates and publication formats for subsequent updated guidelines will make users prone to seeking updates and decrease the probability that practitioners will continue using out-of-date information or practices.

# DEVELOPING GUIDELINES AND RECOMMENDATIONS
Developing guidelines encompasses diverse processes and activities. This section mainly discusses processes for guidelines and recommendations based on evidence obtained from systematic reviews. The main processes involved in systematic reviews are 1) gathering the evidence, 2) abstracting and assessing the evidence, 3) summarizing findings, and 4)
interpreting the evidence to develop the recommendations. Activities within each of the main classifications are included and explained below. If other sources of evidence are used, like expert judgment or surveillance data, the methods used to collect and assess the evidence need to be described in detail in the guideline document. Obtaining evidence in the absence of data is covered briefly in section 4.5. Descriptions of methods to gather information from other sources, including expert opinion, can be found in the literature. 

# Gathering the Evidence
Once decisions are made on the type of evidence, sources for that evidence, and outcomes of interest, the next step of conducting a systematic review is gathering the evidence. The main activities included in gathering the evidence are 1) developing a search protocol and inclusion criteria, 2) conducting a literature search, and 3) selecting the relevant literature according to the pre-determined criteria.

# Developing the Search Protocol
Developing the search protocol involves determining where and how to look for evidence. The
following are examples of sources of evidence:
- Electronic and manual searches of published literature (peer or non-peer reviewed)
- Electronic and manual searches of unpublished literature (e.g., reports, proceedings)
- Searches of published or unpublished surveillance data Studies included in the search can be guided by the public health question of interest. Search protocols describe the selection of search terms (i.e., keywords), the sources and databases used for the search, the earliest date from which studies will be sought, and the inclusion criteria for those studies that will form the body of literature selected for screening.

# Conducting the Literature Search
The search is guided by the search protocol. The types of databases listed in the search protocol will depend greatly on the topic. Most searches, however, include readily available databases.
The development of search terms and the locating of topic-specific databases can pose challenges, which can be mitigated by consulting those who are more experienced in the discipline. CDC's Library Services staff provides assistance when formulating and conducting literature searches. Different search platforms may yield different results, especially for recent data. For example, a search of OVID for search term "X" may yield a slightly different citation list than a search of PubMed for the same search term.
The following databases, accessible via CDC's Library Services, are the most commonly used for systematic literature reviews:  - PubMed
- Medline (OVID)
- Web of Knowledge
- PsycINFO - Cochrane Library - Campbell Library - Embase - CINAHL - Education Resources Information Center (ERIC) - CAB Direct - EconLit
If the body of evidence includes systematic reviews conducted by others, we recommend searching the following databases: AHRQ National Guidelines Clearinghouse, Cochrane Collaboration, Campbell Collaboration, CEBM, NICE, CRD, Clinical Guide, and Community
Guide. 

# Selecting Relevant Literature
The selection of relevant literature involves screening the search output according to the predetermined inclusion criteria. Those studies that meet the inclusion criteria are selected using a two-stage screening approach:
- Titles and abstracts are screened independently by two reviewers
- Full papers are screened independently by two reviewers Any difference of opinion should be resolved between the two reviewers or with the help of a third reviewer under the guidance of the technical lead. The selection process needs to be fully documented. Flow charts may be used to illustrate the inclusion and exclusion process. Software programs can be used to manage search results, but the use of software for this purpose should be explained in the final report.

# Abstracting and Assessing the Evidence
To ensure consistency, reduce bias, and improve validity and reliability, some organizations have developed standardized procedures to abstract and assess the evidence. Most of these tools are published online. For details on these instruments, consult the procedures of the Cochrane Collaboration  and the Clinical Guide or the Community Guide.  Evidence from new systematic reviews can be presented along with evidence from existing systematic reviews, meta-analyses, or review of meta-analyses. Developers need to keep in mind that the use of evidence from existing systematic reviews follows different criteria than the use of evidence from systematic reviews of single studies. For a critical assessment of existing systematic reviews, consult the CEMB.

# Assessing Evidence Quality
Evidence quality is defined as the extent to which one can be confident that an estimate of effect or association is real.
 Assessing evidence quality is important, as it is one of the factors that affect the strength of the recommendation. Methods chosen to assess evidence quality will likely be determined by the evidence-based approach selected for guideline development. These methods vary among guideline authorities like the ones developed by the Clinical Guide, Community Guide, and the Grading of Recommendations Assessment, Development and Evaluation (GRADE) authorities.  Evidence quality assessment usually occurs in two phases: assessing individual study quality (often addressed in part by using study inclusion and exclusion criteria, as discussed in Section 4.1.1), and assessing the quality of the body of evidence. Determining the quality of the body of evidence may include assessing study designs and their limitations, as well as assessing internal validity, generalizability, consistency of results, effect size, confidence intervals, and applicability of the evidence to the target population and setting.
For example, the Clinical Guide appraises evidence quality on the factors noted above and rates the overall certainty regarding the quality of the full body of evidence as high, moderate, or low.
To illustrate, high-quality evidence generally includes consistent results from well-designed, wellconducted studies in representative primary care populations that assess the effects of the preventive service on health outcomes; the conclusion is therefore unlikely to be strongly affected by the results of future studies. In contrast, low-quality evidence signals that the evidence is insufficient to assess effects on health outcomes because of a limited number of studies, flaws in design, gaps in the chain of evidence, etc. The Clinical Guide uses evidence quality to estimate the expected magnitude of benefits, harms, and net benefits (benefits minus harms) that would result from widespread implementation of the preventive service. An indication of evidence certainty is provided within the recommendation statement. More detailed information on how the Clinical Guide rates evidence quality and develops recommendation statements can be found on its website.  Similarly, the Community Guide rates the quality of individual studies based on potential threats to internal validity, such as description of the intervention implementation and potential for biases in sampling, measurement, analysis, recruitment, and interpretation of results.  After individual study quality is assessed, the quality of the full body of evidence is characterized based on execution of the study designs, design suitability, number of studies, consistency of the results, and whether expert judgment was used or not. Final categories of evidence quality include "strong," "sufficient," "insufficient," or "based on expert opinion." Evidence quality directly relates to the Community Guide recommendation statements. For example, when the evidence is strong, the intervention is recommended based on strong evidence of effectiveness; when there is insufficient evidence, it is determined that available studies do not provide sufficient evidence to determine the effectiveness of the intervention.
An alternate approach to rating evidence quality comes from the GRADE Working Group.
This approach ranks evidence quality on several factors including study design, limitation of the studies in execution and analyses, consistency of results across studies, applicability of the evidence to the populations and settings proposed for the intervention, and precision in the estimate of effect.  Study design is the first step for rating evidence quality. Studies are broadly classified into either randomized controlled trials (RCTs) or observational. Observational studies include time-series, cohort studies, case-control studies, case series, and case reports.
Studies are given a numerical quality rating based on their design (e.g., a rating of "1" for RCTs, a rating of "3" for observational studies.) This numerical quality rating can be rated down based on risk of bias (i.e., study limitations that threaten internal validity), publication bias, imprecision, inconsistency, or indirectness. The quality rating can be rated up based on the effect size, doseresponse relationship, or if plausible confounders would have reduced a demonstrated effect.
Based on a review of the evidence quality for all the studies, the full body of evidence is given a rating of either high (very confident that the true effect lies close to that of the estimate of the effect), moderate (moderate confidence in the effect estimate), low (limited confidence in the effect estimate), or very low (very little confidence in the effect estimate).  A notable characteristic of the GRADE approach is the separation of the rating of evidence quality from the process by which the recommendation is determined. That is, evidence quality does not always equate with the strength of the recommendation. Although high-quality evidence usually leads to a strong recommendation, this is not always the case. On the other hand, low-quality evidence can sometimes lead to strong recommendations depending on factors such as values, preferences, and costs. For information on how evidence quality ratings inform recommendations, see Section 4.4.

# Summarizing Findings
The types of studies reviewed, methods used to summarize the evidence, and available statistics will determine the best ways for depicting the results. The summary of findings typically includes a narrative description of the findings along with tables or graphics depicting the outcomes of interest. Typically, findings from quantitative studies are presented in summary tables that describe the characteristics of included studies, sample sizes, measures of prevalence or health burden, effect sizes, and confidence intervals. Figures such as forest plots of effect size or risk of bias plots can be used to display the data. In addition to data that speaks to effectiveness of a particular public health strategy, developers should consider summarizing data on feasibility, economic efficiency, acceptability, risks, and unintended consequences in narrative or graphic form.
Findings from qualitative studies that are included in the body of evidence should also be reported in text, table, or graphic form. Qualitative data might be reported within intervention studies and address questions related to the effectiveness of a public health intervention (e.g., to
understand intervention feasibility and appropriateness; heterogeneity of findings; and values, preferences, and experiences of practitioners and intervention recipients). Qualitative data might also be collected directly by guideline developers through interviews with researchers or practitioners in the field. Also, qualitative data may be obtained among others from program evaluations, gray literature, opinion polls, and policy analyses. Systematic review authorities and academia have developed frameworks and tools for synthesizing qualitative data.  Regardless of the methods used, or whether the data are coded from existing research studies or collected directly by guideline developers, the approach to summarizing the findings should be as transparent as possible. Formats may vary, as long as appropriate findings are summarized thoroughly in the guideline document. 

# Interpreting Evidence and Developing Recommendations
Developing evidence-based recommendations involves inductive reasoning when it is derived from evidence, deductive reasoning when it is drawn from theory or methodological principles, and inferential reasoning when it involves moving from certainty to uncertainty about what would happen if the recommendation is implemented.  Even in those instances when logic frameworks and systematic processes are followed, interpreting and translating evidence to recommendations can be surprisingly difficult. Developing recommendations is influenced by value judgments, policy considerations, and assumptions about a variety of factors, even after a careful review of the evidence. For example, a group that is concerned with feasibility considerations may come to a different set of recommendations than a group that is focused more narrowly on health outcomes, even if both groups are working from the same body of evidence. When considering the same body of evidence, a single specialty group may reach different conclusions than a multidisciplinary group in those cases where the specialty group is biased in favor of implementing interventions in which it has vested interest. Recommendations with higher validity may be produced using multidisciplinary groups where individual biases might be better balanced.
The GRADE approach takes these factors into account when determining the direction and strength of recommendations. Recommendations are developed taking into account: 1) quality of the evidence (determined when assessing the results of the systematic review), 2) benefits vs.
harms (by examining the health outcomes and legal and ethical considerations), 3) values and preferences of the target audience, and 4) resources (cost implications including feasibility and infrastructure requirements as well as cost-effectiveness analysis). By reviewing these four factors, the TDG determines the direction (i.e., for or against) and the strength (i.e., strong or weak/conditional) of the recommendation. It is important that factors used to develop the recommendation be explicitly recognized and reported.
The usefulness of recommendations might be enhanced when they possess the following characteristics, adapted from those provided by NICE:  - Are informed by the most appropriate and available evidence
- Are set within a framework that acknowledges a range of social judgment
- Take into account relevant theories of public health and behavioral change
- Reflect the views and experiences of both those being advised to take action and the people who might be affected by that action
- Are clear
- Are practical Furthermore, developing guidelines includes considering all sources of evidence, including comments from committee discussions and expert judgment. It is important that the processes used for incorporating expert judgment be deliberately considered and be as explicit as possible.

# Developing Guidelines with Limited Data
On occasion, the rigorous research and findings needed for inclusion in the body of evidence are either lacking or limited. In such cases, guidelines may be based on indirect evidence, practitioner experience, and contextual knowledge. Guidelines based on empirical and contextual knowledge are likely more susceptible to bias and self-interest than guidelines based on direct research evidence. Therefore, the methods used to collect and assess information from experiential and contextual knowledge need to be made as explicit as possible.
The complete lack of evidence on a public health intervention, action, or practice is rare. Most often, the information collected to assess public health burden (e.g., from surveillance data), the preferences of practitioners and communities (e.g., from expert opinion or solicitation of patient experience), and the context within which a public health practice is implemented (e.g., from case reports) may be accessed indirectly through the gray literature or drawn from expert opinion. Developers need to report on the nature and source of the evidence when that evidence was obtained from indirect sources. Developers also need to establish clear links between the indirect evidence and the public health problem addressed by the guidelines. Developers can use logic models to highlight how the pieces of evidence link to inform the recommendations.
When this evidence is synthesized relying heavily on expert opinion, the best methods for identifying experts (or documents based on expert judgment) and developing consensus opinion statements need to be used and reported. Minimizing bias is essential when obtaining expert opinion and can be accomplished by selecting appropriate methods to gather and interpret the information (e.g., through structured consensus methods such as the Delphi method, nominal group process, or the Glaser method). 

# ECONOMIC CONSIDERATIONS
The ways economic information is considered and incorporated in guidelines vary among guideline authorities. In the Community Guide for example, the CPSTF reviews the economic efficiency of interventions that it recommends based on strong or sufficient evidence of effectiveness. According to S. Chattopadhyay (personal communication, April 13, 2012), economic evidence is commonly incorporated in the rationale statement of the Task Force findings. However, for interventions that are particularly expensive to implement, such as using home visits to increase vaccination coverage for children or closing schools to control an influenza pandemic, the economic information is also noted in the recommendation statement of the Task Force's findings. When NICE develops recommendations, both clinical effectiveness and cost effectiveness are discussed. If there is strong evidence that one clinical strategy is both more effective and less costly, clearly this strategy is recommended for the target population.
However, when one strategy is more effective but also more costly, then the magnitude of the cost-effectiveness measure is compared to the threshold used by NICE (see below).  The GRADE approach considers cost as one of the factors to be considered in the development of the recommendations.  One of the main concerns when considering economic information is that resources be chosen to obtain the greatest health improvement or, in other words, whether the intervention is efficient in economic terms. Economic efficiency influences the selection of an intervention, and this selection in turn affects the combination and quantity of resources used in implementing the intervention. Economic efficiency is informed by several economic measures. The most widely used in public health are measures such as cost-effectiveness, cost benefit, cost utility, and to a lesser extent, return on investment. The following considerations may be helpful when considering the magnitudes of economic measures for the development of recommendations:
- The recommended level for the cost of an intervention depends on the available budget and on how much money managers are willing to spend in the intervention
- The recommended level for cost-effectiveness ratios is set by convention and depends on what managers consider an affordable cost per outcome
- The recommended level for cost-utility ratios varies among countries and organizations o The ratio used by NICE is 20,000 to 30,000 per quality adjusted life year (QALY).
- The ratio used in the United States varies between $50,000 and $100,000 per QALY.
- The amount recommended by WHO is 1 to 3 times the per capita gross domestic product (GDP).
- In theory, a project is accepted if the net present value from a cost-benefit analysis is larger than 0 where the benefits are larger than the costs; however, in those cases where health interventions are not cost saving, managers must decide what level of cost per dollar of benefit is acceptable given current resources.
- The recommended level for return on investment in financial terms is a positive return per unit of investment; that is, a positive ratio from dividing positive benefits by positive investments. A positive ratio is mathematically feasible by dividing negative benefits by negative investments. However, this is not conceptually feasible because there is no such thing as a "negative investment". By summarizing and interpreting economic studies, systematic reviews make economic information useful and accessible to guideline users. Economic data from systematic reviews facilitates the process of using limited resources in implementing interventions that contribute to making the greatest possible improvement in population health. For information on methods for conducting systematic reviews of economic evaluations, consult Carande-Kulis et al (2000) and the online methods section of the Community Guide.  For standardized methods of economic evaluation and their application to public health, consult the online CDC course Economic Evaluation of Public Health Preparedness and Response Efforts.

# WRITING THE GUIDELINES
Guideline documents introduce recommendations to the public health community, describe the evidence supporting the recommendations and explain the process used to move from the evidence to the recommendations. In other words, when guidelines work groups start the writing process, a shift is needed from focusing on the scientific evidence to communicating the recommendations in a way that is most likely to influence practice. Additionally, the communication styles often used in scientific publications are not necessarily appropriate for guidelines documents. The authors of guidelines will need to carefully consider the needs of the target audience and adjust the format of the document and the communication strategy accordingly. The federal Plain Writing Act requires new or substantially revised documents for the public to be written in plain language. In general, a plain writing style that follows the federal language guidelines will make the information understandable for most readers, particularly for those who do not have technical knowledge of the subject matter.

# Identifying the Level of Detail
A major choice when developing the guidelines document is its level of detail. Although some guideline users may want a full and detailed description of the evidence synthesis process and background data, others may prefer a shorter format that focuses on the recommendations. A longer format can provide a wide range of information needed for multiple audiences (practitioners, policy makers, researchers, etc.) in a single document, but with the downside that user-specific information might be harder to find.
Because the document may have multiple audiences, the authors might consider developing multiple versions instead of a single document for everyone. For example, full guidelines may be developed for publication in the MMWR, a reduced version for publication in a peerreviewed journal, a short synopsis for the practitioner, and a fact sheet for the public. A common approach is to publish a concise set of guidelines for practitioners, accompanied by a more comprehensive treatment of the science that can be published separately (or posted on the Web). Hybrid approaches, such as formats that combine a concise presentation of the guidelines along with a brief summary of the scientific evidence, can also be used. This combined approach makes important evidence available to those who want to dig a little deeper than just the bare-bones guidelines.

# Describing the Audience
The document needs to state: a) who will use the information, b) in what format, and c) how they will use it. Authors need to ensure that the communication style and technical levels are appropriate for the readers and that their needs are being met. Answering the following questions may ease the writing of the guidelines:
- What population (age, gender, race and ethnicity, socioeconomic status) and setting (state or local health agency, community, home, workplace, hospital, ambulatory clinic) will the guidelines target?
- Who are the guideline users (public, public health professional, advocate, or clinician)?
- Whose knowledge, practices, behavior, or decisions will the guidelines attempt to address?
- What health measurement, outcome, or condition (risk factor, prevalence, morbidity, mortality, quality of life, healthy, at-risk and asymptomatic, symptomatic, diseased) will the guidelines target for improvement?
- What intervention, public health service, or provider practice will the guidelines target to implement or improve?
The following questions may help the authors adjust the style and content to the target audience:
- What kind of background information and evidence will the audience need?
- What information will be needed to determine when the guidelines do and do not apply?
- What information and resources will the readers need to implement the guidelines?
- Will the audience be largely receptive to the guidelines, or will some readers be skeptical and need more rationale to support the recommendations?

# Providing a Methods Section
The methods section should provide readers with sufficient understanding of the processes leading to development of the guidelines, enabling them to judge the guidelines' quality, strengths, weaknesses, and limitations. Often, a methods section does not get much attention when guidelines documents are being developed. Sometimes a methods section is added as an afterthought, if at all. However, a well-crafted methods section establishes trust with the reader. Through the methods section, the reader gains an appreciation for the rigor and limitations of the guidelines development process. At a minimum, the methods section should state the following:
- the evidence sources
- what evidence was (and was not) considered
- who participated in the synthesis
- how evidence was summarized
- how effect size of selected outcomes was reported
- how evidence quality was assessed
- how the evidence led to the recommendations
- how the strength of the recommendations was assessed
- how evidence gaps were treated More detail on each of these elements is provided below.

# Describing the Evidence
The methods section should briefly summarize the evidence domains considered and how the evidence was identified. The methods section should indicate the analytic framework and the specific review approach used to obtain the evidence. The method used to obtain the evidence should be described (e.g., systematic review) and justified. The search protocol, including search terms (e.g., keywords), sources used (e.g., databases, journals, repositories), and inclusion criteria (e.g., language, study type, human subjects inclusion, time frames, population) should be described. The criteria used to select the final evidence for developing the guidelines (e.g., study design, population, intervention, outcomes) should be detailed and justified. How evidence was coded and synthesized should be described (e.g., how data were extracted from studies; how data were synthesized, such as through use of effect measures; whether data were combined through meta-analysis; how heterogeneity in effects was explored). The method used to assess evidence quality should be described.

# Describing Relevant Evidence That Was Not Considered
Often, some evidence domains could be viewed as relevant but are not considered. For example, evidence may be excluded if it is not in English, not in the peer-reviewed literature, outside certain disciplines, etc. Excluded domains should be mentioned along with the rationale for why the evidence was not considered. Writers also might want to note topics that were not included (e.g., not addressing issues in pregnant women, in non-adults, a particular period for a birth cohort) and why these groups were not included.

# Describing Partners Who Participated in the Guideline Development Process
Guidelines are developed by people, and knowing who participated may be just as important to the reader as knowing what evidence was considered. Answering the "who" question is a crucial component of all methods sections and is especially important for guidelines that rely upon expert opinion. The document should indicate who participated, how they were selected, their expertise, and disciplines represented. The document should list the professional organizations from which participants were solicited. It should also state whether the work group was composed of or worked under one of CDC's official advisory committees. The document should specify the participants' roles (e.g., authors, advisors, reviewers) distinguishing between participants who approved the final recommendations and those who provided review or input. Finally, authors need to make sure the document specifies how competing interests (e.g., financial, intellectual, professional) among participants were identified, disclosed, and handled (e.g., through recusal from evidence synthesis, deciding on the final recommendation statements).

# Describing How the Evidence Led to the Recommendations
A good methods section describes how the evidence led to the recommendations.
Frequently, this section is the most difficult to write. One challenge is that virtually all guidelines include some measure of expert opinion or judgment. Concisely explaining the rationale used by a panel of experts can be difficult, especially when the evidence base for the recommendation is sparse or a clear effect is not evident. But even for recommendations based on solid evidence, judgment calls will be made, which adds to the difficulty in developing a simple explanation for how the evidence led to the recommendation.
The document should explain how the evidence was weighted, how evidence quality was judged, what evidence was considered more (or less) compelling, and how the strength of the evidence was determined. If information was obtained from expert opinion, the processes used (e.g., Delphi process, focus groups, questionnaires) should be described. The document also needs to include the processes used to reach agreement, such as informal consensus, formal consensus, or voting. Most guideline development efforts use a combination of processes.
Those groups that have used formal evidence-based frameworks for developing guidelines will likely find it easier to describe the links between the evidence and the recommendations.
However, even for guidelines that are largely based on expert opinion, the authors should avoid simply stating that experts reviewed the available materials and made recommendations. Invariably, some level of evidence and rationale exists even for guidelines based on expert opinion, and the authors should provide as much of the rationale as possible. Throughout the process, the authors should record major and possible subjective decisions made and the rationale and assumptions contributing to the decisions, as well as other alternatives considered. These records will enable the authors to accurately describe the suggested details in the methods section. Note that this point should be considered while the experts are still deliberating the recommendations because the rationale may be more difficult to reconstruct after the expert panel has been disbanded.
If economic efficiency measures were presented as part of the evidence, the authors need to explain how any of this evidence was used. Most of the time, this evidence is not included in the decision criteria that translates the evidence into recommendations; instead, this evidence is presented as supplemental information. If this evidence was discussed before consensus on the recommendations was reached, the authors need to explain whether the evidence (or lack thereof) of economic efficiency influenced the recommendation. Economic efficiency is covered in detail in Section 5.

# Describing How Evidence Gaps Were Treated
It is rare for guidelines to lack even a few gaps in the evidence. Guidelines authors should be prepared to explain how these evidence gaps were addressed. Guidelines documents sometimes include statements that evidence gaps were bridged by expert opinion.
Whenever possible, guidelines authors should explain what approaches were used by the experts to bridge the gaps, how incomplete evidence was treated, who made the judgment calls, and what assumptions were in play. For example, some guidelines panels may withhold recommendations when conclusive evidence is lacking, whereas others may follow a more precautionary approach and develop a recommendation even when evidence is scarce (unless there is evidence that the measures are harmful). Or, when evidence is lacking and as long as risks are minimal, guidelines panels may default to current established guidelines or practices. Regardless of the approach, the methods section should give readers sufficient information to understand the major gaps in evidence and to decide whether they agree with how these gaps were treated. The manner in which evidence gaps are treated can impede trust with the target audience. Glossing over evidence gaps may damage the credibility of the guidelines.

# Writing Guidelines for Maximum Impact on Policy and Practice
For guidelines to have maximum impact on policy and practice, the writing needs to be clear, Guidelines might be consulted more often if the recommendations are stated clearly and address WHO should do WHAT to WHOM, UNDER WHAT CIRCUMSTANCES, HOW, and WHY. Newly developed software is now becoming available to assist guidelines developers in crafting recommendation statements in this format.  In addition, the Conference on Guideline Standardization has developed a checklist for reporting practice guidelines that can serve as a helpful resource.  Of critical importance is clearly describing the methods used to synthesize and rate the quality of the evidence. Practitioners may be more likely to follow guidelines when the evidence base is strong and the recommendations are easy to understand.  One international research study suggests that practitioners are more likely to change their decisions about practices when clear descriptions of evidence quality and strength of recommendations are provided, particularly through use of the GRADE approach.  Practitioners also value discussion of factors that might affect the implementation of the recommended intervention or practice such as feasibility, reach, resource requirements, and acceptability.

# EXTERNAL REVIEW AND VETTING OF CDC GUIDELINES
Each release of a new CDC guideline might have a lasting impact on clinical and public health practice. Guidelines may be converted to policy, implying widespread implementation by a broad range of groups. Guidelines may be even converted into law, entailing subsequent regulatory enforcement. Because of the high profile CDC guidelines may acquire following their release, we recommend guidelines be vetted before publication. Vetting of CDC guidelines ensures that key stakeholders have the opportunity to review and provide critical input. Vetting not only assists in fostering transparency and credibility but also improves the clarity and understanding of the guidelines. Even though it may not be possible to incorporate all the feedback received, vetting of the guidelines has many benefits. Some benefits of vetting are:
- Identifying overlooked issues (e.g., new evidence released outside of the literature search period)
- Identifying possible post-guideline problems (e.g., feasibility considerations, misinterpretation by the target audiences)
- Improving chances of buy-in and adoption
- Providing a preview for guideline implementers to prepare to adopt new recommendations

# Internal and External Vetting
We recommend that CDC guidelines be vetted internally and externally. Vetting processes are defined by various factors such as scope (e.g., broad vs. narrow), level of controversy, number of stakeholders, and timeline (e.g., recommendations for urgency vs. routine revision of long-standing recommendations). Internal vetting is critical for guidelines that cross multiple programs. However, guidelines of narrow scope might not need to be vetted internally. Typically, the need for internal vetting should be determined by CDC's crossclearance policies. Questions about the need for cross-clearance should be directed to the program or the CIO Associate Director for Science (ADS). When cross-clearance is required, having involved the program or ADSs throughout the guideline development process will help ensure timely clearance.
External vetting can occur in public meetings during a CDC advisory committee meeting, through informal reviews by key stakeholders, by community engagement forums when public interest in the guideline is expected to be significant, by formal request for comments (e.g., posting in Federal Register), or a combination of the above.
 Regardless of the format used, the vetting process must be carefully planned so adequate personnel and financial resources are available to handle the reviews. For example, following posting in the Federal Register, comments received will need to be compiled and reviewed. Responses should be prepared for each individual who provided comments (e.g., response matrix). The writing group will typically review the comments and craft responses. Having a record of responses will demonstrate that CDC considered each comment. This documentation is especially important when requests for follow-up occur through controlled correspondence or the Freedom of Information Act. Although the vetting process might vary slightly for different guidelines, each must have a process in place. Vetting is the critical step in the CDC guidelines development process that ensures transparency and credibility.
Because of the importance and high impact of guidelines documents, authors should consult with their division or the CIO ADS on whether external peer review is warranted before final clearance. Peer review by external experts is common for guidelines documents as a way to get an independent assessment of the quality of the guidelines. The peer-review process may be limited to a review of the science of the guidelines or may include a review of all aspects of the document. If the review is limited to the science, then reviewers are chosen for their scientific expertise and are typically selected to be as independent of CDC as possible. For other types of review, inclusion of partners and potential users might be desirable, especially those chosen to represent a range of stakeholder positions. Reviews by stakeholders may be conducted within the scientific review process or separately.
A specific type of external peer review is required for guidelines documents that are considered influential scientific information (ISI) or highly influential scientific assessments (HISA) as defined by the Final Information Quality Bulletin for Peer Review issued by OMB in 2004.  This bulletin requires that specific information be reviewed by qualified experts before dissemination. As noted in the CDC clearance policy, "Clearance is not a forum for extensive peer review or for policy debate. Such discussions belong in the pre-clearance phase." Therefore, guidelines authors should ensure that guidelines documents have been appropriately reviewed for quality of execution and reporting and that policy issues have been resolved before submitting these documents for clearance. If guidelines documents have scientific or policy issues that overlap with other CIOs, those CIOs should be consulted early in the guidelines development process (and, should be part of the guidelines development team).

# CLEARANCE OF CDC GUIDELINES AND RECOMMENDATIONS
CDC sometimes develops guidelines jointly with other organizations. If a CDC staff member will be listed as a coauthor of a document, or if the document will include the CDC brand, the document requires clearance by CDC. For jointly developed documents, authors should discuss clearance requirements early in the process so external authors understand CDC clearance requirements and are aware that CDC will need to formally approve the final document. Also, CDC authors should ensure they understand review and clearance requirements that the collaborating organizations will impose. Additionally, CDC authors should consult with the division and CIO ADS and others in the clearance chain early in the development of guidelines documents to avoid surprises during the clearance process.
Note that CDC clearance is required whenever CDC is listed as a coauthor on a guidelines document-even if CDC is not listed as a formal sponsor or if the person is in a group author listing. This CDC clearance requirement has two notable exceptions:
- When a CDC author helps develop guidelines as an approved outside activity (conducted outside of working hours and without the use of CDC resources), the author may be listed without CDC affiliation
- When CDC personnel are listed as contributors, reviewers, or consultants, although they are not listed as coauthors When these situations occur, include a disclaimer stating that these roles do not necessarily imply CDC agreement or endorsement of the guidelines or recommendations. In such cases formal CDC clearance is not necessary (although supervisory approval is needed). One additional clarification should be made on the use of disclaimers in guidelines documents. CDC authors are accustomed to using the standard disclaimer on journal publications required by OMB's Final Information Quality Bulletin for Peer Review.
 However, because CDC guidelines documents represent the official position of CDC, the OMB disclaimer should not be used on guidelines documents when they are sponsored by CDC or a CDC staff member is listed as a coauthor. Guidelines authors can consider whether other types of disclaimers are appropriate for the specific document (e.g., disclaimers noting that mention of specific products does not imply endorsement by CDC).

# RECOMMENDED STANDARDS FOR CDC PUBLIC HEALTH

# GUIDELINES
There are advantages to using standardized approaches for guidelines development. A number of organizations have adopted uniform frameworks for guidelines they develop. However, the breadth of disciplines and topic areas at CDC makes it impractical to use a single, unified approach for CDC guidelines. Therefore, we are not recommending a single approach; instead, we have identified a set of recommended standards we recommend for CDC guidelines.
Although these standards might not apply in special instances, most CDC guidelines can be developed using these principles.
The guidelines development process should include the following:
1. Identify the public health problem on a given topic, need for new or revised guidelines, and required resources for development: Before developing CDC guidelines, the sponsoring program should identify the topic for the guidelines and assess the basis for developing new or revised guidelines. This assessment should include an evaluation of the public health need for the guidelines on the topic, the relationship to CDC's mission and priorities, a review of existing guidelines on the topic, pros and cons of developing the new guidelines (including implications if guidelines are not developed), estimated resources (personnel and budget), timeframe for developing the guidelines, and a list of partners or organizations that will be engaged. This assessment should be shared with senior leadership (e.g., CIO director) before starting development. 8. Involve an explicit scientific quality and policy control process: The guidelines development work groups should be provided with a description of how the guidelines will be finalized and cleared and how disagreements or inconsistencies will be resolved. Typically, guidelines development groups with external participants provide advice to CDC, and thus the role of CDC clearance and final CDC acceptance of the guidelines should be explained.
The guidelines document should report on the following topics:
1. Guideline development process: Guidelines documents should clearly describe the process used to develop the guidelines. This description should include who participated in the process, how they participated, how decisions were made, who had final approval of the document, and potential competing interests among participants.
2. Evidence synthesis process: The guidelines document needs to describe the process used to synthesize the evidence; for example, through systematic literature review. If a systematic review was not feasible (e.g., because of resource allocations or time constraints) and only a narrative review is included, make sure the evidence-gathering process is as complete and methodical as possible and the report is clear about how the evidence was identified and evaluated.
3. The evidence base supporting the recommendations: The evidence base used to develop the guidelines should be described and summarized. A description of the methods used to evaluate the suitability of articles should be provided. Unpublished data sources should be described. If notable sources of evidence were not used, mention them and explain why they were not included in the body of evidence.
4. The quality of the evidence and strength of the recommendations: The document should describe the quality of the evidence, the method used to assess the quality of the evidence, how the evidence led to the recommendations, and the strength of the recommendations based on the effect size of selected outcomes and other factors (e.g., potential harms, values and preferences, economic efficiency).
5. Limitations and applicability of guidelines: Guidelines documents should explain significant limitations or caveats of the guidelines, including (when appropriate) situations where the guidelines may not apply.
6. Relationship to similar or overlapping guidelines: Guidelines should state how they are related to similar or overlapping guidelines. If a guidelines document updates or supersedes previous guidelines, this should be stated.

# APPENDIX I -GLOSSARY
Analytic framework -A process usually represented by a diagram that shows hypothesized links between an intervention and related intermediate or final health and non-health outcomes.  Applicability -Whether the intervention process could be implemented in the local setting, no matter the outcome.  Algorithm -A procedure consisting of a sequence of algebraic formulas and logical steps to calculate or determine a given task.  Bias -Deviation of results or interferences from the truth.  Body of evidence -The complete set of qualifying studies compiled using systematic or nonsystematic reviews.  Case study -Method of study in which persons with the disease (or condition) of interest are compared with a suitable control group of persons without the disease.  Comparison group -A group that was not exposed to a particular intervention; the control group, used to determine what would have happened if the intervention had not been carried
-ut.  Effect -The change in an outcome that results from an intervention.  Effect size -The magnitude of the effect measured in a study of the intervention.  Effectiveness -The degree to which an intervention achieves a desired outcome in practice.
Efficiency -Carrying out production so as to obtain the maximum amount of output for any given set of inputs (technical efficiency) or choosing inputs so as to minimize cost of production (cost efficiency).  Evidence-based  The use of scientific data to confirm that proposed interventions or practices are appropriate in light of their high probability of producing the best and most favorable outcome.  External validity -The ability to generalize study results to populations and context beyond the particular ones included in the studies themselves (see applicability).  Guideline steering committee  In-house group composed of CDC staff with the functions of overseeing each step of the guideline development process. Health outcome -The change in health that is hypothesized to result from the intervention (e.g., reduced morbidity or mortality or increased physical, mental, or psychological function).
Impact -The association between an exposure and an outcome in a meaningful public health context. Measures of impact reflect the degree of exposure contributing to the frequency of disease in the population. Two measures of public health impact often used are the attributable proportion and efficacy or effectiveness.  Intermediate outcome -One in a series of intermediate effects from an intervention potentially linked to a final health outcome. An intermediate outcome with a strong and established link to a final health outcome may serve as a recommendation outcome.  Internal validity -A study trait indicating that the measured effect resulted from implementing the intervention.  Intervention -A specific activity pursued by public health practitioners aimed at reducing disease risk, treating illness, ameliorating the consequences of disease and disability, or preventing a health problem.  Meta-analysis -A systematic, quantitative method for combining information from multiple studies to derive a meaningful answer to a specific question.  Other effects -Outcomes or effects other than anticipated as a possible result of the intervention; effects can be positive (beneficial) or negative (harmful).  Policy -Laws, regulations, and formal and informal rules and understandings that are adopted on a collective basis to guide individual and collective behavior.  Reliability -The degree of stability exhibited when a measurement is repeated under identical conditions; the degree to which the results obtained by a measurement procedure can be replicated.  Systematic review -A process by which a body of literature is reviewed and assessed using systematic methods intended to reduce bias in the body of evidence.  Transferability -Degree to which the results of a study or systematic review can be extrapolated to other circumstances, in particular to routine to public health or health care situations.