*HomeU.S. Government Accountability Office*
Menu

      close


          
          
                          	
				
				
	
	
					Reports & Testimonies
									
																				
				
				
	
	
								Reports & Testimonies
													
																												
																						
							
																Recommendations Database
							
																												
																						
							
																Month in Review
							
																												
																						
							
																Order Copies
							
																												
																						
							
																Restricted Reports
							
																												
																						
							
																View Topics
							
																												
																						
							
																View Agencies
							
																												
																						
							
																Bid Protests & Appropriations Law
									
																				
				
				
	
	
								Bid Protests & Appropriations Law
													
																												
																						
							
																Bid Protests
							
																												
																						
							
																Appropriations Law
							
																												
																						
							
																Federal Vacancies Reform Act
							
																												
																						
							
																Other Legal
							
																												
																						
							
																About
									
																				
				
				
	
	
								About
													
																												
																						
							
																What GAO Does
							
																												
																						
							
																U.S. Comptroller General
							
																												
																						
							
																Careers
							
																												
																						
							
																Contact UsFor Congress
                                
                          
        
                Press Center
                                
                          
        
                Careers
                                
                          
        
                BlogSearch Breadcrumb
        Home
                            Reports & Testimonies
                            Infectious Disease Modeling: Opportunities To Improve Coordination and Ensure Reproducibility
    Infectious Disease Modeling:
                          Opportunities to Improve Coordination and Ensure Reproducibility **GAO-20-372**
        Published: May 13, 2020. Publicly Released: Jun 04, 2020.Jump To:
    Jump ToElder Abuse
          Services for Older Adults
          State and Local-Level Drivers and Trends
          State and Local Fiscal Projections
          Federal Borrowing
          Managing the Debt
          Summary of Funding Provisions
          Auditing the Government's Books
          Unpacking the Financial Report
          Highlights
          Recommendations
          Background
          About This Work
          About This Work
          Trends
          About the Center
          Why It's High Risk
          What Remains To Be Done
          Key Reports
          Newest COVID-Related Reports
          AI Use Cases
          Flood Insurance
          Past Pandemic-Related Reports
          CARES Act Oversight Reports
          All COVID-19 Reports
          Report Suspected Fraud
          Subjects
          How to Get Notified
          Past Reports
          Public Health Threats
          Explore the FY 2022 and 2023 Funds
          Lessons Learned
          Agency Reports
          Recent Reports
          Recommendations
          Supplemental Material
          Full Report
          Additional Data
          View Decision
          Downloads
          Previously Identified Actions
          Related Pages
          Associated Agencies
          Tough Choices and Opportunities Ahead
          Americas Fiscal Future Key Areas
          Retirement Security Key Areas
          Current List
          About The High Risk List
          Area Ratings
          Previous High Risk Products
          Search Federal Vacancies
          Past Federal Vacancies
          Violation Letters
          About the Yellow Book
          Advisory Council
          Related Resources
          Comment Letters
          Related Publications
          Contacts and Resources
          Major Issues Facing the Nation
          Meeting Strategic Challenges
          Additional Resources
          Resources
          GAO Contacts
          Mission Teams
          Operations and Staff Offices
          Organization Chart
          Why a Career at GAO?
          Am I Qualified?
          Company Culture
          Reasonable Accommodations
          Video Gallery
          Paid Internships
          Student Volunteers
          Professional Development Program
          Advancement
          Executive Candidate Program
          Summer Associates
          Externships
          Benefits
          Contacts
                    Fast Facts
              Outbreaks of infectious diseases—such as novel coronavirus and pandemic flu—have raised concerns about how federal agencies use modeling to predict a disease’s course. Models can help decision makers set disease control policies and allocate resources. If models are unsound, they may not produce the reliable predictions needed to make good decisions.
We examined how Health and Human Services, which includes the Centers for Disease Control and Prevention, uses and assesses models. We recommended that HHS improve coordination of modeling across agencies and ensure models are reproducible, which helps build confidence in their results.
An HHS employee working on Infectious Disease Planning and Response
Staff, computers, monitors, and projected screens
Staff, computers, monitors, and projected screens
Skip to Highlights
Highlights
  What GAO Found

Within the Department of Health and Human Services (HHS), the Centers for Disease Control and Prevention (CDC) and the Office of the Assistant Secretary for Preparedness and Response (ASPR) used models to inform decision-making during and after outbreaks of Ebola, Zika, and pandemic influenza. These agencies' modeling efforts informed public health planning, outbreak response, and, to a limited extent, resource allocation. Four CDC centers perform modeling.

HHS agencies reported using multiple mechanisms to coordinate modeling efforts across agencies, but they do not routinely monitor, evaluate, or report on the extent and success of coordination. Consequently, they risk missing opportunities to identify and address modeling challenges—such as communicating clearly, and obtaining adequate data and resources—before and during an outbreak. As a result, agencies may be limiting their ability to identify improvements in those and other areas. Further, there is potential for overlap and duplication of cross-agency modeling efforts, which could lead to inefficiencies.

Office of the Assistant Secretary for Preparedness and Response's Visualization Hub, which Can Be Used for Infectious Disease Planning and Response

U:\Work in Process\VCA_Graphics\FY 20\STAA\102801_STAA_20_372_Hinkle\Master Graphic Files\High01_5_v2_102801.tif

CDC and ASPR generally developed and assessed their models in accordance with four steps GAO identified as commonly-recognized modeling practices: (1) communication between modeler and decision maker, (2) model description, (3) verification, and (4) validation. However, for four of the 10 models reviewed, CDC did not provide all details needed to reproduce model results, a key step that lets other scientists confirm those results. GAO found that CDC's guidelines and policy do not address reproducibility of models or their code. This is inconsistent with HHS guidelines and may jeopardize the reliability of CDC's research.

This report also identifies several modeling-related challenges, along with steps agencies have taken to address them.

Why GAO Did This Study

Outbreaks of infectious diseases—such as Ebola, Zika, and pandemic influenza—have raised concerns from Congress about how federal agencies use modeling to, among other things, predict disease distribution and potential impacts. In general, a model is a representation of reality expressed through mathematical or logical relationships. Models of infectious diseases can help decision makers set policies for disease control and may help to allocate resources.

GAO was asked to review federal modeling for selected infectious diseases. This report examines (1) the extent to which HHS used models to inform policy, planning, and resource allocation for public health decisions; (2) the extent to which HHS coordinated modeling efforts; (3) steps HHS generally takes to assess model development and performance; and (4) the extent to which HHS has addressed challenges related to modeling. GAO reviewed documents and interviewed HHS officials, state officials, and subject matter experts. GAO identified practices commonly used to assess infectious disease model performance and reviewed 10 selected modeling efforts to see if they followed these practices.

Recommendations
      
            GAO recommends that HHS (1) develop a way to routinely monitor, evaluate, and report on modeling coordination efforts across multiple agencies and (2) direct CDC to establish guidelines to ensure full reproducibility of its models. HHS agreed with GAO's recommendations.
      Recommendations for Executive Action
        Agency Affected
                                                  Recommendation
                                                  StatusDepartment of Health and Human Services
                                                                                        Priority Rec.The Secretary of Health and Human Services should develop a mechanism to routinely monitor, evaluate, and report on coordination efforts for infectious disease modeling across multiple agencies. (Recommendation 1)
                                                                                        Open
HHS agreed with and has begun taking steps to implement this recommendation. HHS stated that, as of February 2024, it is developing a process whereby it will coordinate its efforts in infectious disease modeling across its components, which will include monitoring, evaluating, and reporting on such coordination. To fully address this action, HHS needs to finish developing and implementing this process, and provide relevant documentation, while ensuring that the process routinely monitors, evaluates, and reports on coordination of infectious disease modeling efforts across multiple agencies. Successful completion of this effort could help HHS better identify any duplication and overlap among agencies, which could help them to better plan for and respond to disease outbreaks.Department of Health and Human Services
                                                                                        Priority Rec.The Secretary of Health and Human Services should direct CDC to establish guidelines that ensure full reproducibility of CDC's research by sharing with the public all permissible and appropriate information needed to reproduce research results, including, but not limited to, model code. (Recommendation 2)
                                                                                        Open
The Centers for Disease Control and Prevention concurred with and has begun taking steps to implement this recommendation. As of February 2024, CDC stated it was making progress toward improving the speed, quality, and timeliness of its shared data. It stated it has worked to prioritize and drive high-impact, high quality science to better inform federal, state, and local decision-making efforts. Additionally, CDC stated it has implemented system efficiencies and process improvements to improve data clearance efficiency. To fully address this action, CDC needs to finish updating and provide its policies and guidelines and modernize its information sharing processes. This could help to ensure and maximize the quality, objectivity, utility, and integrity of information disseminated to the public, such as by modifying its guidelines to include language relating to model reproducibility, including, but not limited to, model code. Successful completion of this effort could help CDC ensure that its models are reproducible, a key characteristic of reliable, high-quality scientific research.Full Report
    Highlights Page (1 page)
              Full Report (78 pages)
              Accessible PDF (90 pages)
GAO Contacts
    
      Timothy M. Persons
            
            Chief Scientist
      
            personst@gao.gov
      
            (202) 512-6888
Office of Public Affairs
    
      Chuck Young
            
            Managing Director
      
            youngc1@gao.gov
      
            (202) 512-4800
Topics
    
      Health CareDisease controlDiseasesEmerging infectious diseasesEpidemicsHealth careInfectious diseasesInfluenzaInfluenza pandemicsMedical countermeasurespandemicsPathogensPublic healthPublic health emergenciesData collectionDecision making
                                                      MultimediaThe figure below shows the 2 broad categories of infectious disease models: mechanistic models, which use scientific understanding of disease dynamics and human behavior, and statistical models, which rely only on patterns in the data.
          Blog Post
              Disease modeling: How Math Can Help In A Pandemic
            Tuesday, June 9, 2020
      
            The COVID-19 pandemic has brought new attention to models of infectious disease. These models are...
 Receive GAO Updates
  Stay informed as we add new reports & testimonies.

    GAO Logo
      U.S. Government Accountability Office


                                   Column 1
                            Press Center
                              Contact Us
                              Inspector General
                              Restricted Reports
                              Column 2
                            Copyright & Terms of Use
                              Privacy Policy
                              Accessibility
                              Sitemap
                              Column 3
                            FOIA Requests
                              Scam Alerts
                              No FEAR Act Data
                              Health Care Advisory Committees