Infectious Disease Modeling: Opportunities to Improve Coordination and Ensure Reproducibility
Released: 2020-06-04T00:00:00

Letter

Background

Public Health Agency Roles in Infectious Disease Outbreaks and Response

In the United States, HHS is the lead federal agency responsible for public health. Its responsibilities include preparing for, mitigating, responding to, and recovering from public health emergencies. Within HHS, ASPR and CDC prepare for and respond to infectious disease outbreaks.

ASPR leads and coordinates national preparedness and response to outbreaks in the United States. It also coordinates and supports advanced research and development, manufacturing, and procurement and deployment of medical countermeasures, such as vaccines, drugs, therapies, and diagnostic tools that can be used in the event of a potential public health emergency to protect the public from harm.

CDC monitors and responds to outbreaks by, among other things, studying the link between infection and health; monitoring and reporting cases of infection; and providing guidance to the public, travelers, and health care providers. During public health emergencies, CDC may operate an Emergency Operations Center (EOC) for monitoring and coordinating its response to emergencies—including infectious disease outbreaks of Ebola, Zika, and pandemic influenza—in the United States and abroad. The EOC staff helps with directing specific incident operations; acquiring, coordinating, and delivering resources to incident sites; and sharing incident information with the public.

Other agencies perform additional work related to infectious diseases. For example, FDA monitors and protects the blood supply, and NIH makes grant awards that support research related to diseases and modeling.

ASPR, CDC, and FDA have different approaches to modeling. In the cases of Zika, Ebola, and pandemic influenza, CDC and ASPR are two key agencies that conduct federal infectious disease modeling efforts. As of February 2020, ASPR had a centralized modeling unit staffed by about nine people, who are a mix of federal and contract employees, according to ASPR officials. At CDC, however, modeling is decentralized and integrated into the individual centers that make up the agency. Some staff work full time on modeling, while others spend part of their time on other tasks. In addition, some of CDC’s modeling efforts are conducted externally. According to CDC, approximately 70 staff members participated in modeling studies, as of October 2018. Of those staff, CDC’s Health Economics and Modeling Unit employed about 10 modelers who have worked on Ebola and other diseases. For Zika, CDC officials responding to Zika said most modeling work was done by one modeler in CDC’s Division of Vector-Borne Diseases, a part of the National Center for Emerging and Zoonotic Infectious Diseases. CDC influenza officials said influenza modeling is conducted by six or seven members of CDC’s Influenza Division. Agency infectious disease modeling activities are not limited to Ebola, Zika, or pandemic influenza.

Agency efforts to protect the nation from disasters and emergencies can be organized into two elements: preparedness and response. Infectious disease modeling is one tool used to inform a wide range of decisions related to outbreak preparedness and in response to an outbreak. In the context of infectious disease outbreaks, ASPR and CDC perform work on preparedness and response. For example, ASPR leads the Public Health Emergency Medical Countermeasures Enterprise (PHEMCE), an interagency group that helps develop medical countermeasures—FDA- regulated products including drugs, or devices that may be used in the event of a potential public health emergency to protect the public from harm. CDC may activate its EOC to assist with the response during an outbreak. For example, during the 2014-2016 West Africa Ebola outbreak, CDC activated its EOC in July 2014 to help coordinate activities. CDC personnel were deployed to West Africa to assist with response efforts, including surveillance, data management, and laboratory testing.

Infectious Disease Outbreaks

Since the 1980’s, emerging infectious diseases have resulted in more recurrent disease outbreaks, causing an increasing number of human infections. Emerging infectious diseases have at least one of the following characteristics: they are newly recognized, have emerged in new areas, are newly affecting many more individuals, or have developed new attributes. Some of these diseases—including Ebola and Zika—are zoonotic pathogens, meaning they spread from animals to humans. Zoonotic pathogens can be carried from an animal to a human by another animal, such as a mosquito, chicken, or bat, which is known as a vector. Such pathogens sicken approximately 1 billion people annually.

Ebola

According to the World Health Organization, Ebola causes an acute, serious illness, which is often fatal if untreated. Ebola is introduced into human populations through close contact with the blood and other bodily fluids of infected animals. Humans spread Ebola through direct contact with the bodily fluids of infected individuals or objects contaminated with these fluids. Ebola symptoms include fever, muscle pain, vomiting, diarrhea, impaired kidney and liver functioning, and, in some cases, internal and external bleeding. There have been five Ebola outbreaks since 2014, including the 2014-2016 West Africa outbreak which caused more than 28,600 cases and 11,325 deaths. Since 2018, there has been an ongoing outbreak in the Democratic Republic of the Congo. Figure 1 provides a timeline of Ebola outbreaks since 2014.

Zika

Zika is a virus that is primarily transmitted through mosquito bites. It can cause symptoms such as fever, rash, conjunctivitis (red eyes), and joint and muscle pain. It can also be transmitted from mother to child during pregnancy, or around the time of birth, or from person to person through sexual contact or blood transfusion. Many infected people do not have symptoms or will only experience mild symptoms. The Zika outbreak that began in 2015 affected individuals infected with the virus in ways that had not been seen with previous outbreaks of the disease. Specifically, during the 2015-2016 outbreak, Zika infection in pregnant women was linked to microcephaly and other severe brain defects, according to CDC. CDC officials said this was the first time in more than 50 years that an infectious pathogen has been identified as the cause of birth defects. Zika was also linked to other problems, such as miscarriage, stillbirth, and Guillain-Barré syndrome, an uncommon disorder affecting the nervous system. In the Western Hemisphere, the first cases of locally- transmitted Zika were confirmed in Brazil in May 2015. In December 2015, locally-transmitted Zika was reported in Puerto Rico. On January 22, 2016, CDC activated its Emergency Operations Center to respond to outbreaks of Zika occurring in the Americas and to increased reports of birth defects and Guillain-Barré syndrome in areas affected by Zika. Within the continental United States, the first locally-transmitted cases were confirmed in Florida in June 2016. The World Health Organization declared Zika a Public Health Emergency of International Concern from February to November 2016.

Pandemic Influenza

In the spring of 2009, a novel influenza virus emerged, known as influenza A (H1N1)pdm09. According to CDC, it was detected first in the United States and quickly spread across the world, causing a pandemic or global outbreak of a new influenza A virus. This new virus contained a combination of influenza genes not previously identified in animals or people. The virus was very different from other H1N1 viruses circulating at the time, so seasonal influenza vaccines offered little cross-protection against infection with the new H1N1 virus, according to CDC. A vaccine against the new virus was produced, but it was not available in large quantities until late November—after the peak of illnesses during the second wave in the United States. CDC activated its EOC on April 22, 2009, to manage the H1N1 response. From April 12, 2009, to April 10, 2010, CDC estimated there were about 60.8 million cases, 274,304 hospitalizations, and 12,469 deaths in the United States due to the new H1N1 virus. According to CDC, few young people had any existing immunity—as detected by antibody response—to the virus, but nearly one-third of people over 60 years old had antibodies against it, likely from exposure to an older H1N1 virus.

Multiple strains of influenza can infect humans, including strains that originate in animals. According to CDC, human infections with an Asian lineage avian influenza A (H7N9) virus were first reported in China in March 2013. During an epidemic that lasted from October 1, 2016, through September 30, 2017, the World Health Organization reported 766 human infections with H7N9 virus, making it the largest H7N9 epidemic. From 2013 to December 7, 2017, there were 1,565 humans infected with Asian lineage H7N9 reported by the World Health Organization. According to CDC, while the risk posed by H7N9 virus to the public’s health was low, the agency was concerned about its pandemic potential.

Infectious Disease Models

Agencies use infectious disease models to answer a variety of public health questions, including those related to outbreak preparedness and response. A model is a physical, mathematical, or logical representation of a system, phenomenon, or process that allows a researcher to investigate that system, phenomenon, or process in a controlled way. For example, the classic Susceptible-Infected-Recovered or “SIR” model divides a population into three categories: 1) susceptible to the disease, S; 2) infected and infectious, I; and 3) recovered or removed from the infected or susceptible population, R. This model uses equations to determine how many people move between these three categories. The equations contain parameters—numerical descriptors of the disease based, for example, on experiment, expert opinion, or statistics of an ongoing or past outbreak. The equations allow the researcher to estimate how many people are or could be affected by the disease. For example, for past Ebola outbreaks, models estimated that after 40 days, about 44 percent of the population in close contact with infected individuals was susceptible to infection, 31 percent was infected, and 22 percent was recovered. Based on these parameters, equations for transfer between categories, and underlying demographics of the community, an epidemiologist could use the model to estimate how many people within a given town could be susceptible, infected, or removed from the categories of susceptible or infected (due to death or recovery and immunity). Based on model estimates and if a vaccine was available, CDC officials said the decision maker could plan for a specific number of vaccine kits and additional medical staff and supplies to treat infected patients.

Models can also help agency officials anticipate future outbreaks, forecast the spread or severity of a disease, and predict the effects and costs of different intervention options. After an outbreak, models can help sort out what happened, what drove the outbreak, and how it compared to past outbreaks. Other tools are available to accomplish some of these tasks, but models are particularly useful when existing data are not sufficient to answer a given question, or when agencies need to integrate data from disparate sources.

Infectious disease models can be put into two broad categories:

Statistical models. This type of model identifies relationships or patterns that can be used to describe what is occurring or predicts what may occur in the future based on what has occurred in the past. Statistical models tend to use a large amount of data, such as past observed events, to forecast future events, such as disease occurrence, but do not require a fundamental understanding of biological processes or human behavior. They can predict outcomes when causes are not known or understood and when scientific understanding of a disease is limited. They tend to use large amounts of data on past events to forecast future events. Statistical models do not provide full explanations about an infectious disease but may be used when epidemiologists have all or most of the data needed to test a hypothesis. Several benefits can be derived from statistical modeling, including the ability to control for multiple factors that might impact the outcome reviewed, and the ability to isolate the potential effect of infectious disease factors on a particular outcome.

Mechanistic models. Mechanistic models rely heavily on scientific evidence and theory related to infectious diseases, and the understanding of disease dynamics or human behavior from prior knowledge—such as biological processes or interactions between people—to represent known processes. They use basic infectious disease science to inform public health guidance and provide insights into outbreak emergence, spread, and control. For example, population-based models can simulate the course of an epidemic by dividing the population into different categories, such as susceptible, infected, and recovered. Mechanistic models can project the likely course of disease transmission, calculate and predict the effect of proposed interventions, and take into account variable conditions, such as human behavior.

Both statistical and mechanistic models can range from simpler to more complex. A simpler model may, for example, have fewer parameters (inputs) or equations than a more complex model. According to CDC modelers and an expert, a simpler model may be run with a variety of software, ranging from spreadsheet software to more sophisticated software, whereas more complex models are usually run using sophisticated statistical or mathematical programming languages. As a model becomes more complex, it can become harder to describe, recreate, and understand its internal functioning.

Modeling is identified as a beneficial tool in various national plans for disease response and biodefense. These plans do not define the extent to which modeling should occur or how models should be developed for policy, resource allocation, or planning purposes. See table 1 for examples of relevant national plans.

HHS Has Used Infectious Disease Models to Help Inform Policy and Planning

Use of Models to Inform Planning and Policy Decisions

CDC and ASPR use models primarily to answer questions from decision makers. CDC and ASPR officials told us, and documents show, that modeling is one source of information that may inform such decisions, along with sources such as expert opinion, surveillance, other prior work on the disease, and an official’s own knowledge.

CDC modelers and officials said there is no “rule” as to when to use models, and in some situations, it may not be considered useful. For example, CDC did not use modeling when issuing a travel notice for an Ebola outbreak in specific provinces in the Democratic Republic of the Congo, officials said. Instead, CDC based the travel order on an analysis that considered disease incidence and prevalence, public health infrastructure, and the availability of therapeutics, among other things. Similarly, CDC officials responding to Ebola said modeling may be undesirable when it would take too long to engage the necessary external subject matter experts or when modeling would detract from responding to a disease.

CDC and ASPR modelers use models for a variety of purposes. CDC officials said modeling is done differently for each disease, and the amount and type of modeling varies across CDC centers, in part because some centers have less capacity to conduct modeling than others. According to a CDC internal report, the most frequent uses of infectious disease modeling at CDC are: guiding preparedness and response efforts; conducting economic analyses to evaluate the benefits of public health actions, thereby reducing illness and deaths from infectious diseases; understanding pathogen biology, disease transmission, and estimating disease burden; and assessing the effect of interventions and prevention strategies.

ASPR modelers and officials said models have provided information about topics such as: resources, including protective equipment, needed to help respond to an Ebola outbreak; the number of therapeutics and vaccine doses needed to respond to Ebola, both in Africa and domestically; expected U.S. demand for Zika diagnostics; and the number of vaccine doses needed to mitigate the spread of pandemic influenza.

ASPR modelers and officials said modelers tend to serve in a broad role that can include modeling, data analysis, or other tasks. For example, officials said a modeler could provide a team with day-to-day analytic support and not necessarily spend time developing models or use them. Additionally, ASPR maintains a Visualization Hub that can be used for outbreak planning and response, including outbreaks of pandemic influenza and other emerging infectious diseases (see fig. 2).

CDC and ASPR modelers and officials said they generally initiate modeling in response to questions from decision makers. The modelers then work closely with epidemiologists and other subject matter experts to answer the questions. Modeling, according to CDC officials, may be used by individuals or groups within centers, such as division directors, branches, or teams to influence decisions. Who answers a particular question depends, according to ASPR modelers and officials, on the decision maker. Sometimes questions asked will not be within their mission—modelers may suggest such questions be sent to a more relevant agency or part of HHS. CDC and ASPR have modeled to answer a variety of public health questions relevant to Ebola, Zika, and pandemic influenza, and, at times, the results helped inform policy and planning decisions. Modelers and officials provided the following examples:

Planning: ASPR modelers and officials said the bulk of the agency’s modeling is related to the planning, development, and deployment of medical countermeasures. For example, these modelers and officials said many clinical trials for vaccines and therapeutics were planned during the 2014-2016 Ebola outbreak response. As a part of these planning activities, ASPR modelers said modelers developed forecasts of future trajectories of disease incidence under a variety of conditions. These forecasts indicated a significant likelihood the disease incidence in Sierra Leone could decrease to a level that would significantly reduce the success of the trials, according to modelers. Additionally, at the beginning of the 2014-2016 Ebola outbreak response, CDC modelers received modeling questions related to the resources needed to effectively limit the spread of the disease, according to CDC documentation. CDC used models to predict the number of Ebola cases that could be expected over time with and without disease interventions such as Ebola treatment units, community care centers, and safe burials. On the basis of this information and other factors, including a United Nations document on Ebola needs, CDC leadership and other U.S. government officials recommended a rapid increase in Ebola response aid, according to CDC documentation. According to CDC documentation, later analyses demonstrated that this increase helped to greatly reduce the actual number of cases, compared to the likely number if prompt action had not been taken. Additionally, in response to the H7N9 influenza outbreak in 2017, ASPR modeled to determine when doses of influenza vaccine should be delivered and how many doses should be administered in order to mitigate a domestic outbreak. This model found that having a vaccine stockpile could be helpful in preventing disease and that a slow effort to administer an H7N9 vaccine could reduce the vaccine’s usefulness.

Policy: During the Zika outbreak, CDC modelers and officials said they modeled to determine the potential effectiveness of using pesticides to remove insects from aircraft, trains, or ships. According to modelers and agency officials, the issue arose as concern about Zika virus grew, including from other countries and U.S. agencies, like the Department of Transportation and Department of Defense. The model indicated that humans are more likely than insects to transport Zika on airplanes, and officials therefore concluded that the use of pesticides on airplanes would not be an effective intervention. According to CDC modelers and officials, this modeling resulted in an additional sentence being added to World Health Organization policy, which stated that pesticide use was not expected to be effective.

The extent of modeling conducted for Ebola, Zika, and pandemic influenza varied according to the question being asked, along with other factors as follows:

Type of question: CDC and ASPR have used models to answer such questions as who should be prioritized for vaccination or treatment, how transmissible a disease is, and how effective certain interventions are likely to be, according to modelers and agency officials. For example, ASPR modelers and officials said they modeled to help estimate the resources needed to respond to an Ebola outbreak; the number of therapeutics and vaccine doses needed to respond to Ebola, both in Africa and the U.S; and the expected U.S. demand for Zika diagnostics. One ASPR official said that, during the 2009 pandemic influenza outbreak, modeling questions were used to provide decision makers with information on what might happen in a given situation. For example, models were used to provide information related to decisions on early vaccine distribution and how this intervention could affect the potential mortality rate.

Time to model: How soon decision makers needed information also influenced the extent to which CDC and ASPR modeled. For example, if decision makers needed an answer in a week, modelers would inform the decision makers about how much of the answer they could provide within that time frame, ASPR modelers said. Similarly, CDC modelers and officials said that, in one instance, modelers had only 12 hours to provide decision makers with information. Even estimating the time needed to develop and conduct modeling could represent an additional challenge, according to CDC modelers responding to Zika. According to a CDC article on modeling to inform responses to novel influenza viruses, the amount of time required to develop and execute a model can vary from less than a week to more than a month. Agency officials concurred with these time frames.

Personnel and data availability: The availability of qualified personnel was also a factor that affected how much modeling agencies conducted for the selected diseases. For example, CDC modelers and officials said the agency’s Division of Vector-Borne Diseases has focused its resources in other areas, such as building the capacity of states to address vector-borne diseases, and therefore had not invested in individuals with the right skill sets to conduct modeling for the Zika outbreak response. As a result, the division had to call on the three or four CDC modelers from outside of the division who were available to assist with the Zika outbreak response, which limited the amount of modeling that could be performed. Data challenges can also limit the types of modeling conducted. For example, when modeling for Zika, ASPR modelers said they used available information, but data quality and availability limited their ability to model. More data typically become available as an outbreak progresses, but models may be most helpful at the beginning of an outbreak when critical decisions need to be made (see fig. 3).

CDC and ASPR do not keep a list of all modeling conducted, and we therefore cannot quantify the extent of their efforts in terms of a number of models. ASPR modelers and officials said modeling is typically one small aspect of the way the agency carries out its mission. One ASPR official said models are never the sole source of information for decision-making.

According to NIH officials, NIH does not conduct or fund internal modeling for decision-making purposes. NIH’s Fogarty International Center has conducted self-initiated, internal modeling to answer questions generated from research, and from ideas from Center-held workshops. Two NIH institutes—the National Institute of General Medical Sciences and the National Institute of Allergy and Infectious Diseases—along with NIH’s Fogarty International Center have awarded grants for external modeling research for our selected diseases. However, NIH officials said these efforts were intended to advance science, not for policy or outbreak response.

Use of Models to Inform Resource Allocation Decisions

CDC and ASPR modelers and officials said they considered modeling results to a limited extent when making decisions about resource allocation. While modeling can help determine the amount of particular resources needed during an infectious disease outbreak, CDC modelers and officials said it is not central to their resource allocation planning. For example, CDC modelers and officials noted that while a model could inform a decision maker about how many diagnostic testing supplies would be needed based on the range of predicted cases, this would be one input among many into the decision. Decision makers would also consider whether there are other diagnostic test supplies for similar diseases that could be used, the extent of laboratory testing capacity, or the longevity of those supplies.

Models can be used to help plan for the cost of interventions by determining the numbers or types of interventions that can be used during a response to an infectious disease outbreak, according to CDC modelers and officials. It can also help decision makers recognize gaps in their ability to implement resource allocation decisions, according to CDC officials. For example, CDC leadership described how modeling input requirements spurred analysis of the factors limiting hospitals’ use of ventilators during a pandemic influenza outbreak. This work, according to CDC officials, helped determine the number of ventilators that should be included in the national stockpile. While modeling results are important to consider during a public health event, ASPR officials and modelers said it is also important to consider concrete financial estimates based on prior experience and whether recommended medical interventions or countermeasures are available or effective.

For example, ASPR modelers and officials have occasionally been asked to analyze costs for medical countermeasures, but modelers and officials said that few medical countermeasures typically meet the requirements of decision makers, and existing medical countermeasures are typically unavailable for use in a response. ASPR modelers and officials noted that the usefulness of modeling to the decision maker in these instances is limited. In the event that they were asked to model for such questions, ASPR modelers and officials said time would also be a limiting factor in their analysis.

CDC has also developed models to inform decision-making at the state level, specifically to assist state and local public health agencies in developing outbreak response plans. A professional organization of epidemiologists we contacted expressed some concerns with limitations of CDC models, specifically noting that state and local officials viewed CDC models as lacking the level of refinement needed for their state- and local-level planning needs. To follow up, we interviewed officials from a non-generalizable selection of five states based on their reported use of CDC models, the level of selected disease activity in the state, and geographic variation. Two of the five state health departments we contacted reported using one of CDC’s models for Ebola, Zika, or pandemic influenza. These two states confirmed that the usefulness of the CDC FluSurge pandemic influenza model was limited by unrealistic assumptions or a lack of predictive capability, but added that the models were useful to them when considering how to allocate resources or otherwise prepare for a severe pandemic. Officials from one state health department told us they had similar concerns with the CDC Ebola model regarding an unrealistic overestimate of the potential cases, but added that it was useful for informing staff allocation planning as part of their overall response.

Officials from another state health department told us they used CDC’s Zika modeling results that indicated how many emergency room visits they could expect and what symptoms it would take to confirm a Zika infection. At the time, state officials said, commercial testing for Zika was not available, so this modeling was very helpful to health officials looking to recommend who hospitals should test based on the presence of Zika symptoms. State health department officials added that many other factors are considered when deciding on resource allocation, such as local leadership and willingness to embrace the public health response.

Agencies Coordinate Infectious Disease Modeling Efforts but Do Not Fully Monitor, Evaluate, and Report on Coordination

The four HHS agencies that work on infectious disease modeling reported using multiple mechanisms to coordinate their efforts. However, they do not routinely monitor these efforts, evaluate their effectiveness, or report on them to identify areas for improvement.

HHS Agencies Coordinate Infectious Disease Modeling Efforts in Multiple Ways

The four HHS agencies that work on infectious disease modeling—ASPR, CDC, FDA, and NIH—reported using multiple mechanisms to varying extents to coordinate such efforts. For example:

Emergency Operations Center (EOC). During the response to an outbreak, CDC activates its EOC—a temporary, formal organizational structure for coordinating expertise within CDC and among agencies. The four HHS agencies—ASPR, CDC, FDA, and NIH—used EOCs to coordinate modeling efforts during responses to Ebola, Zika, and pandemic influenza outbreaks. For example, during the 2015-2016 Zika outbreak, CDC’s EOC served as the command center for monitoring and coordinating the response by bringing together CDC scientists with expertise in areas such as arboviruses (the category that includes Zika), reproductive health, birth defects, and developmental disabilities. CDC modelers and officials told us that they had weekly strategy meetings and briefings with response leadership within the EOC where they discussed which modeling questions to prioritize. In general, CDC modelers in the EOC were expected to coordinate with modelers from other agencies within and outside of HHS—such as ASPR, FDA, NIH, and the Department of Homeland Security—to produce timely estimates of cases, hospitalizations, and deaths. These estimates can inform response leadership and enable them to assess the speed and impact of the geographic spread of the pandemic. Modelers in the EOC also provide support to decision makers as they examine the potential effects of various response options. These options include when and how to deploy Strategic National Stockpile assets, such as influenza antiviral drugs and mechanical ventilators. We found the use of EOCs to be consistent with leading collaboration practices we have previously identified, such as defining and articulating a common outcome.

Public Health Emergency Medical Countermeasures Enterprise (PHEMCE). The four HHS agencies also participated in PHEMCE, a federal interagency body formed by HHS in 2006 that coordinates the development, acquisition, stockpiling, and recommendations for use of medical products that are needed to effectively respond to a variety of high-consequence public health emergencies. PHEMCE is led by ASPR and also includes partners at the Departments of Defense, Veterans Affairs, Homeland Security, and Agriculture. PHEMCE’s 2017-2018 strategy and implementation plan, its most recent, identified Ebola, pandemic influenza, and emerging infectious diseases more broadly as high-priority threats. PHEMCE leadership could ask modelers to address questions related to these infectious diseases, according to ASPR modelers and officials. According to ASPR officials, such questions tend to support larger response- related efforts, and modeling results are often incorporated into final reports and products. According to ASPR officials, as of February 2020, the PHEMCE structure has been updated and it is unclear how modeling fits into the new structure. We found that coordination through PHEMCE is consistent with leading collaboration practices such as establishing mutually-reinforcing or joint strategies.

Working groups. Modelers with the four HHS agencies have participated in working groups related to infectious disease modeling (see table 2). The use of working groups and similar bodies is consistent with leading collaboration practices that we have previously reported as useful for enhancing and sustaining interagency collaboration, such as identifying and addressing needs by leveraging resources. For example, CDC and ASPR modelers participated in the National Science and Technology Council’s Pandemic Prediction Forecasting Science and Technology Working Group, which facilitates coordination among numerous federal agencies. In 2016, this group produced a report that identified challenges in outbreak prediction and modeling for federal agencies and offered recommendations for federal actions to advance the development and effective application of outbreak prediction capabilities.

Description This interagency working group, directed by the National Science and Technology Council, is responsible for analyzing the state of infectious disease modeling and prediction, and facilitating coordination among numerous federal agencies. According to CDC modelers and officials, as of October 2018, the charter for this group is no longer active, and it meets on a voluntary, ad hoc basis. According to CDC officials, this group connects modelers by holding seminars, managing an email list, and arranging for members to peer review one another’s models. This group had over 160 participants from various centers across CDC, as of June 2019. During the 2014-2016 Ebola and 2015-2016 Zika outbreaks, the Department of Health and Human Services’ (HHS) Office of the Assistant Secretary for Preparedness and Response (ASPR) established temporary modeling coordination groups that brought together government agencies and academics to share early modeling results and discuss pressing questions that could be answered through modeling, according to ASPR modelers and officials. A wide range of entities participated in these groups, including the four HHS agencies, other federal agencies such as the Departments of Defense and Homeland Security, universities, and foreign entities, such as the World Health Organization and the United Kingdom. According to ASPR modelers and officials, there are no plans to convene modeling coordination groups unless there is an ongoing infectious disease outbreak.

Joint model development. ASPR and CDC modelers jointly developed some modeling products during outbreak responses. For example, during the 2014-2016 Ebola response, ASPR and CDC developed a model to estimate future numbers of Ebola patients needing treatment at any one time in the United States. According to a publication describing the model, policymakers have used it to evaluate responses to the risk for arrival of Ebola-infected travelers, and it can be used in future infectious disease outbreaks of international origin to plan for persons requiring treatment within the United States. Building these positive working relationships can help bridge organizational cultures by building trust and fostering communication, which facilitates collaboration and is vital in responding to emergencies. For example, in our 2011 report, we found that, through interagency planning efforts, federal officials built relationships that helped facilitate the federal response to the H1N1 influenza pandemic. Similarly, HHS officials said that federal coordination during the H1N1 pandemic was much easier because of these formal networks and informal relationships built during pandemic planning activities and exercises.

Memoranda of understanding. The four HHS agencies have entered into various agreements through memoranda of understanding in order to define their relationships for coordinating infectious disease modeling (see table 3). Generally these memoranda were between individual agencies rather than department-wide. We found that the use of memoranda of understanding was consistent with leading collaboration practices, such as agreeing on roles and responsibilities. Our prior work found that agencies that articulate their agreements in formal documents can strengthen their commitment to working collaboratively. Similarly, CDC modelers and officials said that written agreements can reduce the possibility of misunderstandings or disagreements and help ensure that participants have a mutual understanding of collaboration goals. For example, in the absence of such written agreements, the potential for duplication is increased because agencies could be working on similar types of models without one another’s knowledge.

Table 3. Selected Examples of Memoranda of Understanding for Coordinating on Infectious Disease Modeling Collaborating agencies The Office of the Assistant Secretary for Preparedness and Response (ASPR) and Centers for Disease Control and Prevention (CDC)

ASPR and the Food and Drug Administration (FDA)

Description From 2013 to 2018, CDC and ASPR had a memorandum of understanding to promote collaboration, provide expertise, and facilitate data and information exchange related to infectious disease modeling. This agreement expired in 2018. ASPR modelers and officials told us that, as of August 2019, it had not been updated, and there were no plans to do so. Despite this, according to CDC modelers and officials, the substance of the agreement is still being followed. CDC modelers and officials told us they continue to collaborate with ASPR modelers on the development of models that address questions of mutual interest. For example, for the ongoing Ebola response, CDC modelers and officials said they have kept ASPR informed on modeling efforts, and ASPR shares data on vaccine production that is included in one of the models. ASPR and FDA have a memorandum of understanding to promote collaboration and enhance knowledge and efficiency by providing for the sharing of information and expertise. This memorandum was in place from 2012 to 2017, and was then renewed in 2019. It remains valid unless modified by consent of both parties or terminated by either party immediately upon written notice in the event that a federal statute is enacted or a regulation is issued by a federal partner that materially affects the memorandum. According to FDA modelers and officials, the agreement facilitates collaboration related to FDA’s Medical Countermeasure Initiative and FDA’s role in supporting the HHS-led Public Health Emergency Medical Countermeasures Enterprise (PHEMCE). FDA modelers and officials told us that the agreement supports the frequent, ongoing collaborations between FDA and ASPR, including collaboration related to preparedness for emerging infectious diseases. However, FDA modelers and officials said, while no specific steps have been taken with regards to collaborating on infectious disease modeling under the agreement, modeling assistance could be provided in the future, if needed.

Description From 2013-2018, ASPR had a memorandum of understanding with NIH’s Models of Infectious Disease Agent Study program to (1) enable Models of Infectious Disease Agent Study program researchers to work with ASPR as part of public health preparedness and response activities, (2) share data and information, and (3) support model development and use in the HHS modeling hub. This agreement has expired. ASPR modelers and officials told us that, as of August 2019, it has not been updated, and there were no plans to do so. Since 2015, CDC has had a memorandum of understanding with NIH’s Models of Infectious Disease Agent Study program, to promote collaboration and facilitate the exchange of data, tools (models), methods, and information. It was set to expire in February 2020. From 2013 to 2018, ASPR had separate memoranda of understanding with the Departments of Defense and Homeland Security to promote collaboration, provide expertise, and facilitate data and information exchange. The goals of the collaboration in both agreements were to explore ways to, among other things: share analytical approaches and efforts, such as modeling and simulation tools, in support of public health preparedness and response activities; provide personnel as needed to facilitate analytical efforts; and share data and information. These goals were similar to those laid out in the agreement between CDC and ASPR. These agreements expired in 2018. ASPR modelers and officials told us that, as of October 2019, they have not been updated, and there were no plans to do so.

Forecasting competitions. CDC and NIH have sponsored formal forecasting competitions to improve modeling for Ebola, Zika, and seasonal influenza. According to a report from the National Science and Technology Council, controlled, multi‐center modeling contests and projects generate valuable insights. For example, they often show that simpler models perform as well as more complex models and that ensemble models, which combine the results of multiple models to predict an outcome, perform better than an individual model. Such competitions are consistent with a leading collaboration practice we previously reported: identifying and addressing needs by leveraging resources. In this case, such leveraging allowed CDC and NIH to obtain additional benefits and insights on models that may not otherwise be available. These modeling competitions can therefore help the HHS agencies better prepare for future outbreaks through coordination with participants. The following are examples of forecasting competitions sponsored by CDC or NIH:

Ebola competition. NIH’s Fogarty International Center held an Ebola forecasting competition from August to December 2015, related to the 2014-2016 West African Ebola outbreak, to compare the accuracy of predictions from different Ebola models, among other things. According to NIH modelers and officials, lessons learned from the challenge were that (1) with regard to short-term incidence predictions, ensemble estimates were more consistently accurate than predictions by any individual participating model; (2) as expected, more accurate and granular epidemiological data improved forecasting accuracy; (3) the availability of contextual information, including patient-level data and situational reports, is important for accurate predictions; (4) the accuracy of forecasting was not positively associated with more complex models; and (5) coordination of modeling teams and comparison of different models is important to ensure robustness of predictions. According to NIH officials, based on these lessons and in response to the most recent Ebola outbreak, NIH has established a coordination group to share information about modeling and data sharing for this particular outbreak and a formal model comparison is underway under World Health Organization leadership.

Aedes (Zika) competition. In 2019, CDC hosted a forecasting competition related to using models to predict the presence of Aedes mosquitoes, which is a vector for the Zika virus. Evaluating these models can, according to CDC, help clarify model accuracy and utility, the seasonal and geographical dynamics of these mosquitoes, and key directions for future research. According to CDC documentation, these advances can contribute to improved preparedness for arboviral invasion in the United States and in other regions where Aedes suitability may be limited and changing. CDC plans to evaluate forecasts for this competition in early 2020, as soon as final surveillance data for 2019 are available.

FluSight (seasonal influenza) competition. CDC holds an annual seasonal influenza forecasting competition—known as FluSight—to facilitate efforts to engage external researchers to improve the science and usability of seasonal influenza forecasts. The results of the competition are evaluated by the CDC Influenza Division, which works with state and local partners to determine whether the results are useful to them and if there are other metrics, milestones, or targets that would be more helpful in making public health decisions. According to CDC officials in February 2020, the results from the FluSight competition are not directly incorporated into pandemic influenza forecasting because the most accurate seasonal influenza forecasts would not necessarily be the most accurate pandemic influenza forecasts. According to these officials, the overall lessons learned from the FluSight competition relate to how to quantify, visualize, and communicate model results and model accuracy, as well as the value of forecast ensembles to summarize multiple models. CDC officials said these lessons are incorporated into pandemic influenza forecasting plans.

Coordination with academic and other modelers. CDC coordinated infectious disease modeling efforts with academic and other modelers through various means, including the following: Intergovernmental Personnel Act agreements. CDC has used agreements under the Intergovernmental Personnel Act of 1970 to collaborate with external experts on modeling efforts. For example CDC’s Division of Vector-Borne Diseases had an agreement from 2014 to 2017 to assign a CDC official to the Harvard T.H. Chan School of Public Health. The agreement was to help CDC integrate with a larger modeling community and provide the Harvard School of Public Health with expertise in arboviral diseases and applied public health.

Vector-Borne Disease Centers of Excellence. CDC has funded the Vector-Borne Disease Centers of Excellence, which are engaged in modeling-specific projects. In 2017, CDC established five universities as regional centers of excellence to help prevent and rapidly respond to emerging vector-borne diseases across the United States. According to CDC, the goals of the centers are to build effective collaboration between academic communities and public health organizations at federal, state, and local levels for surveillance, prevention, and response, among other things.

Support for other governmental entities. CDC has coordinated with other entities—such as state and local officials—to provide modeling tools, estimates of case counts, or effects of interventions during the Ebola, Zika, and pandemic influenza outbreaks. For example, CDC developed pandemic influenza models for state and local health departments to use in influenza pandemic planning activities. The tools are available on the CDC pandemic influenza website and from ASPR’s emergency preparedness information portal. As previously discussed, officials from two of the states we spoke with said they generally were unaware of the availability of the models. According to CDC modelers and officials, these models were developed in the mid- 2000s for pandemic influenza planning and remain useful but had not been a priority to update because they have not received a request to do so.

Informal collaboration. CDC has engaged in a range of informal collaborations related to infectious disease modeling. According to CDC modelers and officials, modelers often develop relationships through conferences or other contacts. For example, CDC modelers and officials said they informally collaborated on Ebola modeling needs with academic institutions, as well as modelers and analysts in the World Health Organization and other U.S. government agencies, such as the Federal Emergency Management Agency. For example, CDC modelers and officials told us that model estimates produced under collaboration with academics helped inform decisions about how many beds to be ordered and delivered on the ground in West Africa during the 2014-2016 Ebola Outbreak. Similar to the forecasting competitions described above, such informal coordination mechanisms are consistent with the best practice of identifying and addressing needs by leveraging resources, thus obtaining additional benefits that may not be available if they were working separately. For example, we have previously reported that informal collaboration mechanisms—such as building relationships between key personnel and soliciting input for research projects—can provide the opportunity to leverage expertise.

HHS Agencies Do Not Fully Monitor, Evaluate, and Report on Coordination Efforts

CDC and ASPR modelers and officials did not routinely monitor, evaluate, and report on coordination efforts for infectious disease modeling. While CDC did conduct after-action reviews for Ebola and Zika, which included a review of modeling efforts, such reviews are not routine outside of a response and do not examine modeling coordination between agencies. ASPR modelers and officials told us they saw no reason to monitor coordination efforts under the memorandum of understanding with CDC because such memoranda outline expectations rather than requirements. However, we have found that agencies that create a means to monitor, evaluate, and report the results of collaborative efforts can better identify areas for improvement. We have previously reported that progress reviews or after action reviews can be useful mechanisms for monitoring, evaluating, and reporting on collaborative efforts. For example, we previously reported that, to monitor, evaluate, and report on the status of achieving the Healthy People 2010 objectives, HHS held progress reviews in which the federal agencies with lead responsibilities for a focus area reported on the progress towards achieving the objectives. During these reviews, the participating agencies discussed the data trends, barriers to achieving the objectives, strategies undertaken to overcome barriers, and alternative approaches to attain further progress. By holding similar progress reviews in which CDC and ASPR evaluate and report on coordination efforts for infectious disease modeling, these agencies could be better positioned to identify and address challenges prior to infectious disease outbreaks occurring, which could lead to improved responses. Further, there is the potential for overlap and duplication of modeling efforts across agencies, which may not be identified if coordination efforts are not effectively being monitored, and which could lead to inefficiencies.

The memorandum of understanding between CDC and ASPR had expired in 2018. Agency officials told us they had no plans to review or update the agreement. According to ASPR modelers and officials, the agreement has not been updated because it was not a priority and the substance of the expired agreement is being followed. However, without an active agreement in place that clearly defines the goals of the collaborative effort and the roles and responsibilities of participants, a lack of understanding and agreement becomes more likely, particularly as agencies’ priorities evolve over time. Our prior work on leading collaboration practices found that agencies that articulate their agreements in formal documents can strengthen their commitments to working collaboratively, and that such agreements are most effective when they are regularly reviewed and updated.

Further, we found that the memorandum of understanding between ASPR and CDC was not fully implemented when it was active. For example, according to this agreement, CDC was to appoint a designee to participate in a steering committee related to modeling within HHS. However, ASPR modelers and officials told us that this steering committee was never formed because of changing leadership and priorities. They told us that HHS does not have any intention to form such a steering committee in the future. However, our past work shows creating a steering committee or other similar coordination mechanism could help facilitate monitoring of coordination efforts.

We similarly found that other memoranda of understanding related to infectious disease modeling were not fully implemented. For example, although ASPR had a 2013-2018 memorandum of understanding with NIH’s Models of Infectious Disease Agency Study program, ASPR modelers and officials said they rarely use models funded by NIH, including those funded through the program. In particular, ASPR modelers and officials recalled only using one such model in recent years. That model, known as “FluTE,” is an influenza model that was used as part of a larger study on vaccine availability. However, ASPR modelers faced challenges in using this model. Specifically, these ASPR modelers and officials said the FluTE model initially was not compatible with ASPR’s computer system, so software engineers had to modify the source code to resolve the compatibility issue. The model did not have documentation describing its parameters, according to ASPR modelers and officials, so they had to read through the model’s source code to understand them. Similarly, regarding a separate agreement between ASPR and FDA, FDA modelers and officials said that, while there is ongoing information sharing, no specific steps have been taken with regard to collaborating on infectious disease modeling under the agreement. However, these modelers and agency officials said that modeling assistance could be provided in the future, if needed.

CDC and ASPR Generally Followed Identified Practices for Infectious Disease Modeling, but CDC Has Not Fully Ensured Model Reproducibility

We identified four elements of practices for developing and assessing models: (1) communication between decision maker and modeler, (2) description of the model, (3) verification, and (4) validation. We determined that CDC and ASPR generally followed these GAO-identified practices for 10 models we reviewed. However, for four of the 10 models, CDC modelers did not provide all of the details needed in the verification steps to reproduce their model results, which is inconsistent with HHS guidelines on transparency and reproducibility.

CDC and ASPR Generally Followed Identified Modeling Practices but Did Not Always Fully Assess Model Performance

According to our interviews with agency modelers and experts, along with our review of selected literature, there are no documented standards that prescribe the steps agencies must or should follow when developing and assessing models. However, based on our interviews and review, we identified four broad elements of the modeling process that modelers generally consider. They are: 1 communication between modelers and officials to refine questions to be addressed by the model, such as geographic spread of the disease and total cases of the disease; 2 description of the model, including detailed descriptions of assumptions and data sources used; 4 validation.

Figure 4 outlines the model development and assessment process.

Based on our assessment of 10 selected models, we found that CDC and ASPR generally took steps that corresponded to our four elements, and agency modelers generally agreed with our assessment of each model. See table 4 for more information on the elements. See appendix III for a list of models we reviewed and a complete list of the steps we identified that make up each element.

Communication between modeler and decision maker. In all 10 agency models we reviewed, we found that agencies took all the steps we identified for communication between decision maker and modeler. In some cases, these steps were formalized, while in others they were informal. For example, CDC modelers responding to Ebola ensured communication with decision makers by following a memo template they developed, which has a section requiring modelers to communicate key aspects of their model. These modelers noted, however, that they would not follow all the steps in their memo template for models developed during an outbreak because of time constraints. CDC modelers responding to pandemic influenza noted they do not have formal best practices for communication about key model aspects to decision makers, and a CDC modeler responding to Zika highlighted the role of CDC’s Emergency Operations Center (EOC) in communication between decision makers and modelers, which is activated only during a response. ASPR modelers noted that—as a best practice—they hold a discussion for all new models, in which decision makers describe what they are looking for and modelers describe what they can provide.

Description of the model. In nine of the 10 models we reviewed, modelers took all steps we identified for describing their model type, inputs, outputs, assumptions, and limitations. In one case, ASPR’s “flumodels” package, the agency did not carry out the step of describing the model’s limitations. ASPR modelers told us they did not do so because they expected the model’s intended users—primarily federal public health modeling experts—would understand the limitations of their model, an assumption we find reasonable.

Verification. In six of 10 models reviewed, we found agency modelers followed most of the steps we identified for model verification. However, in four of the seven CDC models reviewed, CDC did not publish the model’s code, a part of model reproducibility and a model verification step. We examine CDC’s policy and efforts on reproducibility in more detail below.

Validation. For four of the 10 models we reviewed, agencies performed few validation steps. In all three CDC pandemic influenza models we reviewed, and the ASPR Zika model, sensitivity analysis was the only validation step performed. CDC influenza modelers said they did not perform other validation steps because of a lack of comparable external models or applicable data which could be used for other types of model validation. For example, they said they could not validate their models using real-world data because they made projections for scenarios that did not come to pass (e.g., an unmitigated pandemic influenza outbreak). They said they have continued to look for comparable models that could be used to cross-validate their model estimates. ASPR modelers responding to the Zika outbreak also did not have access to comparable external models or applicable data to confirm their model projections, but have since attempted to validate their model. For the other six models we reviewed, agencies carried out most but not all validation steps. For example, CDC modelers responding to Zika also said they did not perform cross-validation (comparison of different model results to each other) for their Zika model because of a lack of comparable models. However, these ASPR and CDC Zika modelers said they have attempted to validate their model since its publication as new data emerges, and we found this occurred.

Assessing Model Validity Assessing model validity means determining whether a model is sufficiently accurate for its purpose. Several methods are available, including the following:

Modelers can compare the results of the model against real-world data the model was designed to predict. If there are no such data, another method is to determine how much the model projections change in response to changes in input data. This is known as model sensitivity analysis.

Modelers can also withhold a part of the available data in building the model and then confirm the model can reproduce the withheld data. real-world data is to run the model along with a separate, independent model using the same input data, and comparing the outputs.

CDC modelers and ASPR modelers responding to Zika followed identified practices and validated their model projections for the Zika outbreak, although their efforts yielded mixed results for model performance. CDC modelers responding to Zika attempted to estimate whether there was an enhanced risk of microcephaly in infants born to expectant mothers infected with Zika. Using data available during the initial stage of the outbreak, they calculated the enhanced risk to be between 0.88 and 13.2 percent if the mother was infected in the first trimester. In two subsequent studies using later data on the actual incidence of microcephaly as a result of the outbreak, other researchers found the enhanced risk was within the bounds of CDC modelers’ earlier projections: a 10 percent enhanced risk in one study and an 8.3 percent enhanced risk in the other. In the second case, ASPR modelers attempted to estimate potential new cases of Guillain-Barré syndrome, a rare disorder in which the body’s immune system attacks part of its own nervous system, in places burdened by Zika infection. Their initial projections were that there would be between 191 and 305 new cases in Puerto Rico, a three- to five-fold increase above the number normally expected. ASPR modelers attempted to verify these results themselves and found that the incidence did increase, but only two-fold, to 123 new cases. through independent performance evaluations. For example, agencies sometimes host modeling competitions, in which independent modelers compare the predictive performance of multiple models under controlled conditions using standardized data. The National Institutes of Health hosted an Ebola forecasting competition in 2015, and the Centers for Disease Control and Prevention (CDC) launched its FluSight competition in 2013.

The Challenge of Modeling During an Outbreak. Early in the 2014-2016 Ebola outbreak, Centers for Disease Control and Prevention (CDC) officials faced the challenge of answering questions with limited data and time. In order to estimate the potential number of future cases and to aid in planning for additional disease-control efforts, CDC developed EbolaResponse, an Excel spreadsheet-based model that could forecast how interventions would impact the outbreak. Using EbolaResponse, CDC predicted in early September 2014 that 1.4 million cases of Ebola could occur in Liberia and Sierra Leone by January 2015, if the world health community did not increase interventions. These estimates included a correction factor intended to account for the underreporting of cases and that, according to officials, was to represent model uncertainty. Partly because of these estimates of rapidly increasing cases, CDC and others increased intervention by sending more treatment units, personnel, and medical supplies in late 2014. EbolaResponse was created to model the effects of intervention, and it later turned out to be unreliable for the 4-month forecast that CDC used to support its request for increased intervention. Independent analysis found that the model could forecast cases up to a month ahead well but could not provide any measure of uncertainty. Furthermore, the model was unable to make accurate forecasts much beyond 3 months, a limitation that was common among the models used during the outbreak. CDC later reported that roughly 8,500 cases, or 34 percent of the corrected EbolaResponse prediction of 25,000 cases, occurred in Liberia by the end of January 2015.

We also found that CDC and ASPR modeling approaches varied somewhat, while generally remaining within the bounds of our identified practices. For example, all the agency modeling groups reviewed their model assumptions, but they also varied in whether this review was formal or informal and internal or external. CDC modelers responding to Ebola use a formal internal peer review process during non-outbreak periods, as well as a detailed checklist to ensure communication with decision makers, full consideration of model inputs and outputs, quantification of model uncertainty, and validation of the model. By contrast, CDC modelers responding to Zika told us they do not have a formal system for evaluating their models, and instead rely on their own review of model assumptions. ASPR and CDC pandemic influenza modelers told us their modeling approach also relied on peer review, but the review was done by external experts; informally for ASPR and formally for CDC pandemic influenza modelers.

There are several reasons agency modeling approaches can vary. According to agency modelers, agency modeling practices can be influenced by the availability of time, data, and comparable models. For example, CDC pandemic influenza modelers and officials said they follow a shortened process when facing time constraints by documenting model development in a journal publication after the model has already been put to use. Similarly, CDC modelers responding to Ebola noted that, during a response, a lack of time may mean models are not reviewed through CDC’s formal clearance process; instead, a more informal review of model results may occur.

CDC and ASPR modelers also described variation in the complexity of the models they use. They said they sometimes use both simple and complex models for the same disease and during the same outbreak. CDC modelers and officials responding to Ebola said that they preferred models run in spreadsheet programs for their transparency and communicability, whereas CDC influenza modelers mostly use dedicated statistical software programs to run models and spreadsheets for communicating with state and local health departments. ASPR modelers develop more complex prediction models so that they can be reused to answer more than one question, as opposed to models run in spreadsheet programs that are designed to answer one question.

Experts and agency modelers generally agreed that infectious disease models should not be more complex than is necessary to answer the questions they were developed to address. A simpler model may be run on a variety of software programs, ranging from spreadsheet programs to specialized programming languages that can do statistical analysis. One downside of models run in spreadsheet programs, according to CDC influenza modelers, is that it is harder to conduct quality control measures. Two experts we spoke to, along with CDC Zika modelers, also expressed concerns with reliability and reproducibility of models run in spreadsheet programs.

CDC Has Not Fully Implemented a Policy to Ensure Model Reproducibility

Since 2002, HHS agencies responsible for disseminating influential scientific, financial, or statistical information have been required to ensure methods used to develop this information are “reproducible.” A 2019 report from the National Academies of Sciences, Engineering, and Medicine noted that the scientific enterprise depends on the ability of the scientific community to scrutinize scientific claims and to gain confidence over time in results and inferences that have stood up to repeated testing. As part of this process of scrutiny, a study’s data and code should be made available so that the study is reproducible by others. The National Academies report defines reproducibility as obtaining consistent computational results using the same input data, computational steps, methods, code, and conditions of analysis. Reproducibility is specifically addressed earlier in this section in our discussion of model verification, a step that requires making code available for independent review.

HHS requires its component agencies to either follow HHS department guidelines on reproducibility or to ensure their own guidelines include a high degree of transparency about the data and methods used to generate scientific information. HHS guidelines require that, in a scientific context, agencies identify the supporting data and models for their published scientific information and provide sufficient transparency about data and methods that an independent reanalysis could be undertaken by a qualified member of the public. When asked whether CDC has specific policies related to reproducibility that would have applied to provision of model code in their published scientific research, CDC referred to its guidelines developed in response to the 2002 HHS Guidelines. However, CDC guidelines do not contain any reference to reproducibility, models, or provision of model code. CDC guidelines for review of scientific information provided to the public focus on completeness, accuracy and timeliness, data management and analysis, clarity and accuracy of presentation, and validity of interpretation of findings. CDC’s policy on public health research and non-research data management and access does not make any reference to reproducibility or model code. This lack of reference to reproducibility in CDC’s guidelines and policies is not in accordance with HHS guidelines.

Our review found four instances in which CDC modelers did not provide model code when they published their models. CDC modelers said in some instances, issues with publication formats made the code difficult to share, they did not have time to produce a user-friendly version of the code, or they would share the code upon request.

By contrast, ASPR modelers provided code for every model within our review when they published their models. While neither agency cited a specific HHS policy that required them to share model code, ASPR modelers noted that their internal peer review process typically includes sharing model source code with other modelers within PHEMCE. In our review of HHS guidelines and agency-specific guidance for these HHS guidelines, we found that, of three published agency guidance, two require reproducibility, or transparency for the methods used in the reports they issue to the public. Of these agencies, CDC was the only one that did not explicitly require transparency or reproducibility.

The National Academies report noted that researchers have to be able to understand others’ research in order to build on it. This report also notes that the ability of qualified third parties to reproduce a model using published code is important because it can reveal mistakes in model code, which can lead to serious errors in interpretation and reported results. If researchers do not share an important aspect of their study, such as their model code, it is difficult to confirm the results of their research and ultimately produce new knowledge. One agency official acknowledged the importance of releasing model code, noting that HHS could benefit by ensuring policies across the agency are consistent regarding reproducibility and transparency in modeling. By not specifically addressing reproducibility in their policy on dissemination of scientific information, CDC risks undermining the reliability of the scientific information they disseminate to the public.

Modelers Faced Several Challenges and Have Worked to Address Them

Based on our review of documents and reports from agencies, as well as expert and agency interviews, we identified three categories of challenges that CDC modelers and officials and ASPR modelers faced when modeling for Ebola, Zika, and pandemic influenza, along with steps they took to address the challenges. The categories are data, resources, and communicating results.

Data Challenges

According to a 2016 report from the National Science and Technology Council (NSTC), obtaining timely and accurate data and information has long been a major challenge to an effective response during an infectious disease outbreak. One expert described reliable data as a modeler’s most limited resource. Until data of sufficient quality and quantity are available and usable, the predictive value of models will be limited.

Agency modelers and officials provided examples of data-related challenges, which we categorize as follows:

Data Access. Public health data, according to one expert, often has access restrictions. For example, ASPR modelers said their ability to access data during the 2014-2016 Ebola outbreak was reduced by a need to enter into agreements with data-owning countries in order to obtain patient data. Modelers said there were agreements between CDC and data owners, but further agreements would have been required for ASPR to obtain data because the agreements did not authorize CDC to share data with its partners. In addition to the example above, the lack of data sharing agreements during the 2014- 2016 Ebola outbreak response led to modeling projects being delayed, according to a CDC publication. ASPR modelers said their inability to obtain data without a data-sharing agreement made it challenging for them to developing a current, reliable estimate of Ebola incidence before modelers could start creating future estimates of disease incidence. They said that, as a result, they instead developed a statistical model, which provided less reliable estimates of future numbers of disease cases than they would have preferred. Modelers said they worked to address this challenge by obtaining data and indirect information through personal relationships with other modelers. In addition to the example provided above, CDC modelers and officials responding to Ebola described experiencing data access challenges.

Data availability. Without sufficient data, models may be unable to identify an epidemic’s key drivers, which could result in misdirected intervention efforts. For example, ASPR modelers noted that during the 2015-2016 Zika outbreak response, there were substantial limits on available data, and data that were available could be unreliable and delayed. They said it was very difficult, and in many cases effectively impossible, to determine the accuracy of forecasting models for the evolving Zika outbreak. In addition, CDC officials and modelers responding to Ebola, Zika, and influenza described encountering limits on available data as an ongoing challenge. Steps that modelers said they have taken to address data availability challenges include designing models to use a minimum amount of data, building trust and communication with stakeholders who might be able to provide additional data, and updating data systems to provide all available information. According to CDC modelers, data availability will likely continue to pose a challenge to public health responses.

Data collection. There is limited manpower during an infectious disease outbreak response, which can limit the health care system’s ability to collect data, according to CDC modelers and officials responding to Ebola and ASPR modelers. ASPR modelers said if a provider has to fill out a time-consuming form, then they will be delayed in treating the next patient. In order to address this challenge, CDC modelers and officials and ASPR modelers said data requesters should ask for the minimum amount of data needed. For example, CDC modelers and officials said they focus on understanding what data are essential, how they are collected, and the policy implications of reporting those data. A 2016 NSTC report recommended the federal government address this challenge by identifying questions likely to arise during an outbreak response, in order to help define and prioritize data collection and modeling goals.

Data quality. Experts said creating models with low-quality data can result in inaccurate models that may not provide clear answers to decision maker questions. For example, CDC modelers and officials responding to the 2015-2016 Zika outbreak said the data quality varied, based on many factors such as surveillance systems that were doing different things and defining reporting Zika cases differently, and the availability of diagnostic testing. Because of data quality concerns, there were questions about whether modeling could be conducted, but through discussions modelers and agency officials said they were able to address challenges. To address such challenges, CDC modelers and officials responding to Zika said they worked to improve public data sharing, sent an official to the Pan- American Health Organization to help interpret data and understand the outbreak from an international perspective, and used modeling methods appropriate for data with high levels of uncertainty. In addition to the example provided above, CDC modelers and officials responding to Ebola, ASPR modelers, and experts described experiencing data quality challenges.

Data integration. CDC modelers and officials responding to Ebola and Zika also faced the challenge of integrating multiple data sets, which may not be standardized or in a readily usable form. For example, CDC modelers and officials responding to Zika found it challenging to integrate data as the definition of the disease was refined over time. As the definition got more specific and monitoring systems became available, it was hard to establish data trends, these officials said. Further, there were variations in who would be tested, with all people who exhibited symptoms being tested in some areas, and only pregnant women in others, and also when data would be placed into a combined form and reported to state, national, or international officials, according to these officials. This integration issue may have complicated efforts to conduct modeling such as determining the risk of microcephaly in infants over time. In order to address this challenge, Zika modelers said they set up an online data repository to, among other things, standardize shared data.

Resource-Related Challenges

CDC modelers and officials responding to Ebola and Zika, along with experts, said finding staff with sufficient training to support modeling during an infectious disease outbreak represented an ongoing challenge. For example, CDC modelers responding to Zika said it can be difficult to find modelers with both an epidemiological background and skills in coding and mathematics. Modelers and agency officials said those who had the correct skills were in high demand, and it was difficult to fully engage them in the Zika outbreak response. They said they could have conducted more modeling or completed modeling efforts more rapidly if they had had access to more modelers with the right skills. To address this challenge, modelers participate in trainings on how to communicate what models can and cannot do, participate in working groups that support modeling efforts, employ the Intergovernmental Personnel Mobility Act Program, maintain collaborations with external partners, and host students and researchers. ASPR modelers said they faced personnel challenges in their modeling efforts but that they were wide-ranging and not specific to Ebola, Zika, or pandemic influenza.

According to a 2016 NSTC report, time constraints make it challenging for researchers to keep up with scientific literature during an outbreak. CDC influenza modelers said they faced this challenge and that they conduct weekly searches for new influenza publications, which normally identify about 150 publications each week. To address this challenge, modelers said they conduct literature searches, share the responsibility of reviewing publications and informing others of their content, talk to experts, and attend conferences. Modelers said this challenge was more easily addressed than others.

Communication Challenges

Communicating model results can be difficult and, as modelers and agency officials pointed out, decision makers will not give credence to results from a model they do not understand. Model results, according to CDC influenza modelers, are often nuanced and complicated, and officials have to think about what pieces of information are the most important to convey to a decision maker, the public, or health officials. Furthermore, as one expert noted, the complexities of modeling can get lost in translation, especially with the media, which may focus on only a worst-case scenario. When modeling for infectious diseases, appropriately communicating complex information has been described as a constant challenge, and CDC influenza modelers described it as their biggest challenge. CDC influenza modelers particularly noted the challenge of communicating uncertainty. CDC influenza and ASPR modelers said if decision makers did not understand the models, they could misunderstand the results, which, according to ASPR modelers, could lead to errors in decision making. CDC modelers and officials responding to Ebola and Zika, CDC influenza modelers, ASPR modelers, and experts described experiencing challenges communicating model results to decision makers.

Clear communication may help prevent misunderstandings. For example, one review article said officials may not understand what models can and cannot do before an epidemic, and modelers may not be fully aware of a decision maker’s needs. An expert said there is a need to constrain the use of models intended to inform decisions so that the model does not over- or under-influence a decision maker. And, according to ASPR modelers, decision makers sometimes want a model to make a decision for them, although models can only inform the decision making process.

They said this is less of a problem during an outbreak response, when decision makers know they have to act based on incomplete information.

Some steps officials described taking to address communication challenges were similar across CDC and ASPR officials. For example, CDC modelers and officials and ASPR modelers said they took steps to improve communication, such as working to develop relationships outside of an outbreak and to improve how data are visualized. For example, ASPR modelers and officials said they provided decision makers with a website that displays an interactive influenza model known as ShinyFlu. The website lets users adjust a model to see how its results could change based on its inputs used. However, modelers said this only works if the decision maker is willing to engage with data.

Other steps to address communication challenges were not discussed by all modelers we spoke to. For example, ASPR modelers said that, when they use models with high uncertainty, they do additional research to assess and communicate how a model could be misrepresenting a real- world problem. Additionally, CDC modelers responding to Zika and CDC influenza modelers said they sometimes use the language of weather forecasting—which provides information on the risk of an event occurring over a specified period of time—to help communicate model outcomes.

For all 10 of the models we reviewed, modelers communicated all the information they had agreed to provide to decision makers, including information about model uncertainty. Agency modelers and officials said they provided this information through discussions with decision makers and by showing decision makers the results of multiple modeling situations to convey uncertainty.

Conclusions

Infectious disease modeling is one tool that can provide decision makers with valuable information to support outbreak preparedness and response. In particular, modeling can help answer questions that are difficult to address in other ways because of practical, ethical, or financial reasons. Federal agencies have recognized the importance of modeling. CDC and ASPR reported using it to inform policy and planning questions and, to a more limited extent, to inform planning and the use of resources.

HHS agencies that work on infectious disease modeling—ASPR, CDC, FDA, and NIH—reported using multiple mechanisms to coordinate their modeling efforts, including working groups, memoranda of understanding, and coordination with academic and other external modelers. The use of these mechanisms was consistent with many leading collaboration practices, such as defining and articulating a common outcome and addressing needs by leveraging resources. However, HHS does not routinely monitor and evaluate its coordination efforts, as called for by another leading collaboration practice, which limits the department’s ability to identify areas for improvement. Further, there is the potential for overlap and duplication of modeling efforts across agencies, which may not be identified if coordination efforts are not effectively being monitored, and could lead to inefficiencies. By holding progress reviews in which CDC and ASPR evaluate and report on coordination efforts for infectious disease modeling, these agencies could be better positioned to identify and address challenges prior to infectious disease outbreaks, which could lead to improved response efforts.

CDC and ASPR modelers generally followed GAO-identified modeling practices, with the notable exception of model verification. Specifically, CDC did not make model code available to others for four of the seven CDC models we reviewed. HHS does not have a policy that requires its agencies to share model code, but it does require its component agencies to either follow its guidelines or ensure that their own guidelines include a high degree of transparency to facilitate reproducibility by qualified third parties. Without sharing code and other important information, CDC cannot ensure that its models are reproducible, a key characteristic of reliable, high-quality scientific research.

Recommendations for Executive Action

In order to facilitate HHS infectious disease modeling efforts, we are making two recommendations.

The Secretary of Health and Human Services should develop a mechanism to routinely monitor, evaluate, and report on coordination efforts for infectious disease modeling across multiple agencies. (Recommendation 1)

The Secretary of Health and Human Services should direct CDC to establish guidelines that ensure full reproducibility of CDC’s research by sharing with the public all permissible and appropriate information needed to reproduce research results, including, but not limited to, model code. (Recommendation 2)

Agency Comments and Our Evaluation

We provided a draft of this report to the Department of Health and Human Services (HHS) for review and comment. In its comments, reproduced in appendix IV, HHS agreed with our recommendations and noted that it was developing a process to coordinate its infectious disease modeling efforts across its components.

With regard to our second recommendation—that HHS should direct CDC to establish guidelines that ensure the full reproducibility of CDC’s research by sharing all permissible and appropriate information needed to reproduce research results, including, but not limited to, model code— HHS’s comments indicated that CDC believes it has already completed actions to implement this recommendation.

For example, the HHS comments state that CDC has established policies such as “Public Access to CDC Funded Publications” and “Policy on Public Health Research and Nonresearch Data Management and Access” that ensure that results are made available to the public, as appropriate. However, as we state in our report, these policies do not contain any reference to reproducibility, models, or provision of model code and therefore do not fully address our recommendation.

CDC also said in the HHS comments that its methods—including its practice of providing a copy of model code upon request—are in line with standard practice in the scientific community and peer- reviewed journals. However, in the four instances we identified where CDC modelers did not share code, code being available upon request was only one of the reasons cited. Further, this practice is inconsistent with those of the other HHS agencies we reviewed, and may limit the ability of external researchers to confirm the results of CDC’s research and ultimately produce new knowledge.

As noted in our report, by not specifically addressing reproducibility in its policies on access to data and publications, CDC risks undermining the reliability of scientific information disseminated to the public. Therefore, we did not change our recommendation in response to HHS’s comments. We did, however, revise our report to include information on other HHS agency policies related to reproducibility.

HHS also provided technical comments, which we incorporated as appropriate.

As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies of this report to the appropriate congressional committees, the Secretary of Health and Human Services, and to other interested parties. In addition, this report will be available at no charge on the GAO website at http://www.gao.gov.

If you are your staff have questions about this report, please contact Timothy M. Persons, Chief Scientist, at (202) 512-6888 or personst@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix V.

Appendix I: Objectives, Scope, and Methodology

In conducting our review of infectious disease modeling by the Department of Health and Human Services (HHS) agencies, our objectives were to (1) examine the extent to which HHS has used various types of models to inform policy, planning, and resource allocation for public health decisions for selected infectious diseases, (2) examine the extent to which HHS coordinated their modeling efforts for selected infectious diseases, (3) examine the steps HHS generally took to develop and assess the performance of its models for the selected diseases and steps it applied to a selection of infectious disease models, and (4) describe the extent to which HHS has addressed challenges related to modeling for selected infectious diseases.

For purposes of this review, we focused on HHS because of its focus on scientific and technical issues related to disease modeling, role in infectious disease outbreak preparedness and response activities, and use of modeling for policy and regulatory issues related to disease. Within HHS, we identified four agencies—HHS’s Office of the Assistant Secretary for Preparedness and Response (ASPR), the Centers for Disease Control and Prevention (CDC), National Institutes of Health (NIH), and Food and Drug Administration (FDA)—which may develop or use infectious disease models.

To inform all four objectives, we selected three naturally-occurring infectious diseases that have pandemic or epidemic potential—Ebola virus disease (Ebola), Zika virus disease (Zika), and pandemic influenza—to use as examples of broader infectious disease modeling efforts. We selected these diseases based on document review, their inclusion on NIH’s pathogen priority list, modeling being conducted by HHS agencies, and interviews with experts that we selected based on their experience with infectious disease. Based on these steps, the team selected diseases that fit into one of the three categories on NIH’s pathogen priority list: the disease (1) can be transmitted easily from person to person, resulted in a high mortality rate and had the potential for major public health impact, might cause social disruption, and may require special action for public health preparedness (Ebola), (2) was moderately easy to disseminate, and required specific enhancements for diagnostic capacity and enhanced disease surveillance (Zika), or (3) was an emerging pathogen that could be engineered for mass dissemination in the future because of availability, ease of production and dissemination, and have the potential for high morbidity and mortality rates and major health impacts (pandemic influenza).

HHS Use of Models to Inform Policy, Planning, and Resource Allocation Decisions

To examine the types of models developed by HHS agencies to inform policy, planning, and resource allocation decisions, we reviewed documents from 2009—the year of the last pandemic influenza outbreak in the United States—to April 2019 to identify examples of models developed by the agencies for the three selected diseases. For context on and examples of the types of modeling that CDC and ASPR have conducted, we reviewed published articles that CDC and ASPR officials and experts provided to us or cited during the course of our review, such as articles identified during interviews which we later obtained. We also obtained selected internal memoranda, when available, that described models used in the Ebola virus outbreak. We did not include FDA and NIH in this review because FDA has a limited role in modeling, and NIH generally funds, rather than conducts, modeling. This review yielded articles and memoranda describing about 60 CDC and ASPR models. See appendix II for a bibliography of model publications reviewed. We then categorized the models using categories derived from a federal working group report to characterize the types of modeling conducted and the purpose of the modeling, when that purpose was identified.

To analyze each study, one analyst initially coded each study, and each classification was then independently reviewed to verify that it had been correctly classified and to resolve any categorization discrepancies. We used these categories to describe types of modeling efforts undertaken by HHS agencies. Because we focused on studies published between 2009 and 2019, our findings are not generalizable to models that were developed outside of that time period. Additionally, because we relied on agency officials or reviews of relevant agency documents and publications to identify studies, we may not have captured all studies relevant to our scope. Further, because CDC and ASPR modelers and officials said that they do not publish every model they conduct, our review was not intended to develop an inventory of the modeling conducted during the time period. Therefore, we were unable to determine the extent to which the models we identified represented agency modeling efforts as a whole.

To describe the extent of model use for public health decision making, we interviewed officials from HHS agencies identified as decision makers for conducting the response to these selected diseases—CDC, ASPR, and FDA—and officials who conducted the modeling. We also interviewed two NIH institutes and one center about funding for research related to modeling for the selected diseases. Additionally, we conducted semi- structured interviews of officials from five states concerning their use of models prepared by HHS agencies for decision making, among other topics. We selected these states based on a review of a CDC draft report on states’ use of CDC models, on the level of influenza activity experienced by states, and consideration of geographic variation by U.S. region. During our review, we sought to identify the common types of decisions that could be informed by models, as well as the considerations that could impact the extent to which a decision maker requests and uses models for specific types of decisions. Based on interviews with agency officials and our review of HHS models we identified examples of models that were used to make specific decisions during response and non- response times. Because we relied on officials to describe the extent to which models inform decision making, we may not have captured all relevant instances when models for the selected infectious diseases informed decision makers.

HHS Coordination of Modeling Efforts

To examine coordination and collaboration across HHS agencies, we reviewed documents describing HHS agencies’ collaboration and coordination mechanisms such as Memoranda of Understanding, descriptions of Emergency Operations Center procedures, and after- action reports following infectious disease outbreaks. We also conducted interviews with and requested information from HHS officials, asking them to provide information on their efforts to coordinate their infectious disease modeling activities. In this report, and in our past work, we define coordination broadly as any joint activity that is intended to produce more public value than could be produced when organizations act alone. We compared these actions to relevant selected collaboration leading practices: define and articulate a common outcome; establish mutually reinforcing or joint strategies; identify and address needs by leveraging resources; agree on roles and responsibilities; establish compatible policies, procedures, and other means to operate across agency boundaries; and develop mechanisms to monitor, evaluate, and report on results.

Because we judgmentally selected a group of experts and diseases, the results of our review cannot be generalized to HHS coordination efforts for other infectious diseases. However, our assessment of collaboration and coordination activities did cover modeling efforts for the three selected diseases.

Developing Infectious Disease Models and Assessing Their Performance

To identify steps that are generally considered when modelers develop infectious disease models and assess their performance, we conducted semi-structured interviews with relevant experts from academia and other organizations and CDC and ASPR officials, and reviewed literature identified by experts.

We used a snowball sampling approach to identify relevant experts and groups. We initially identified five infectious disease modeling experts through informal conversation with individuals working in the field, infectious disease modeling experts known through GAO work, as well as a review of websites, publications, and grants funded by NIH. Using a snowball sampling approach, we reviewed key literature related to the steps generally taken to develop models and assess their performance, consulted with infectious disease modeling experts, and interviewed agency officials to identify relevant groups, as well as individual experts, who could convey to us the steps generally taken during infectious disease modeling.

Through literature searches, the team identified literature from public health journals or other major sources. The team applied personal background and knowledge in public health, infectious disease modeling, and statistics to help identify key sources. For the selected literature, we reviewed references and used a snowball approach to identify further relevant studies. Finally, we reviewed CDC guidance on decision making for data access and long-term preservation as it related to documentation standards.

Based on our review of identified literature, we developed a data collection instrument to assess the extent to which CDC and ASPR used the steps for infectious disease model development identified by experts and in the literature. Through this data collection instrument, we gathered information about the elements of developing and assessing model performance and the steps that could be taken within each element. In order to develop the data collection instrument, based on our review of literature, we mapped out steps to develop and assess model performance, and developed broad categories of assessment elements. Within each assessment element, we included steps modelers could take as a part of each assessment element. For example, the data collection instrument included items that recorded model verification steps that might have been taken by modeler(s) within the broader model verification element. The instrument was reviewed by internal stakeholders, who provided feedback on its content. Prior to sending the data collection instrument to the agency, we filled in information on verification steps taken for each of the 10 selected models, based on provided model documentation to reflect steps we determined modelers took as a part of the model development and assessment process. In order to provide officials with this information, two analysts reviewed each model’s documentation, with one analyst providing an initial coding of the model and the other reviewing and verifying the first analyst’s findings. This method was first tested on one of the 10 selected models by two analysts independently coding information from the model’s documentation into the data collection instrument and then reviewing coding choices to reconcile any differences found. We then sent the instruments with filled-in information to CDC and ASPR modelers to receive their feedback concerning the steps taken to develop models and assess their performance, provide any missing information, and resolve any ambiguities. See Appendix III for a list of the 10 selected models reviewed and steps to develop and assess model performance included in the data collection instrument. The data collection instrument was intended to record whether a specific step had been taken, but did not assess the quality of the modeling steps.

In order to determine steps CDC and ASPR took to develop and assess its models, we selected a non-generalizable sample of 10 models for review in our data collection instrument that demonstrated steps that HHS agencies took to develop models and assess their performance. The model selection process described above informed our selection of infectious disease models. To be selected for inclusion in our non- generalizable sample, the model had to be (1) developed by CDC, or ASPR officials or contractors; (2) developed to answer a question about Ebola, Zika, or pandemic influenza; and (3) used to inform public health decision makers during an outbreak or for preparedness activities. We selected 10 models that differed in form and answered different types of questions, which included studies prepared during both outbreak preparedness and response times, and covered topics such as the impact of vaccination programs on deaths and hospitalization. For Ebola and Zika, we focused on review of selected papers or memos produced since 2014 in order to capture the time period following the 2014-2016 Ebola and 2015-2016 Zika outbreaks. For pandemic influenza, we focused on papers and memos produced since 2009, when the H1N1 pandemic occurred in the United States. Because we selected from a group of models identified by HHS modelers and officials for Ebola, Zika, and pandemic influenza, the results of our review cannot be generalized to other diseases outside of the scope of this report. Furthermore, we requested models that informed public health decision making, and did not consider models that were not used for this purpose. Because we reviewed a non-generalizable sample of 10 models, the results of our review cannot be generalized to a larger population of models prepared by HHS agencies.

Challenges to Effective Modeling

To identify challenges associated with modeling for the selected infectious diseases, we reviewed documents and reports to identify modeling challenges and steps to address those challenges, and interviewed agency officials and modelers, and experts identified through the previously-described snowball sampling methodology. We used semi- structured interview protocols that included open-ended questions about challenges associated with infectious disease modeling and limitations associated with model development. Not all officials and experts we interviewed provided comments on every challenge or limitation. In addition, because we judgmentally selected a group of experts and diseases, the results of our review cannot be generalized to all infectious disease modeling efforts.

We conducted this performance audit from May 2018 to May 2020, in accordance with generally accepted government auditing standards. These standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Bibliography of Selected Model Publications Reviewed

Ebola Models

Carias, Cristina, et al. “Preventive Malaria Treatment for Contacts of Patients with Ebola Virus Disease in the Context of the West Africa 2014- 15 Ebola Virus Disease Response: An Economic Analysis.” The Lancet Infectious Diseases, vol. 16, no. 4 (April 2016): pp. 449-458.

Christie, Athalia, et al. “Possible Sexual Transmission of Ebola Virus— Liberia, 2015.” Morbidity and Mortality Weekly Report, vol. 64, no. 17 (May 8, 2015): pp. 479-481.

Martin I. Meltzer, et al. “Estimating the Future Number of Cases in the Ebola Epidemic - Liberia and Sierra Leone, 2014-2015.” Morbidity and Mortality Weekly Report, vol. 63, no. 3 suppl. (September 26, 2014): pp. 1-14.

Meltzer, Martin I., et al. “Modeling in Real Time during the Ebola Response.” Morbidity and Mortality Weekly Report, vol. 65, no. 3 suppl. (July 8, 2016): pp. 85-89.

Rainisch, Gabriel, et al. “Estimating Ebola Treatment Needs, United States.” Emerging Infectious Diseases, vol. 21, no. 7 (July 2015): pp. 1273-1275.

Rainisch, Gabriel, et al. “Regional Spread of Ebola Virus, West Africa, 2014.” Emerging Infectious Diseases, vol. 21, no. 3 (March 2015): pp. 444-447.

Undurraga, Eduardo A., Cristina Carias, Martin I. Meltzer, Emily B. Kahn. “Potential for Broad-Scale Transmission of Ebola Virus Disease during the West Africa Crisis: Lessons for the Global Health Security Agenda.” Infectious Diseases of Poverty, vol. 6, no. 159 (2017).

Washington, Michael L., Martin I. Meltzer. “Effectiveness of Ebola Treatment Units and Community Care Centers Liberia, September 23- October 31, 2014.” Morbidity and Mortality Weekly Report, vol. 64, no. 3 (January 30, 2015): pp. 67-69.

Zika Models

Adamski, Alys, et al. “Estimating the Numbers of Pregnant Women Infected with Zika Virus and Infants with Congenital Microcephaly in Colombia, 2015–2017.” Journal of Infection, vol. 76 (2018): pp. 529-535.

Dirlikov, Emilio, et al. “Guillain-Barré Syndrome and Healthcare Needs during Zika Virus Transmission, Puerto Rico, 2016.” Emerging Infectious Diseases, vol. 23, no. 1 (January 2017): pp.134-136.

Ellington, Sascha R., et al. “Estimating the Number of Pregnant Women Infected With Zika Virus and Expected Infants With Microcephaly Following the Zika Virus Outbreak in Puerto Rico, 2016.” JAMA Pediatrics, vol. 170, no. 10 (2016): pp. 940-945.

Grills, Ardath, et al. “Projected Zika Virus Importation and Subsequent Ongoing Transmission after Travel to the 2016 Olympic and Paralympic Games—Country-Specific Assessment, July 2016.” Morbidity and Mortality Weekly Report, vol. 65, no. 28 (July 22, 2016): pp.711-715.

Johansson, Michael A., et al. “Zika and the Risk of Microcephaly.” The New England Journal of Medicine, vol. 375 (July 7, 2016): pp.1-4.

Johnson, Tammi L., et al. “Modeling the Environmental Suitability for Aedes (Stegomyia) aegypti and Aedes (Stegomyia) albopictus (Diptera: Culicidae) in the Contiguous United States.” Journal of Medical Entomology, vol. 54, no. 6 (November 7, 2017): pp. 1605-1614.

Mitchell, Patrick K. et al., “Reassessing Serosurvey-Based Estimates of the Symptomatic Proportion of Zika Virus Infections.” American Journal of Epidemiology, vol. 188, no. 1 (January 2019): pp. 206-213.

Mier-y-Teran-Romero, Luis, Mark J. Delorey, James J. Sejvar, Michael A. Johansson. “Guillain-Barré Syndrome Risk Among Individuals Infected with Zika Virus: a Multi-Country Assessment.” BMC Medicine, vol. 16, no. 67 (2018).

Mier-y-Teran-Romero, Luis, Andrew J. Tatem, Michael A. Johansson. “Mosquitoes on a Plane: Disinsection Will Not Stop the Spread of Vector- Borne Pathogens, a Simulation Study.” PLoS Neglected Tropical Diseases, vol. 11, no. 7 (July 3, 2017).

Reefhuis, Jennita, et al. “Projecting Month of Birth for At-Risk Infants after Zika Virus Disease Outbreaks.” Emerging Infectious Diseases, vol. 22, no. 5 (May 2016): pp. 828-832.

Russell, Steven, et al. “Detecting Local Zika Virus Transmission in the Continental United States: A Comparison of Surveillance Strategies.” PLoS Currents Outbreaks (November 22, 2017).

Watts. Alexander G., et al. “Elevation as a Proxy for Mosquito-Borne Zika Virus Transmission in the Americas.” PLoS ONE, vol. 12, no. 5 (May 24, 2017).

Influenza Models

Atkins, Charisma Y., et al. “Estimating Effect of Antiviral Drug Use during Pandemic (H1N1) 2009 Outbreak, United States.” Emerging Infectious Diseases, vol. 17. no. 9 (September 2011): pp. 1591-1598.

Biggerstaff, Matthew, et al. “Estimates of the Number of Human Infections With Influenza A(H3N2) Variant Virus, United States, August 2011–April 2012.” Clinical Infectious Diseases, vol. 57, suppl. 1 (2013): pp. S12-S15.

Biggerstaff, Matthew, et al. “Estimating the Potential Effects of a Vaccine Program Against an Emerging Influenza Pandemic—United States.” Clinical Infectious Diseases, vol. 60, suppl. 1 (2015): pp. S20-S29.

Carias, Cristina, et al. “Potential Demand for Respirators and Surgical Masks during a Hypothetical Influenza Pandemic in the United States.” Clinical Infectious Diseases, vol. 60, suppl. 1 (2015): pp. S42-S51.

Cauchemez, Simon, et al. “Role of Social Networks in Shaping Disease Transmission during a Community Outbreak of 2009 H1N1 Pandemic Influenza.” Proceedings of the National Academy of Sciences of the United States, vol. 108, no. 7 (February 15, 2011): pp. 2825-2830.

Dawood, Fatimah S., et al. “Estimated Global Mortality Associated with the First 12 Months of 2009 Pandemic Influenza A H1N1 Virus Circulation: a Modelling Study.” The Lancet Infectious Diseases, vol. 12 (September 2012): pp. 687-695.

Fung, Isaac Chun-Hai, et al. “Modeling the Effect of School Closures in a Pandemic Scenario: Exploring Two Different Contact Matrices.” Clinical Infectious Diseases, vol. 60, suppl. 1 (2015): pp. S58-S63.

Iuliano, A. Danielle, et al. “Estimates of Global Seasonal Influenza- Associated Respiratory Mortality: a Modelling Study.” The Lancet, vol. 391, no. 10127 (March 31, 2018): pp. 1285-1300.

Jain, Seema, et al. “Hospitalized Patients with 2009 H1N1 Influenza in the United States, April–June 2009.” The New England Journal of Medicine, vol. 361, no. 20 (November 12, 2009): pp. 1935-1944.

Kostova, Deliana, et al. “Influenza Illness and Hospitalizations Averted by Influenza Vaccination in the United States, 2005–2011.” PLoS ONE, vol. 8, no. 6 (June 19, 2013).

Lafond, Kathryn E., et al. “Global Role and Burden of Influenza in Pediatric Respiratory Hospitalizations, 1982–2012: A Systematic Analysis.” PLoS Medicine, vol. 13, no. 3 (March 24, 2016).

Meltzer, Martin I., Nancy J. Cox, Keiji Fukuda. “The Economic Impact of Pandemic Influenza in the United States: Priorities for Intervention.” Emerging Infectious Diseases, vol. 5, no. 5 (September-October 1999): pp. 659-671.

Meltzer, Martin I., et al. “Estimates of the Demand for Mechanical Ventilation in the United States during an Influenza Pandemic.” Clinical Infectious Diseases, vol. 60, suppl. 1 (2015): pp. S52-S57.

O’Hagan, Justin J., et al. “Estimating the United States Demand for Influenza Antivirals and the Effect on Severe Influenza Disease during a Potential Pandemic.” Clinical Infectious Diseases, vol. 60, suppl. 1 (2015): pp. S30-S41.

Presanis, Anne M., et al. “The Severity of Pandemic H1N1 Influenza in the United States, from April to July 2009: A Bayesian Analysis.” PLoS Medicine, vol. 6, no. 12 (December 8, 2009).

Reed, Carrie, et al. “Estimates of the Prevalence of Pandemic (H1N1) 2009, United States, April-July 2009.” Emerging Infectious Diseases, vol. 15, no. 12 (December 2009): pp. 2004-2007.

Reed, Carrie, Martin I. Meltzer, Lyn Finelli, Anthony Fiore. “Public Health Impact of Including Two Lineages of Influenza B in a Quadrivalent Seasonal Influenza Vaccine.” Vaccine, vol. 30 (2012): pp. 1993-1998.

Reed, Carrie, et al. “Estimating Influenza Disease Burden from Population-Based Surveillance Data in the United States.” PLoS ONE, vol. 10, no. 3 (March 4, 2015).

Rolfes, Melissa A., et al. “Annual Estimates of the Burden of Seasonal Influenza in the United States: A Tool for Strengthening Influenza Surveillance and Preparedness.” Influenza and Other Respiratory Viruses, vol. 12 (2018): pp. 132-137.

Russell, K., et al. “Utility of State-Level Influenza Disease Burden and Severity Estimates to Investigate an Apparent Increase in Reported Severe Cases of Influenza A(H1N1) pdm09 – Arizona, 2015–2016.” Epidemiology and Infection, vol. 146 (June 14, 2018): pp. 1359-1365.

Shrestha, Sundar S., et al. “Estimating the Burden of 2009 Pandemic Influenza A (H1N1) in the United States (April 2009–April 2010).” Clinical Infectious Diseases, vol. 52, suppl. 1 (2011): pp. S75-S82.

Tokars, Jerome I., Melissa A. Rolfes, Ivo M. Foppa, Carrie Reed. “An Evaluation and Update of Methods for Estimating the Number of Influenza Cases Averted by Vaccination in the United States.” Vaccine, vol. 36 (2018): pp. 7331-7337.

Appendix III: Ten Selected Infectious Disease Models and Questions from Data Collection Instrument

Appendix III: Ten Selected Infectious Disease Models and Questions from Data Collection Instrument Document describing model Meltzer, Martin I., Charisma Y. Atkins, Scott Santibanez, Barbara Knust, Brett W. Petersen, Elizabeth D. Ervin, Stuart T. Nichol, Inger K. Damon, Michael L. Washington. Estimating the Future Number of Cases in the Ebola Epidemic–Liberia and Sierra Leone, 2014-2015, MMWR. Volume 63, Number 3, September 26, 2014. Rainisch, Gabriel, Manjunath Shankar, Michael Wellman, Toby Merlin, and Martin I. Meltzer. Regional Spread of Ebola Virus, West Africa, 2014. Emerging Infectious Diseases. Volume 21, Number 3, March 2015. Asher, Jason. Forecasting Ebola with a Regression Transmission Model. Epidemics. Volume 22, 2018.

Ellington, Sascha R., Owen Devine, Jeanne Bertolli, Alma Martinez Quiñones, Carrie K. Shapiro-Mendoza, Janice Perez-Padilla, Brenda Rivera-Garcia, Regina M. Simeone, Denise J. Jamieson, Miguel Valencia-Prado, Suzanne M. Gilboa, Margaret A. Honein, Michael A. Johansson. Estimating the Number of Pregnant Women Infected With Zika Virus and Expected Infants With Microcephaly Following the Zika Virus Outbreak in Puerto Rico, 2016. JAMA Pediatrics. Volume 170, Number 10, October 2016.

Johansson, Michael A., Luis Mier-y‐Teran-Romero, Jennita Reefhuis, Suzanne M. Gilboa, and Susan L. Hills. Zika and the Risk of Microcephaly. New England Journal of Medicine. Volume 375, Number 1, July 7, 2016. Dirlikov, Emilio, Krista Kniss, Chelsea Major, Dana Thomas, Cesar A. Virgen, Marrielle Mayshack, Jason Asher, Luis Mier-y-Teran-Romero, Jorge L. Salinas, Daniel M. Pastula, Tyler M. Sharp, James Sejvar, Michael A. Johansson, Brenda Rivera-Garcia. Guillain-Barré Syndrome and Healthcare Needs during Zika Virus Transmission, Puerto Rico, 2016. Emerging Infectious Diseases. Volume 23, Number 1, January 2017. Biggerstaff, Matthew, Carrie Reed, David L. Swerdlow, Manoj Gambhir, Samuel Graitcer, Lyn Finelli, Rebekah H. Borse, Sonja A. Rasmussen, Martin I. Meltzer, Carolyn B. Bridges. Estimating the Potential Effects of a Vaccine Program against an Emerging Influenza Pandemic—United States, Clinical Infectious Diseases. Volume 60, Issue Supplement 1, 2015. Carias, Cristina, Gabriel Rainisch, Manjunath Shankar, Bishwa B. Adhikari, David L. Swerdlow, William A. Bower, Satish K. Pillai, Martin I. Meltzer, Lisa M. Koonin. Potential Demand for Respirators and Surgical Masks during a Hypothetical Influenza Pandemic in the United States. Clinical Infectious Disease. Volume 60, Issue Supplement 1, 2015. Reed, Carrie, Frederick J. Angulo, David L. Swerdlow, Marc Lipsitch, Martin I. Meltzer, Daniel Jernigan, and Lyn Finelli. Estimates of the Prevalence of Pandemic (H1N1) 2009, United States, April–July 2009, Emerging Infectious Diseases. Volume 15, Number 12, December 2009. Asher, Jason, Matthew Clay. Deterministic compartmental models for influenza with mitigations. R: “flumodels” package. Version: 1.0.7, April 24, 2017.

Data Collection Instrument

GAO Review of Model Assessment Steps for Selected Agency Models

Purpose: The Government Accountability Office has been asked by the Congress to review the Department of Health and Human Services’ agency efforts to model infectious disease. As part of our methodology, we selected and reviewed published papers and internal memoranda from the sources provided to us. We reviewed these sources to describe the steps taken to describe, verify, validate, and communicate results of these modeling efforts. The purpose of this inquiry is to provide the authors of the selected papers the opportunity to confirm, clarify, or provide additional information in the table below.

Instructions: In the table below, we have two sets of columns: one set indicating GAO’s assessment of whether the document contained information about a step being taken. The second set of columns is for the authors of the selected paper to fill out. If you agree with information in the GAO columns, please indicate your concurrence in the Reviewer Comments column. Otherwise, please provide information accordingly.

If a step is marked “Step taken” please review the entries we have made in the GAO Reviewer Comments column for accuracy and completeness and indicate your concurrence in the Reviewer Comments column. Please also provide additional supporting documentation if available. For any steps that were taken, but where we indicated either “not taken” or “not enough information to determine” in our review, please provide a description of the actual steps and any documentation you may have. If a step was not taken, please provide an indication as to why that step was not taken and, if possible, please provide supporting documentation. For example, if limited data availability impacted the ability to conduct a model validation step(s), then please include this information in the appropriate table cells.

Assessment Element

Clarify Objectives

Model Description

Model Verification (Internal Validation, Internal Consistency, Technical Validity)

10 Independent expert (internal or external) review of key programming 11 Debugging tests and checks for coding accuracy 12 Model’s code or Excel spreadsheet is available 13 Test model assumptions (i.e. confirming model assumptions are reasonable and appropriate for question), for example:

Distributional assumptions about model residuals

Form of the model 14 Model handling of input data/parameters is verified as correct (i.e. as intended by developers)

Model Validation

16 Sensitivity analysis (assessing impact of assumption/parameter uncertainty on output or model form) 17 Cross validation or between model comparisons: Compare results to other models that address the same problem 18 External validation: Compare model results to actual event data 19 Predictive validation: Compare model predictions for future events to actual outcomes.

Communication

21 Modelers supply customer with agreed upon information, which may vary depending on the model 22 Modeler provides customer with clear information on uncertainty in model results, such as inclusion of standard errors or confidence intervals, or qualitative explanations of uncertainty in the model results Assessment Steps Question: Do you think that the assessment elements identified in the table above sufficiently reflect the steps that should generally be taken to develop and assess the performance of models? Would you remove any steps, add any steps, or make any other adjustments to these steps in order to consider them best practices in assessing performance of models, generally? Please explain.

Appendix IV: Comments from the Department of Health and Human Services

Appendix V: GAO Contacts and Staff Acknowledgments

GAO Contacts

Staff Acknowledgments

In addition to the contact named above, the following individuals made contributions to this report: Sushil Sharma (Assistant Director), Charlotte E. Hinkle (Analyst-in-Charge), Sam Amrhein, Breanne Cave, Jehan Chase, Carol A. Gotway Crawford, Justin Cubilo, Karen Doran, Nancy Fasciano, Douglas G. Hunker, Dennis Mayo, Anika McMillon, Sarah Resavy, Edward Rice, Ben Shouse, Amber Sinclair, Walter Vance, Sarah Veale, and Richard Zarrella.