Influenza Pandemic: Opportunities Exist to Clarify Federal Leadership Roles and Improve Pandemic Planning
Published: 2007-09-26T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: An influenza pandemic is a real and significant potential threat facing the United States and the world. Pandemics are unlike other emergencies because they are not a singular event nor discretely bounded in space and time. This testimony addresses (1) federal leadership roles and responsibilities for preparing for and responding to a pandemic, (2) our assessment of the Strategy and Plan, and (3) opportunities to increase clarity of federal leadership roles and responsibilities and improve pandemic planning. GAO used its characteristics of an effective national strategy to assess the Strategy and Plan. The issues discussed in the testimony are based primarily on the GAO report, Influenza Pandemic: Further Efforts Are Needed to Ensure Clearer Federal Leadership Roles and an Effective National Strategy (GAO-07-781). In this report, GAO recommended that (1) The Secretaries of Homeland Security and Health and Human Services develop rigorous testing, training, and exercises for pandemic influenza to ensure that federal leadership roles and responsibilities are clearly defined, understood and work effectively and (2) HSC set a time frame to update the Plan, involve key stakeholders, and more fully address the characteristics of an effective national strategy. The Departments of Homeland Security and Health and Human Services concurred. The HSC did not comment.
section_title: What GAO Found
paragraphs: The administration has taken an active approach to this potential disaster by, among other things, issuing a National Strategy for Pandemic Influenza (Strategy) in November 2005, and a National Strategy for Pandemic Influenza Implementation Plan (Plan) in May 2006. However, much more needs to be done to ensure that the Strategy and Plan are viable and can be effectively implemented in the event of an influenza pandemic. Key federal leadership roles and responsibilities for preparing for and responding to a pandemic continue to evolve and will require further clarification and testing before the relationships of the many leadership positions are well understood. Most of these leadership roles involve shared responsibilities and it is unclear how they will work in practice. Because initial actions may help limit the spread of an influenza virus, the effective exercise of shared leadership roles and responsibilities could have substantial consequences. However, only one national, multi-sector pandemic-related exercise has been held, and that was prior to issuance of the Plan. The Strategy and Plan do not fully address the characteristics of an effective national strategy and contain gaps that could hinder the ability of key stakeholders to effectively execute their responsibilities. Specifically, some of the gaps include (1) The Strategy and Plan do not address resources, investments, and risk management and consequently do not provide a picture of priorities or how adjustments might be made in view of limited resources. (2) State and local jurisdictions were not directly involved in developing the Plan, even though they would be on the front lines in a pandemic. (3) Relationships and priorities among action items are not always clear. (4) Performance measures are focused on activities that are not always linked to results. (5) The linkage of the Strategy and Plan with other key plans is unclear. (6) The Plan does not contain a process for monitoring and reporting on progress. (7) The Plan does not describe an overall framework for accountability and oversight and does not clarify how responsible officials would share leadership responsibilities. (8) Procedures and time frames for updating and revising the Plan were not established. These gaps can affect the usefulness of these planning documents for those with key roles to play. Also, the lack of mechanisms for future updates or progress assessments limit opportunities for congressional decision makers and the public to assess the extent of progress being made or to consider what areas or actions may be need additional attention. Although the Homeland Security Council (HSC) publicly reported on the status of action items in December 2006 and July 2007, it is unclear when the next report will be issued or how much information will be released.

section_title: Letter
paragraphs: I am pleased to appear here today to discuss the federal government’s  efforts to prepare for and respond to a possible influenza pandemic. An  influenza pandemic is a real and significant threat facing the United States  and the world. Although the timing and severity of the next pandemic is  unpredictable, there is widespread agreement that a pandemic will occur  at some point. Unlike incidents that are discretely bounded in space or  time (such as a storm or a terrorist attack), a pandemic is not a singular  event, but is likely to come in waves, each lasting weeks or months, and  could pass through communities of all sizes across the nation and the  world simultaneously.
Today, I will discuss (1) federal leadership roles and responsibilities for  preparing for and responding to a pandemic, (2) our assessment of the  National Strategy for a Pandemic Influenza (Strategy) and the  Implementation Plan for the National Strategy for a Pandemic Influenza  (Plan), and (3) opportunities to increase the clarity of federal leadership  roles and responsibilities and improve pandemic planning.
This statement is based on our August 14, 2007, report, requested by the  Ranking Member, Senate Budget Committee; the Chairman and Ranking  Member, House Committee on Oversight and Government Reform; and the  Chairman, House Committee on Homeland Security. Our objectives in  that report were to address the extent to which (1) federal leadership roles  and responsibilities for preparing for and responding to a pandemic are  clearly defined and documented and (2) the Strategy and the Plan address  the characteristics of an effective national strategy; we conducted our  work in accordance with generally accepted government auditing  standards. We analyzed relevant documents, interviewed cognizant federal  officials, and assessed the Strategy and Plan to determine the extent to  which they jointly addressed the six desirable characteristics of an  effective national strategy that we developed and used in previous work.
While national strategies necessarily vary in content, the six  characteristics we identified apply to all such planning documents and can  help ensure that they are effective management tools.
In summary, although the administration has taken an active approach to  this potential disaster by developing a Strategy and Plan, and has  undertaken a number of other efforts, much more needs to be done to  ensure that the Plan is more viable and can be effectively implemented in  the event of an influenza pandemic.
Key federal leadership roles and responsibilities for preparing for and  responding to a pandemic continue to evolve and will require further  clarification and testing before the relationships of the many leadership  positions are well understood. Most of these leadership roles involve  shared responsibilities, and it is not clear how these would work in  practice. Because initial actions may help limit the spread of an influenza  virus, the effective exercise of shared leadership roles and responsibilities  could have substantial consequences. However, only one national,  multisector pandemic-related exercise has been held, and that was prior to  issuance of the Plan.
The Strategy and Plan do not fully address the characteristics of an  effective national strategy and contain gaps that could hinder the ability of  key stakeholders to effectively execute their responsibilities. In addition to  the fact that the Strategy and Plan do not clarify how responsible officials  will share leadership responsibilities, they do not include a description of  the resources required to implement the Plan, and consequently do not  provide a picture of priorities or how adjustments might be made in view  of resource constraints. Additionally, state and local jurisdictions that will  play crucial roles in preparing for and responding to a pandemic were not  directly involved in developing the Plan, and the linkage of the Strategy  and Plan with other key plans is unclear.
The gaps in the Strategy and Plan are particularly troubling because they  can affect the usefulness of these planning documents for those with key  roles to play and, with no mechanisms for future updates or progress  assessments, limit opportunities for congressional decision makers and  the public to assess the extent of progress being made or to consider what  areas or actions may need additional attention.
We made two recommendations in our August 2007 report to address  these concerns.
We recommended that the Secretaries of Homeland Security and Health  and Human Services work together to develop and conduct rigorous  testing, training, and exercises for pandemic influenza to ensure that the  federal leadership roles are clearly defined and understood and that  leaders are able to effectively execute shared responsibilities to address  emerging challenges. Once the leadership roles have been clarified  through testing, training and exercising, the Secretaries of Homeland  Security and Health and Human Services should ensure that these roles  and responsibilities are clearly understood by nonfederal partners.
Our report also recommended that the Homeland Security Council (HSC)  establish a specific process and time frame for updating the Plan. This  process should involve key nonfederal stakeholders and incorporate  lessons learned from exercises and other sources. The next update of the  Plan could be improved by addressing the gaps we identified.  The Department of Health and Human Services (HHS) and the Department  of Homeland Security (DHS) concurred with the first recommendation.  The HSC did not comment on the draft report or our recommendation.
subsections: section_title: Background
paragraphs: To address the potential threat of an influenza pandemic, the President  and his HSC issued two planning documents. The Strategy was issued in  November 2005 and is intended to provide a high-level overview of the  approach that the federal government will take to prepare for and respond  to an influenza pandemic. It also articulates expectations for nonfederal  entities—including state, local, and tribal governments; the private sector;  international partners; and individuals—to prepare themselves and their  communities.
The Plan was issued in May 2006 and is intended to lay out broad  implementation requirements and responsibilities among the appropriate  federal agencies and clearly define expectations for nonfederal entities.  The Plan includes 324 action items related to these requirements,  responsibilities, and expectations and most of them are to be completed  before or by May 2009. It is intended to support the broad framework and  goals articulated in the Strategy by outlining specific steps that federal  departments and agencies should take to achieve these goals. It also  describes expectations regarding preparedness and response efforts of  state and local governments, tribal entities, the private sector, global  partners, and individuals. The Plan’s chapters cover categories of actions  that are intended to address major considerations raised by a pandemic,  including protecting human and animal health; transportation and borders;  and international, security, and institutional considerations.
section_title: Federal Leadership Roles Are Unclear, Evolving, and Untested
paragraphs: Several federal leadership roles involve shared responsibilities for  preparing for and responding to an influenza pandemic, including the  Secretaries of Health and Human Services and Homeland Security, the  Administrator of the Federal Emergency Management Agency (FEMA), a  national Principal Federal Official (PFO), and regional PFOs and Federal  Coordinating Officers (FCO). Many of these leadership roles and  responsibilities have not been tested under pandemic scenarios, leaving  unclear how all of these new and developing relationships would work.
subsections: section_title: Federal Leadership Roles and Responsibilities Are Unclear and Evolving
paragraphs: The Strategy and Plan do not clarify the specific leadership roles and  responsibilities for a pandemic. Instead, they restate the existing  leadership roles and responsibilities, particularly for the Secretaries of  Homeland Security and Health and Human Services, prescribed in the  National Response Plan (NRP)—an all-hazards plan for emergencies  ranging from hurricanes to wildfires to terrorist attacks. However, the  leadership roles and responsibilities prescribed under the NRP need to  operate somewhat differently because of the characteristics of a pandemic  that distinguish it from other emergency incidents. For example, because a  pandemic influenza is likely to occur in successive waves, planning has to  consider how to sustain response mechanisms for several months to over  a year—issues that are not clearly addressed in the Plan.
In addition, the distributed nature of a pandemic, as well as the sheer  burden of disease across the nation, means that the support states,  localities, and tribal entities can expect from the federal government  would be limited in comparison to the aid it mobilizes for geographically  and temporarily bounded disasters like earthquakes and hurricanes.  Consequently, legal authorities, roles and responsibilities, and lines of  authority at all levels of government must be clearly defined, effectively  communicated, and well understood to facilitate rapid and effective  decision making. This is also important for public and private sector  organizations and international partners so everyone can better  understand what is expected of them before and during a pandemic.
The Strategy and Plan state that the Secretary of Health and Human  Services is responsible for leading the medical response in a pandemic,  while the Secretary of Homeland Security is responsible for overall  domestic incident management and federal coordination. However, since a  pandemic extends well beyond health and medical boundaries, to include  sustaining critical infrastructure, private sector activities, the movement of  goods and services across the nation and the globe, and economic and  security considerations, it is not clear when, in a pandemic, the Secretary  of Health and Human Services would be in the lead and when the  Secretary of Homeland Security would lead.
A pandemic could threaten our critical infrastructure, such as the  capability to deliver electricity or food, by removing essential personnel  from the workplace for weeks or months. The extent to which this would  be considered a medical response with the Secretary of Health and Human  Services in the lead, or when it would be under the Secretary of Homeland  Security’s leadership as part of his/her responsibility for ensuring that  critical infrastructure is protected, is unclear. According to HHS officials  we interviewed, resolving this ambiguity will depend on several factors,  including how the outbreak occurs and the severity of the pandemic.  Although DHS and HHS officials emphasize that they are working together  on a frequent basis, these roles and responsibilities have not been  thoroughly tested and exercised.
Moreover, under the Post-Katrina Emergency Management Reform Act of  2006 (referred to as the Post-Katrina Reform Act in this testimony), the  FEMA Administrator was designated the principal domestic emergency  management advisor to the President, the HSC, and the Secretary of  Homeland Security, adding further complexity to the leadership structure  in the case of a pandemic. The act also gives the Administrator  responsibility for carrying out a national exercise program to test and  evaluate national preparedness for responding to all-hazards, including an  influenza pandemic.
Other evolving federal leadership roles include those of PFOs and FCOs.  To assist in planning and coordinating efforts to respond to a pandemic, in  December 2006 the Secretary of Homeland Security predesignated a  national PFO and established five pandemic regions each with a regional  PFO and FCO. PFOs are responsible for facilitating federal domestic  incident planning and coordination, and FCOs are responsible for  coordinating federal resources support in a presidentially declared major  disaster or emergency. However, the relationship of these roles to each  other as well as with other leadership roles in a pandemic is unclear.
U.S. Coast Guard and FEMA officials we met with recognized that  planning for and responding to a pandemic would require different  operational leadership roles and responsibilities than for most other  emergencies. For example, a FEMA official said that given the number of  people who would be involved in responding to a pandemic, collaboration  between HHS, DHS, and FEMA would need to be greater than for any  other past emergencies. Officials are starting to build on these  relationships. For example, some of the federal officials with leadership  roles for an influenza pandemic met during the week of March 19, 2007, to  continue to identify issues and begin developing solutions. One of the  participants told us that although additional coordination meetings are  needed, it may be challenging since there is no dedicated funding for the  staff working on pandemic issues to participate in these and other related  meetings.
It is also unclear whether the newly established national and regional  positions for a pandemic will further clarify leadership roles in light of  existing and newly emerging plans and issues. For example, in 2006, DHS  made revisions to the NRP and released a Supplement to the Catastrophic  Incident Annex—both designed to further clarify federal roles and  responsibilities and relationships among federal, state, and local  governments and responders. However, we reported in February 2007 that  these revisions had not been tested and there was little information  available on the extent to which these and other actions DHS was taking  to improve readiness were operational. We also reported in May 2007 that  FEMA has predesignated five teams of FCOs and PFOs in the Gulf Coast  and eastern seaboard states at risk of hurricanes. However, there is still  some question among state and local first responders about the need for  both positions and how they will work together in disaster response.
More recently, DHS reviewed the NRP and its supplemental documents.  One of the issues this review intended to address was clarifying roles and  responsibilities of key structures, positions, and levels of government,  including the role of the PFO and that position’s current lack of  operational authority over the FCO during an emergency. On September  10, 2007, DHS released a draft National Response Framework to replace  the NRP, for public comment. Comments on the framework are due  October 11, 2007, and comments on the supplemental documents, such as  revised Emergency Support Function specifications, are due by November  9, 2007.
section_title: Exercising and Testing of Plans Is Crucial in Ensuring Capacity
paragraphs: Disaster planning, including for a pandemic influenza, needs to be tested  and refined with a rigorous and robust exercise program to expose  weaknesses in plans and allow planners to refine them. Exercises— particularly for the type and magnitude of emergency incidents such as a  severe influenza pandemic for which there is little actual experience—are  essential for developing skills and identifying what works well and what  needs further improvement. Our prior work examining the preparation for  and response to Hurricane Katrina highlighted the importance of realistic  exercises to test and refine assumptions, capabilities, and operational  procedures, and build upon strengths.
While pandemic influenza scenarios have been used to exercise specific  response elements, such as the distribution of stockpiled medications at  specific locations or jurisdictions, no national exercises have tested the  new federal leadership structure for pandemic influenza. The only  national multisector pandemic exercise to date was a tabletop simulation  conducted by members of the cabinet in December 2005. This tabletop  exercise was prior to the release of the Plan in May 2006, the  establishment of a national PFO and regional PFO and FCO positions for a  pandemic, and enactment of the Post-Katrina Reform Act.
section_title: Gaps in the National Strategy and Plan Limit Their Usefulness
paragraphs: Our work found that the Strategy and Plan do not address all of the  characteristics of an effective national strategy as identified in our prior  work. While national strategies necessarily vary in content, the six  characteristics we identified apply to all such planning documents and can  help ensure that they are effective management tools. Gaps and  deficiencies in these documents are particularly troubling in that a  pandemic represents a complex challenge that will require the full  understanding and collaboration of a multitude of entities and individuals.  The extent to which these documents, that are to provide an overall  framework to ensure preparedness and response to a pandemic influenza,  fail to adequately address key areas could have critical impact on whether  the public and key stakeholders have a clear understanding and can  effectively execute their roles and responsibilities.
Specifically, we found that the documents fully address only one of the six  characteristics of an effective national strategy—problem definition and  risk assessment—because they identified the potential problems  associated with a pandemic as well as potential threats, challenges, and  vulnerabilities. The Strategy and Plan did not address one characteristic— resources, investments, and risk management—because they did not  discuss the financial resources and investments needed to implement the  actions called for and therefore, do not provide a picture of priorities or  how adjustments might be made in view of resource constraints. They  partially addressed the four remaining characteristics, as shown in table 1.
More specifically, the following are highlights of some of the gaps in the  Strategy and Plan.
The Strategy and Plan do not address resources, investments, and  risk management. Developing and sustaining the capabilities  stipulated in the Plan would require the effective use of federal, state,  and local funds. Given that funding needs may not be readily  addressed through existing mechanisms and could stress existing  government and private resources, it is critical for the Plan to lay out  funding requirements. For example, one of the primary objectives of  domestic vaccine production capacity is for manufacturers to produce  enough vaccine for the entire U.S. population within 6 months.  However, the Plan states that production capacity would depend on  the availability of future appropriations. Despite the fact that the  production of enough vaccine for the population would be critical if a  pandemic were to occur, the Plan does not provide even a rough  estimate of how much the vaccine could cost for consideration in  future appropriations.
State and local jurisdictions were not directly involved in developing  the Strategy and Plan. Neither the Strategy nor Plan described the  involvement of key stakeholders, such as state, local, and tribal  entities, in their development, even though these stakeholders would  be on the front lines in a pandemic and the Plan identifies actions they  should complete. Officials told us that state, local, and tribal entities  were not directly involved in reviewing and commenting on the Plan,  but the drafters of the Plan were generally aware of their concerns.
Relationships and priorities among action items are not always  clear. While some action items depend on other action items, these  linkages are not always apparent in the Plan. An HHS official who  helped draft the Plan acknowledged that while an effort was made to  ensure linkages among action items, there may be gaps in the linkages  among interdependent action items within and across the Plan’s  chapters that focused on such issues as human health, animal health,  and transportation and borders considerations.
In addition, we found that the Plan does not establish priorities among  its 324 action items, which becomes especially important as agencies  and other parties strive to effectively manage scarce resources and  ensure that the most important steps are accomplished.
Performance measures are focused on activities that are not always  linked to results. Most of the Plan’s performance measures are focused  on activities such as disseminating guidance, but the measures are not  always clearly linked with intended results. This lack of linkages to  outcomes and results makes it difficult to ascertain whether progress  has in fact been made toward achieving the national goals and  objectives described in the Strategy and Plan.
The linkage of the Strategy and Plan with other key plans is unclear.
Although the Strategy states that it is consistent with the National  Security Strategy and the National Strategy for Homeland Security, it  does not state how it is consistent or describe the relationships with  these two strategies. In addition, the Plan does not specifically address  how the Strategy, Plan, or other related pandemic plans should be  integrated with the goals, objectives, and activities of the national  initiatives already in place, such as the interim National Preparedness  Goal.
Further, the Strategy and Plan do not provide sufficient detail about  how the Strategy, action items in the Plan, and a proposed set of  agency plans are to be integrated with other national strategies and  frameworks. For example, the Plan contains 39 action items that are  response related (i.e., specific actions are to be taken within a  prescribed number of hours or days after an outbreak). However,  these action items are interspersed among the 324 action items, and  the Plan does not describe the linkages of these response-related  action items with the NRP or other response related plans.
The Plan does not contain a process for monitoring and reporting on  progress. While most of the action items have deadlines for  completion, ranging from 3 months to 3 years, the Plan does not  identify a process to monitor and report on the progress of the action  items nor does it include a schedule for reporting progress. According  to agency officials, the HSC is monitoring executive branch agencies’  efforts to complete the action items. However, there is no specific  documentation describing this process or institutionalizing it. This is  important since some of the action items are not expected to be  completed during this administration. Also, a similar monitoring  process for those actions items for which nonfederal entities have the  lead responsibility does not appear to exist. Additionally, there is no  explicit timeline for the HSC to report on the overall progress and thus,  when progress is reported is left to the HSC’s discretion.
The Plan does not describe an overall framework for accountability  and oversight. While the plan contains broad information on roles and  responsibilities and describes coordination mechanisms for  responding to a pandemic, it does not, as noted earlier, clarify how  responsible officials would share leadership responsibilities. In  addition, it does not describe an overall accountability and oversight  framework. Agency officials told us that they had identified individuals  to act as overall coordinators to monitor the action items for which  their agencies have lead responsibility and provide periodic progress  reports to the HSC. However, we could not identify a similar oversight  mechanism for the action items that fall to state and local governments  or the private sector. This is a concern since some action items,  particularly those that are to be completed by state, local, and tribal  governments or the private sector, do not identify an entity  accountable for carrying out the action.
Procedures and time frames for updating and revising the Plan were  not established. The Plan does not describe a mechanism for updating  it to reflect policy decisions, such as clarifications in leadership roles  and responsibilities and other lessons learned from exercises, or to  incorporate other needed changes. Although the Plan was developed  as initial guidance and was intended to be updated and expanded over  time, officials in several agencies told us that specific processes or  time frames for updating and revising it have not been established.
section_title: Opportunities Exist To Clarify Federal Leadership Roles and Improve Pandemic Planning
paragraphs: A pandemic poses some unique challenges and would be unlike other  emergencies given the likelihood of its duration and geographic coverage.  Initial actions may help limit the spread of an influenza virus, reflecting the  importance of a swift and effective response. Therefore, the effective  exercise of shared leadership roles and implementation of pandemic plans  could have substantial consequences, both in the short and long term.
Since no national pandemic exercises of federal leadership roles and  responsibilities have been conducted since the release of the Plan in May  2006, and key leadership roles continue to evolve, rigorous testing,  training, and exercising is needed. Exercises test whether leadership roles  and responsibilities, as well as procedures and processes, are clear and  well-understood by key stakeholders. Additionally, they help identify  weaknesses and allow for corrective action to be taken before an actual  emergency occurs. Consequently, in our August 2007 report, we  recommended that the Secretaries of Homeland Security and Health and  Human Services work together to develop and conduct rigorous testing,  training, and exercises for pandemic influenza to ensure that the federal  leadership roles are clearly defined and understood and that leaders are  able to effectively execute shared responsibilities to address emerging  challenges. Once the leadership roles have been clarified through testing,  training, and exercising, the Secretaries of Homeland Security and Health  and Human Services should ensure that these roles and responsibilities  are clearly understood by state, local, and tribal governments; the private  and nonprofit sectors; and the international community. DHS and HHS  concurred with the recommendation, and DHS stated that it is taking  action on many of the shortfalls identified in the report.
The Strategy and Plan are important because they broadly describe the  federal government’s approach and planned actions to prepare for and  respond to a pandemic and also set expectations for states and  communities, the private sector, and global partners. The extent to which  the Strategy and Plan fail to adequately address key areas could have a  critical impact on whether key stakeholders and the public have a clear  understanding of their roles and responsibilities. However, gaps in the  Strategy and Plan limit their usefulness as a management tool for ensuring  accountability and achieving results. The plan is silent on when  information will be reported or when it will be updated. Although the HSC  publicly reported on the status of action items in December 2006 and July  2007, it is unclear when the next report will be issued or how much  information will be released. This lack of transparency makes it difficult to  inform a national dialogue on the progress made to date or what further  steps are needed. It also inhibits congressional oversight of strategies,  funding priorities, and critical efforts to enhance the nation’s level of  preparedness.
Therefore, in our August 2007 report we recommended that the HSC  establish a specific process and time frame for updating the Plan. We  stated that this process should involve key nonfederal stakeholders and  incorporate lessons learned from exercises and other sources. Further, we  stated that the Plan could be improved by including the following  information in the next update: (1) resources and investments needed to  complete the action items and where they should be targeted, (2) a  process and schedule for monitoring and publicly reporting on progress  made on completing the action items, (3) clearer linkages with other  strategies and plans, and (4) clearer descriptions of relationships or  priorities among actions items and greater use of outcome-focused  performance measures. The HSC did not comment on the draft report.
Mr. Chairman and Members of the Subcommittee, this completes my  statement. I would be pleased to respond to any questions that you might  have.
section_title: Contacts and Acknowledgments
paragraphs: For further information on this testimony, please contact  Bernice Steinhardt, Director, Strategic Issues, at (202) 512-6806 or  steinhardtb@gao.gov. Individuals making key contributions to this  testimony include Susan Ragland, Allen Lomax, Catherine Myrick,  Susan Sato, and Jordan Wicker.
This is a work of the U.S. government and is not subject to copyright protection in the  United States. The published product may be reproduced and distributed in its entirety  without further permission from GAO. However, because this work may contain  copyrighted images or other material, permission from the copyright holder may be  necessary if you wish to reproduce this material separately.