Federal Emergency Management Agency: Opportunities to Achieve Efficiencies and Strengthen Operations
Published: 2014-07-24T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Preparing for, responding to, and recovering from disasters is becoming increasingly complex and costly. GAO reported that from fiscal years 2002 through 2013, the federal government appropriated about $41 billion for preparedness grant programs and $6.2 billion to FEMA's Disaster Relief Fund in fiscal year 2014. In addition, FEMA obligated over $80 billion in federal disaster assistance for major disasters declared from fiscal years 2004 through 2011 and responded to more disasters than in any other year in its history during fiscal year 2011.
The larger number and size of disasters has required increasingly complex and costly FEMA operations and processes to prepare for and respond to these events. For example, Hurricane Sandy in September 2012 required one of the largest deployment of disaster personnel in FEMA's history. Similarly, FEMA's own administrative costs—such as the cost to house and deploy its disaster personnel—have also increased.
This testimony discusses GAO's work on opportunities to enhance efficiencies in FEMA's operations in three areas: (1) disaster administrative costs, (2) workforce management, and (3) preparedness grant management. This testimony is based on previous GAO reports issued from 2008 to 2014 with selected updates and preliminary observations from GAO's ongoing work on disaster administrative costs and workforce management issues in response to Hurricane Sandy.
section_title: What GAO Found
paragraphs: GAO's recent and ongoing work examining the Federal Emergency Management Agency's (FEMA) administrative costs of providing disaster assistance highlights opportunities to increase efficiencies and potentially reduce these costs. In September 2012, GAO reported that FEMA's administrative costs for disaster assistance had doubled in size as a percentage of the overall cost of the disasters since fiscal year 1989, and often surpassed its targets for controlling administrative costs. GAO also concluded that FEMA's administrative costs were increasing for all sizes of disasters and for all types of disaster assistance. FEMA issued guidelines intended to improve the efficiency of its efforts and to help reduce administrative costs. However, FEMA did not make this guidance mandatory because it wanted to allow for flexibility in responding to a variety of disaster situations. In 2012, GAO recommended that the FEMA Administrator implement goals for administrative cost percentages and monitor performance to achieve these goals. However, as of June 2014, FEMA had not taken steps to implement GAO's recommendation. GAO's ongoing work indicates that FEMA is implementing a new system to, among other things, collect and analyze data on the administrative costs associated with managing disasters to enable managers to better assess performance. However, according to officials, FEMA is still working on systematically collecting the data. As a result, it is too early to assess whether this effort will improve efficiencies or reduce administrative costs.
GAO has also reported on opportunities to strengthen and increase the effectiveness of FEMA's workforce management. Specifically, GAO reviewed FEMA human capital management efforts in 2012 and 2013 and has made a number of related recommendations, many of which FEMA has implemented; some of which are still underway. For example, GAO recommended that FEMA identify long-term quantifiable mission-critical goals and establish a time frame for completing the development of quantifiable performance measures for workforce planning and training, establish lines of authority for agency-wide efforts related to workforce planning and training, and develop systematic processes to collect and analyze workforce and training data. FEMA concurred and is still working to address these recommendations. For example, FEMA's deployment of its disaster assistance workforce during the response to Hurricane Sandy revealed a number of challenges. In response, according to agency officials, FEMA is, among other things, analyzing its disaster assistance workforce structure to ensure the agency is capable of responding to large and complex incidents. GAO will continue to evaluate these efforts to assess their effectiveness.
In March 2011, GAO reported that FEMA could enhance the coordination of application reviews of grant projects across four of the largest preparedness grants (Urban Areas Security Initiative, State Homeland Security Program, Port Security Grant Program, and Transit Security Grant Program) which have similar goals, fund similar types of projects, and are awarded in many of the same urban areas. GAO recommended that FEMA coordinate the grant application process to reduce the potential for duplication. FEMA has attempted to use data to coordinate two programs and also proposed to consolidate its preparedness grant programs, but FEMA's data system has been delayed, and Congress did not approve FEMA's consolidation proposal for either fiscal year 2013 or 2014.

section_title: Letter
paragraphs: I appreciate the opportunity to testify today about efforts by the Federal  Emergency Management Agency (FEMA), a component of the  Department of Homeland Security (DHS), to more efficiently lead the  nation’s efforts to prepare for, respond to, and recover from disasters. We  have reported on a broad range of issues related to FEMA’s programs  and operations over the last decade and currently have ongoing work for  this committee evaluating disaster administrative costs and workforce  planning, as well as federal catastrophic response efforts, post-disaster  mitigation, and states’ disaster budgeting practices.
Large disasters are increasingly complex and costly. For example, in  fiscal year 2014 Congress appropriated $6.2 billion for FEMA’s Disaster  Relief Fund.  As we reported in 2012, FEMA obligated over $80 billion in  federal assistance for major disasters declared from fiscal years 2004  through 2011. In addition, the increasing number and size of disasters  require an increasingly large federal disaster assistance workforce. As we  reported in April 2012, FEMA responded to more disasters in fiscal year  2011 than in any other year in its history when FEMA’s disaster  assistance workforce responded to 124 declared major disasters and  emergencies. Then, in October 2012, Hurricane Sandy hit the East Coast  and more than 17,000 federal personnel—including more than 7,500  FEMA staff—were deployed as part of one of the largest deployment of  personnel ever. For fiscal year 2014, Congress appropriated $1.5 billion    As we reported last year,  for six FEMA preparedness grant programs.from fiscal years 2002 through 2013, the federal government appropriated  about $41 billion to a variety of DHS preparedness grant programs.larger number and size of disasters have required increasingly complex  and costly FEMA operations and processes to prepare for and respond to  these events, and increases the importance that FEMA operations are as  efficient and effective as possible.
My testimony today discusses our previously published and ongoing work  on opportunities to enhance efficiencies in FEMA’s operations in the  following areas: (1) disaster administrative costs, (2) workforce  management, and (3) preparedness grant management.
This statement is based on our prior work issued from November 2008 to  May 2014 related to FEMA’s disaster administration, human capital  management, and preparedness grants and selected updates.  To  perform the work for these reports, we reviewed key federal documents  and efforts such as FEMA’s analyses and reports on these issues,  analyzed data from FEMA and interviewed FEMA officials, analyzed  information collected from other relevant agencies, and visited various  locations. Further details on the scope and methodology of our previously  issued reports are available within each of the published products. In  addition, this statement includes preliminary observations from our  ongoing work assessing the rising administrative costs of providing  disaster assistance and FEMA workforce management efforts.  For the  ongoing work, we are reviewing FEMA documents such as its  administrative cost analyses and Strategic Human Capital Plan, among  other things. We are also conducting our own analyses of FEMA data and  interviewing FEMA and select state officials. We shared a copy of the  new information in this statement with DHS officials.
The work upon which this testimony is based was conducted in  accordance with generally accepted government auditing standards.
Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
subsections: section_title: Costs of Providing Disaster Assistance Have Increased, but FEMA Has Not Taken Recommended Action to Control Its Administrative Costs
paragraphs: We reported in September 2012 that FEMA’s administrative costs had  been increasing for all sizes of disasters.  According to FEMA,  administrative costs include, among other things, the salary and travel  costs for its disaster workforce, rent and security expenses associated  with establishing and operating its field office facilities, and supplies and  information technology support for its deployed staff. In September 2012,  based on our analysis of 1,221 small, medium, and large federal disaster  declarations during fiscal years 1989 through 2011, we found that the  average administrative cost percentage for these disaster declarations  doubled from 9 percent in the 1989-to-1995 period to 18 percent in the  2004-to-2011 period, as shown in table 1.
We also found that the growth in administrative costs occurred for all  types of disaster assistance, including those related to providing  Individual Assistance, Public Assistance, and assistance for those  disasters that provided both Individual Assistance and Public Assistance.As shown in table 2, since fiscal year 1989, administrative cost  percentages doubled for disaster declarations with Individual Assistance  only, quadrupled for declarations with Public Assistance only, and  doubled for declarations with Public Assistance and Individual Assistance.
To address these rising costs, FEMA issued guidelines and targets  intended to improve the efficiency of its efforts and to help reduce  administrative costs. In November 2010, FEMA issued guidance on how  to better control administrative costs associated with disaster  declarations. The guide noted that incidents of similar size and type had  witnessed growing administrative costs for 20 years, and that, in the past,  little emphasis had been placed on controlling overall costs. The  document provided guidance on how to set targets for administrative cost  percentages, plan staffing levels, time the deployment of staff, and  determine whether to use “virtual” field offices instead of physical field  offices. However, in September 2012, we found that FEMA did not require  that this guidance be followed or targets be met because the agency’s  intent was to ensure that it was providing guidance to shape how its  leaders in the field think about gaining and sustaining efficiencies in  operations rather than to lay out a prescriptive formula. As a result, we  concluded that FEMA did not track or monitor whether its cost targets  were being used or achieved.
In September 2012, we also found that in many cases, FEMA exceeded  its cost targets for administrative costs. Specifically, based on our  analysis of the 539 disaster declarations during fiscal years 2004 through  2011, we found that 37 percent of the declarations exceeded the 2010  administrative cost percentage targets. Specifically:
For small disaster declarations (total obligations of less than $50  million), FEMA’s target range for administrative costs is 12 percent to  20 percent; for the 409 small declarations that we analyzed, 4 out of  every 10 had administrative costs that exceeded 20 percent.
For medium disaster declarations (total obligations of $50 million to  $500 million), the target range for administrative costs is 9 percent to  15 percent; for the 111 declarations that we analyzed, almost 3 out of  every 10 had administrative costs that exceeded 15 percent.
For large disaster declarations (total obligations greater than $500  million to $5 billion), the target range for administrative costs is 8  percent to 12 percent; for the 19 large declarations that we analyzed,  about 4 out of every 10 had administrative costs that exceeded 12  percent.
As a result, in September 2012, we recommended that FEMA implement  goals for administrative cost percentages and monitor performance to  achieve these goals. However, as of July 2014, FEMA had not taken  steps to implement our recommendation. In December 2013, FEMA  officials stated that they are implementing a system called FEMAStat to,  among other things, collect and analyze data on the administrative costs  associated with managing disasters to enable managers to better assess  performance and progress within the organization. As part of the  FEMAStat effort, in 2012 and 2013, FEMA collected and analyzed data  on the administrative costs associated with managing disasters. However,  as of July 2014, FEMA is still working on systematically collecting the  data and utilizing them to develop a model for decision making. As a  result, it is too early to assess whether this effort will improve the  efficiencies or reduce the cost associated with administering assistance in  response to disasters.
As part of our ongoing work, we will be reviewing these efforts and  working with FEMA to better understand the progress the agency has  made in monitoring and controlling its administrative costs associated  with delivery of disaster assistance and its efforts to decrease the  administrative burden associated with its Public Assistance program.
section_title: Opportunities to Increase the Effectiveness of FEMA’s Workforce
paragraphs: We have also reported on opportunities to strengthen and increase the  effectiveness of FEMA’s workforce. More specifically, we previously  reported on various FEMA human capital management efforts (as well as  human capital management efforts across the federal government) and  have made a number of related recommendations for improvement.  FEMA has implemented some of these, but others are still underway.  Specifically:  In June 2011, we found that FEMA’s Strategic Human Capital Plan  did not define critical skills and competencies that FEMA would need  in the coming years or provide specific strategies and program  objectives to motivate, deploy, and retain employees, among other  things. As a result, we recommended that FEMA develop a  comprehensive workforce plan that identifies agency staffing and  skills requirements, addresses turnover and staff vacancies, and  analyzes FEMA’s use of contractors. FEMA agreed, and in  responding to this recommendation, reported that it had acquired a  contractor to conduct an assessment of its workforce to inform the  agency’s future workforce planning efforts.
In April 2012, we found that FEMA had taken steps to incorporate  some strategic management principles into its workforce planning and  training efforts but could incorporate additional principles to ensure a  more strategic approach is used to address longstanding  management challenges. Further, FEMA’s workforce planning and  training could be enhanced by establishing lines of authority for these  efforts. We also found that FEMA had not developed processes to  systematically collect and analyze agencywide workforce and training  data that could be used to better inform its decision making. We  recommended that FEMA: identify long-term quantifiable mission- critical goals that reflect the agency’s priorities for workforce planning  and training; establish a time frame for completing the development of  quantifiable performance measures related to workforce planning and  training efforts; establish lines of authority for agency-wide workforce  planning and training efforts; and develop systematic processes to  collect and analyze workforce and training data. DHS concurred with  all the recommendations and FEMA is still working to address them.  For example, in April 2014, FEMA issued a notice soliciting  contracting services for a comprehensive workforce structure analysis  for the agency. As part of our ongoing review of FEMA’s workforce  management, we are gathering information on FEMA’s other efforts to  address our recommendations.
In May 2012, we reported on the management and training of FEMA  Reservists, a component of FEMA’s workforce, referred to at that time  as Disaster Assistance Employees (DAE).  Specifically, we found  that FEMA did not monitor how the regions implement DAE policies  and how DAEs implement disaster policies across regions to ensure  consistency. While FEMA’s regional DAE managers were responsible  for hiring DAEs, FEMA had not established hiring criteria and had  limited salary criteria. Regarding FEMA’s performance appraisal  system for DAEs, we found that FEMA did not have criteria for  supervisors to assign DAEs satisfactory or unsatisfactory ratings. We  also found that FEMA did not have a plan to ensure DAEs receive  necessary training and did not track how much of the Disaster Relief  Fund was spent on training for DAEs. We recommended, among  other things, that FEMA develop a plan for how it will better  communicate policies and procedures to DAEs when they are not  deployed; establish a mechanism to monitor both its regions’  implementation of DAE policies and DAEs’ implementation of FEMA’s  disaster policies; establish standardized criteria for hiring and  compensating DAEs; and establish a plan to ensure that DAEs have  opportunities to participate in training and are qualified. DHS  concurred with the recommendations and FEMA has taken steps to  address several of them. For example, in June 2012, FEMA  implemented a communication strategy with its reservist workforce  that included video conferences, a web blog series, and a FEMA  weekly bulletin sent to Reservists’ personal email addresses, among  other things. Also, in October 2012, DHS reported that FEMA had  resolved the outstanding issues of inconsistent implementation of  DAE policies by centralizing control over hiring, training, equipment,  and deployment within a single headquarters-based office. FEMA is  working to address our other recommendations, and we will continue  to monitor its progress.
In our March 2013 report, we examined how FEMA’s reservist  workforce training compared with training of other similar agencies,  and the extent to which FEMA had examined these agencies’ training  programs to identify useful practices.  We found that FEMA had not  examined other agencies’ training programs, and therefore, we  recommended that FEMA examine the training practices of other  agencies with disaster reservist workforces to identify potentially  useful practices; DHS concurred with our recommendation and  described plans to address it. As part of our ongoing review, we are  gathering information on FEMA’s efforts to address our  recommendation.
At the request of this committee, we are also currently assessing the  impact of workforce management and development provisions in the  Post-Katrina Act on FEMA’s response to Hurricane Sandy. We also have  plans to conduct additional work to assess the impact of a variety of other  emergency management related provisions in the Post-Katrina Act (for  example, provisions related to FEMA’s contracting efforts, information  technology systems, and disaster relief efforts). Among other things, the  Post-Katrina Act directed FEMA to implement efforts to enhance  workforce planning and development, standards for deployment capabilities, including credentialing of  personnel, and establish a surge capacity force (SCF) to deploy to  natural and man-made disasters, including catastrophic incidents.
Some of these efforts were highlighted during Hurricane Sandy when  FEMA executed one of the largest deployments of personnel in its history.
5 U.S.C. §§ 10101-10106; 6 U.S.C. §§ 414-415.
For example, the agency’s response to Hurricane Sandy marked the first  activation of the DHS SCF, with nearly 2,400 DHS employees deploying  to New York and New Jersey to support response and recovery efforts.  The agency also launched the new FEMA Qualification System (FQS) on  October 1, 2012, just in time for FEMA employees’ deployment to areas  affected by Hurricane Sandy. In 2012, FEMA also created a new disaster  assistance workforce component called the FEMA Corps. Forty-two  FEMA Corps teams, consisting of approximately 1,100 members, were  deployed to support Hurricane Sandy response and recovery efforts in  the fall of 2012.
FEMA’s deployment of its disaster assistance workforce during the  response to Hurricane Sandy revealed a number of challenges and,  as a result, FEMA is analyzing its disaster assistance workforce  structure to ensure the agency is capable of responding to large and  complex incidents, as well as simultaneous disasters and  emergencies. For example, FEMA reported that: before deployment  for Hurricane Sandy, 28 percent of the staffing positions called for by  FEMA’s force structure analysis were vacant (approximately 47  percent of positions required by the force structure were filled with  qualified personnel, and the remaining 25 percent were filled by  trainees).
Deployment of its disaster workforce nearly exhausted the number of  available personnel. By November 12, 2012, FEMA had only 355  Reservists (5 percent) available for potential deployment: 4,708 (67  percent) were already deployed to ongoing disasters, and 1,854 (26  percent) were unavailable.
Its plans had not fully considered how to balance a large deployment  of personnel and still maintain day-to-day operations.
As part of our ongoing work, we will be evaluating FEMA’s efforts to  address the challenges identified during the agency’s response to  Hurricane Sandy and assessing their impact. We will also determine what  progress the agency has made in its workforce planning and development  efforts.
section_title: Opportunities to Increase the Efficiency of Preparedness Grant Administration
paragraphs: In March 2011, we reported on another area of opportunity for FEMA to  increase the efficiency of its operations—the management of its  preparedness grants.  We found that FEMA could benefit from  examining its grant programs and coordinating its application process to  eliminate or reduce redundancy among grant recipients and program  purposes. As we again reported in February 2012, four of FEMA’s largest  preparedness grants (Urban Areas Security Initiative, State Homeland  Security Program, Port Security Grant Program, and Transit Security  Grant Program) which have similar goals, fund similar types of projects,  and are awarded in many of the same urban areas, have application  review processes that are not coordinated.
In March 2014 in our annual update to our duplication and cost savings  work in GAO’s Online Action Tracker, we reported that FEMA has  attempted to capture more robust data from grantees during applications  for the Port Security Grant Program and the Transit Security Grant  Program—because applicants provide project-level data. However,  applications for the State Homeland Security Grant Program and Urban  Areas Security Initiative do not contain enough detail to allow for the  coordinated review across the four grants, according to FEMA officials.  FEMA intends to begin collecting and analyzing additional project-level  data using a new system called the Non-Disaster Grants Management  System (NDGrants).  However, FEMA officials said that implementation  of NDGrants had been delayed until 2016 because of reduced funding.
While implementing NDGrants should help FEMA strengthen the  administration and oversight of its grant programs, a report released by  the DHS Office of Inspector General (OIG) in May 2014 identified a  number of information control system deficiencies associated with FEMA  development and deployment of the NDGrants system that could limit the  usefulness of the system. Specifically, the OIG reported NDGrants  system deficiencies related to security management, access control, and  configuration management.  According to the OIG’s report, DHS  management concurred with the findings and recommendations in the  report and plans to work with component management to address these  issues. We will continue to monitor FEMA’s implementation of the system  as part of our annual update for our duplication and cost savings work.
FEMA has proposed, through the President’s budget requests to  Congress, to consolidate its preparedness grant programs to streamline  the grant application process, responding to a recommendation we made  in March 2011 by eliminating the need to coordinate application reviews.
GAO, Government Operations: Opportunities to Reduce Potential Duplication in  Government Programs, Save Tax Dollars, and Enhance Revenue GAO-11-318SP,  (Washington, D.C.: Mar 1, 2011.) committees, however, expressed concern that the consolidation plan  lacked detail, and the NPGP was not approved for either fiscal year 2013  or 2014.  Nonetheless, FEMA again proposed the NPGP consolidation  approach for 2015 providing additional details such as clarification and  revised language relating to governance structures under the proposed  program. In responding to questions submitted by the House Committee  on Homeland Security’s Subcommittee on Emergency Preparedness,  Response and Communications in April 2014, FEMA officials reported  that the NPGP would help increase the efficiency of preparedness grants  by requiring fewer grants notices for staff to issue and fewer grants to  award, and reduce processing time and monitoring trips due to the  reduction in the number of grantees. If approved in the future, and  depending on its final form and execution, we believe a consolidated  NPGP could help reduce redundancies and mitigate the potential for  unnecessary duplication and is consistent with our prior recommendation.
Chairman Begich, Ranking Member Paul, and members of the  subcommittee, this completes my prepared statement. I would be happy  to respond to any questions you may have at this time.
section_title: GAO Contacts and Staff Acknowledgements
paragraphs: If you or your staff members have any questions about this testimony,  please contact me at (404) 679-1875 or curriec@gao.gov. Contact points  for our Offices of Congressional Relations and Public Affairs may be  found on the last page of this statement. Christopher Keisling, Assistant  Director; Aditi Archer, Andrew Berglund, Jeffrey Fiore, Michelle R. Su,  Tracey King, David Alexander, and Jessica Orr made contributions to this  testimony.
section_title: GAO Related Products
paragraphs: Disaster Resilience: Actions Are Underway, but Federal Fiscal Exposure  Highlights the Need for Continued Attention to Longstanding Challenges.  GAO-14-603T. Washington, D.C.: May 14, 2014.
Extreme Weather Events: Limiting Federal Fiscal Exposure and  Increasing the Nation’s Resilience. GAO-14-364T. Washington, D.C.:  February 12, 2014.
National Preparedness: FEMA Has Made Progress, but Additional Steps  Are Needed to Improve Grant Management and Assess Capabilities.  GAO-13-637T. Washington, D.C.: June 25, 2013.
FEMA Reservists: Training Could Benefit from Examination of Practices  at Other Agencies. GAO-13-250R. Washington, D.C.: March 22, 2013.
National Preparedness: FEMA Has Made Progress in Improving Grant  Management and Assessing Capabilities, but Challenges Remain.   GAO-13-456T. Washington, D.C.: March 19, 2013.
High-Risk Series: An Update. GAO-13-283. Washington, D.C.: February  14, 2013.
Federal Disaster Assistance: Improved Criteria Needed to Assess a  Jurisdiction’s Capability to Respond and Recover on Its Own.   GAO-12-838. Washington, D.C.: September 12, 2012.
Disaster Assistance Workforce: FEMA Could Enhance Human Capital  Management and Training. GAO-12-538. Washington, D.C.: May 25,  2012.
Federal Emergency Management Agency: Workforce Planning and  Training Could Be Enhanced by Incorporating Strategic Management  Principles. GAO-12-487. Washington, D.C.: April 26, 2012.
Homeland Security: DHS Needs Better Project Information and  Coordination among Four Overlapping Grant Programs. GAO-12-303.  Washington, D.C.: February 28, 2012.
More Efficient and Effective Government: Opportunities to Reduce  Duplication, Overlap and Fragmentation, Achieve Savings, and Enhance  Revenue. GAO-12-449T. Washington, D.C.: February 28, 2012.
Government Operations: Opportunities to Reduce Potential Duplication in  Government Programs, Save Tax Dollars, and Enhance Revenue.   GAO-11-318SP. Washington, D.C.: March 1, 2011.
FEMA: Action Needed to Improve Administration of the National Flood  Insurance Program. GAO-11-297. Washington, D.C.: June 9, 2011   Government Operations: Actions Taken to Implement the Post-Katrina  Emergency Management Reform Act of 2006. GAO-09-59R. Washington,  D.C.: November 21, 2008.
Natural Hazard Mitigation: Various Mitigation Efforts Exist, but Federal  Efforts Do Not Provide a Comprehensive Strategic Framework.   GAO-07-403. Washington, D.C.: August 22, 2007.
High Risk Series: GAO’s High-Risk Program. GAO-06-497T. Washington,  D.C.: March 15, 2006.
Disaster Assistance: Information on the Cost-Effectiveness of Hazard  Mitigation Projects. GAO/T-RCED-99-106. Washington, D.C.: March 4,  1999.
Disaster Assistance: Information on Federal Disaster Mitigation Efforts.  GAO/T-RCED-98-67. Washington, D.C.: January 28, 1998.
Disaster Assistance: Information on Expenditures and Proposals to  Improve Effectiveness and Reduce Future Costs. GAO/T-RCED-95-140.  Washington, D.C.: March 16, 1995.
Federal Disaster Assistance: What Should the Policy Be? PAD-80-39.  Washington, D.C.: June 16, 1980.
This is a work of the U.S. government and is not subject to copyright protection in the  United States. The published product may be reproduced and distributed in its entirety  without further permission from GAO. However, because this work may contain  copyrighted images or other material, permission from the copyright holder may be  necessary if you wish to reproduce this material separately.