Department of Homeland Security: Continued Actions Needed to Strengthen Oversight and Coordination of Research and Development
Published: 2014-07-31T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Conducting R&D on technologies for detecting, preventing, and mitigating terrorist threats is vital to enhancing the security of the nation. Since its creation, DHS has spent billions of dollars researching and developing technologies used to support its missions including securing the border, and detecting nuclear material among others. Within DHS, S&T conducts and is responsible for coordinating R&D across the department. Other components also conduct R&D to support their respective missions.
This statement discusses (1) how much DHS invests in R&D and the extent to which DHS has policies and guidance for defining and overseeing its R&D efforts across the department, (2) the extent to which R&D is coordinated across DHS, and (3) the results of DHS border and maritime security R&D efforts and the extent to which DHS has obtained feedback on these efforts. This statement is based on GAO's previously issued work from September 2012 to September 2013, and selected updates conducted in July 2014 on the status of GAO's prior recommendations. To conduct the updates, GAO reviewed agency documentation.
section_title: What GAO Found
paragraphs: In September 2012, GAO reported that the Department of Homeland Security (DHS) did not know the total amount its components invested in research and development (R&D) and did not have policies and guidance for defining R&D and overseeing R&D resources across the department. According to DHS, its Science & Technology Directorate (S&T), Domestic Nuclear Detection Office (DNDO), and Coast Guard were the only components that conducted R&D, and GAO found that these were the only components that reported budget authority, obligations, or outlays for R&D activities to the Office of Management and Budget. However, GAO identified an additional $255 million in R&D obligations made by other DHS components. At the time of GAO's review, DHS reported it was difficult to identify all R&D investments across the department because DHS did not have a department wide policy defining R&D or guidance directing components how to report all R&D activities. GAO recommended that DHS develop policies to assist components in better understanding how to report R&D activities and better position DHS to determine R&D investments. DHS concurred with the recommendation and, as of July 2014, had updated its guidance to include a definition of R&D but had not yet determined the most effective path to guide R&D across the department. GAO will continue to monitor DHS's efforts to develop its approach for overseeing R&D at the department.
GAO also reported in September 2012 that S&T had taken some steps to coordinate R&D efforts across DHS, but the department's R&D efforts were fragmented and overlapping, which increased the risk of unnecessary duplication. GAO recommended that DHS develop a policy defining roles and responsibilities for coordinating R&D and establish a mechanism to track all R&D projects to help DHS mitigate existing fragmentation and overlap and reduce the risk of unnecessary duplication. DHS concurred with the recommendation. As of July 2014, S&T has not developed new policy guidance but is conducting portfolio reviews across the department, as directed by the fiscal year 2013 appropriations act, aimed at coordinating R&D activities. GAO will continue to monitor DHS's efforts to develop a policy to better coordinate and track R&D activities at the department.
In September 2013, GAO reported that DHS border and maritime R&D components reported producing 97 R&D deliverables from fiscal year 2010 through 2012 at an estimated cost of $177 million. GAO found that the type of border and maritime R&D deliverables produced by S&T, the Coast Guard, and DNDO varied, and R&D customers GAO met with had mixed views on the impact of the deliverables. These deliverables included knowledge products and reports, technology prototypes, and software. For example, S&T developed prototype radar and video systems for use by Border Patrol. However, GAO reported that S&T had not established timeframes for collecting and evaluating feedback on the extent to which deliverables met customers' needs. GAO recommended that S&T collect such feedback from its customers to better determine the usefulness and impact of its R&D projects and deliverables and make better-informed decisions regarding future work. As of July 2014, DHS had taken steps to address this recommendation, including making plans to gather customer feedback. GAO will continue to monitor DHS's efforts in this area.
section_title: What GAO Recommends
paragraphs: In its prior reports, GAO recommended, among other things, that DHS develop policies and guidance for defining, overseeing, coordinating, and tracking R&D activities across the department; and that S&T collect and evaluate feedback from its customers. DHS concurred with GAO's recommendations and has actions underway to address them.

section_title: Letter
paragraphs: I appreciate the opportunity to testify today about our prior work  discussing the results of the Department of Homeland Security’s (DHS)  research and development (R&D) efforts, including the extent to which its  R&D efforts are coordinated within DHS and the results of DHS’s border  and maritime security R&D efforts. According to the Office of  Management and Budget (OMB), R&D activities comprise creative work  undertaken on a systematic basis in order to increase the stock of  knowledge, including knowledge of man, culture, and society, and the use  of this stock of knowledge to devise new applications. R&D is further  broken down into the categories of basic research, applied research, and  development.
Conducting R&D on technologies for detecting, preventing, and mitigating  terrorist threats is vital to enhancing the security of the nation. DHS,  through its Science and Technology Directorate (S&T) and other  components, conducts research, development, testing, and evaluation of  new technologies that are intended to achieve a range of homeland  security goals, including detecting and preventing the unauthorized entry  of persons or contraband into the United States; strengthening efforts to  prevent and respond to nuclear, biological, explosive, and other types of  attacks; and securing U.S. ports and inland waterways. DHS S&T has  responsibility for coordinating and integrating all R&D activities of the  department, as provided by the Homeland Security Act of 2002.
Although S&T conducts R&D and has responsibility for coordinating R&D,  other DHS components, including the Domestic Nuclear Detection Office  (DNDO) and the U.S. Coast Guard, conduct R&D in support of their  respective missions. Since it began operations in 2003, DHS, through  both S&T and other components, has spent billions of dollars researching  and developing technologies used to support a wide range of missions.
With respect to border and maritime R&D specifically, S&T’s Borders and  Maritime Security Division (BMD) is responsible for most of S&T’s border  and maritime related R&D and its primary DHS customer is U.S. Customs  and Border Protection (CBP). Within S&T, the Office of University  Programs manages the DHS Centers of Excellence, which constitute a  network of universities that conduct research for DHS component  agencies, with two centers dedicated specifically to border and maritime  R&D. DNDO conducts R&D applicable to border and maritime security as  it relates to its mission of detecting the use of an unauthorized nuclear  explosive device, fissile material, or radiological material in the United  States. The U.S. Coast Guard’s R&D efforts support all of the various  Coast Guard missions, such as search and rescue, migrant interdiction,  and marine safety.
My testimony today addresses (1) how much DHS invests in R&D and the  extent to which it has policies and guidance for defining R&D and  overseeing R&D resources and efforts across the department; (2) the  extent to which R&D is coordinated within DHS to prevent overlap,  fragmentation, and unnecessary duplication across the department; and  (3) the results of DHS’s border and maritime security R&D and the extent  to which DHS obtained and evaluated feedback on these efforts.
This statement is based on our previous reports and testimonies issued  from September 2012 to September 2013 with selected updates  conducted in July 2014 related to S&T’s efforts to better manage and  coordinate its border and maritime R&D efforts. To conduct our earlier  work, among other things, we analyzed data related to DHS’s R&D  budget authority for fiscal years 2010 through 2013, R&D contracts  issued by components to private industry and universities for fiscal years  2007 through 2011, and the Department of Energy’s (DOE) national  laboratories from fiscal years 2010 through 2012 to identify how much  DHS components obligated for R&D-related work at the national  laboratories. We also met with R&D project managers and customers. For  the selected updates, we reviewed agency documentation on DHS’s  progress in implementing our prior recommendations. The reports cited  provide detailed explanations of our scope and methodology.conducted this work in accordance with generally accepted government  auditing standards. Those standards require that we plan and perform the  audit to obtain sufficient, appropriate evidence to provide a reasonable  basis for our findings and conclusions based on our audit objectives. We  believe that the evidence obtained provides a reasonable basis for our  findings and conclusions based on our audit objectives.
subsections: section_title: DHS Does Not Know Its Total Investment in R&D, but Has Taken Some Steps to Update Guidance
paragraphs: In September 2012, we found that DHS did not know how much its  components invested in R&D, making it difficult to oversee R&D efforts  across the department. According to DHS budget officials, S&T, DNDO,  and the U.S. Coast Guard were the only components that conducted R&D  and we found that they were the only components that reported budget  authority, obligations, or outlays for R&D activities to OMB as part of the  budget process. However, we reported that the data DHS submitted to  OMB underreported DHS’s R&D obligations because DHS components  obligated money for R&D contracts that were not reported to OMB as  R&D. Specifically, for fiscal year 2011, we identified an additional $255  million in R&D obligations by other DHS components. These obligations  included DHS components providing S&T with funding to conduct R&D on  their behalf and components obligating funds through contracts directly to  industry, universities, or with DOE’s national laboratories for R&D.
Further, we found that the data for fiscal years 2010 through 2013 DHS  submitted to OMB also underreported DHS’s R&D budget authority and  outlays because DNDO did not properly report at least $293 million in  R&D budget authority and at least $282 million in R&D outlays. We  reported that DHS budget officials agreed that DHS underreported its  R&D spending and when asked, could not provide a reason why the  omission was not flagged by DHS review.
In addition, in our 2012 report, we found that DHS’s R&D budget  accounts included a mix of R&D and non-R&D spending. For fiscal year  2011, we estimated that 78 percent of S&T’s Research, Development,  Acquisition, & Operations account, 51 percent of DNDO’s Research,  Development, & Operations account, and 43 percent of the Coast  Guard’s R&D budget account funded R&D activities. As a result, this  further complicated DHS’s ability to identify its total investment in R&D.
We also reported in September 2012 that DHS did not have a department  wide policy defining R&D or guidance directing components how to report  R&D activities. As a result, we concluded that it was difficult to identify the  department’s total investment in R&D, which limited DHS’s ability to  oversee components’ R&D efforts and align them with agency wide R&D  goals and priorities, in accordance with Standards for Internal Control in  the Federal Government.OMB’s definition of R&D, but the definition was broad and its application  may not be uniform across components, and thus, R&D investments may  not always be identified as R&D. We found that the variation in R&D  definitions may contribute to the unreliability of the reporting mechanisms   DHS officials told us at the time that DHS used  for R&D investments in budget development and execution, as discussed  above.
We recommended that DHS develop and implement policies and  guidance for defining and overseeing R&D at the department that include,  among other things, a well-understood definition of R&D that provides  reasonable assurance that reliable accounting and reporting of R&D  resources and activities for internal and external use are achieved. DHS  agreed with our recommendation and stated that it planned to evaluate  the most effective path forward to guide uniform treatment of R&D across  the department in compliance with OMB rules and was considering a  management directive, multi-component steering committee, or new  policy guidance to help better oversee and coordinate R&D. As of July  2014, DHS has updated its guidance to include a definition of R&D, but  as discussed in more detail below efforts to develop a specific policy  outlining R&D roles and responsibilities and a process for coordinating  R&D with other offices remain ongoing and have not yet been  completed. We will continue to monitor DHS’s efforts to implement these  recommendations.
section_title: S&T Has Taken Some Actions to Coordinate R&D across DHS, but R&D Activities are Fragmented and Overlapping
paragraphs: We reported in September 2012 that the Homeland Security Act of 2002  provides S&T with the responsibility for, among other things, coordinating  and integrating all research, development, demonstration, testing, and  evaluation activities within DHS and establishing and administering the  primary R&D activities of the department. S&T developed coordination  practices that fall into four general categories: (1) S&T component  liaisons, (2) R&D agreements between component heads and S&T, (3)  joint R&D strategies between S&T and components, and (4) various R&D  coordination teams made up of S&T and component project managers,  which are discussed in detail in our 2012 report and 2013 testimony.
Despite S&T’s efforts to coordinate R&D activities, in September 2012,  we reported that R&D at DHS was inherently fragmented because several  components within DHS—S&T, the Coast Guard, and DNDO—were each  given R&D responsibilities in law, and other DHS components may  pursue and conduct their own R&D efforts as long as those activities are  coordinated through S&T. Fragmentation among R&D efforts at DHS may  be advantageous if the department determines that it could gain better or  faster results by having multiple components engage in R&D activities  toward a similar goal; however, it can be disadvantageous if those  activities are uncoordinated or unintentionally overlapping or duplicative.  Specifically, we found at least six department components involved in  R&D activities in our review of data on about 15,000 federal procurement  contract actions coded as R&D taken by DHS components from fiscal  years 2007 through 2012. We examined 47 R&D contracts awarded by  these components—selected because they appeared to have similar  activities to another contract—and found 35 instances among 29  contracts in which the contracts overlapped with activities conducted  elsewhere in the department. Taken together, these 29 contracts were  worth about $66 million. In one example of the overlap, we found that two  DHS components awarded five separate contracts that each addressed  detection of the same chemical.
While we did not identify instances of unnecessary duplication among  these contracts, in September 2012 we found that DHS had not  developed a policy defining who is responsible for coordinating R&D  activities at DHS that could help prevent overlap, fragmentation, or  unnecessary duplication and did not have tracking mechanisms or  policies to help ensure that overlap is avoided and efforts are better  coordinated consistent with Standards for Internal Control in the Federal  Government. S&T officials told us at the time that a process did not exist  at DHS or within S&T to prevent overlap or unnecessary duplication but  that relationships with components mitigate that risk. They also stated  that S&T has improved interactions with components over time. We  concluded that the existence of overlapping R&D activities coupled with  the lack of policies and guidance defining R&D and coordination  processes was an indication that not all R&D activities at DHS were  coordinated to ensure that R&D is not unnecessarily duplicative. We also  found in September 2012 that neither DHS nor S&T tracked all ongoing  R&D projects across the department, including R&D activities contracted  through the national laboratories. As part of our review, we identified 11  components that reimbursed the national laboratories for R&D from fiscal  years 2010 through 2012, but S&T’s Office of National Laboratories could  not provide us with any information on those activities and told us it did  not track them. According to S&T, the Office of National Laboratories’  ability to provide information on activities across the department is limited  by components inconsistently operating within the defined process for  working with the national laboratories.
As a result, we recommended that DHS develop and implement policies  and guidance for overseeing R&D that includes, among other things, a  description of the department’s process and roles and responsibilities for  overseeing and coordinating R&D investments and efforts, and a  mechanism to track existing R&D projects and their associated costs  across the department. DHS agreed with our recommendation and stated  at the time that S&T was implementing a collaborative, end-user focused  strategy to coordinate and interact with components to better ensure  S&T’s efforts aligned with components’ needs and that it was considering  developing new policy guidance for R&D activities across the department.  As of July 2014, DHS has not developed new policy guidance but is  conducting portfolio reviews across the department, as directed in  committee reports accompanying the fiscal year 2013 DHS appropriation  act, aimed at coordinating R&D activities.recommendation to develop a policy that defines roles and responsibilities  for coordinating R&D and coordination processes, as well as a  mechanism that tracks all DHS R&D projects, could better position DHS  to mitigate the risk of overlapping and unnecessarily duplicative R&D  projects. We will continue to monitor DHS’s efforts to develop a policy to  better coordinate and track R&D activities at the department.
section_title: S&T Has Taken Steps to Obtain Feedback and Evaluate the Impact of Its Border and Maritime R&D Efforts
subsections: section_title: Costs and Types of Completed Border and Maritime R&D Projects Varied
paragraphs: In September 2013, we reported that DHS S&T, Coast Guard, and DNDO  reported producing 97 Border and Maritime R&D deliverables at an  estimated cost of $177 million from fiscal years 2010 through 2012. The  type of border and maritime R&D deliverables produced by these R&D  entities were wide-ranging in their cost and scale, and included  knowledge products and reports, technology prototypes, and software.  For example:
Knowledge products or reports: One of the DHS Centers of  Excellence developed formulas and models to assist in randomizing  Coast Guard patrol routes and connecting networks together to assist  in the detection of small vessels.
Technology prototypes: S&T BMD developed prototype radar and  upgraded video systems for use by Border Patrol agents and a  prototype scanner to screen interior areas of small aircraft without  removing panels or the aircraft skin.
Software: DNDO developed software that extracts data from radiation  portal monitors and uses the data to improve algorithms used in  detecting radioactive material.
As we reported in September 2013, R&D customers we met with had  mixed views on the impact of the R&D deliverables they received. For  example, we reviewed 20 S&T BMD deliverables produced from fiscal  years 2010 through 2012 at a cost of $28.7 million. We found that the  customers of 7 deliverables stated that the deliverables met their office’s  needs, customers of 7 did not, customers of 4 did not know, and  customers for 2 could not be identified.CBP’s Office of Technology Innovation and Acquisition reported that  S&T’s analysis and test results on aircraft-based use of wide area  surveillance technology helped CBP to make a decision on whether it  should pursue acquiring such technology. In cases where customers said  that the deliverables were not meeting their needs, the customers  explained that budget changes, other ongoing testing efforts, or changes  in mission priorities were the reasons deliverables had not met their  needs, and customers pointed out that their relationship with S&T had  been positive and highly collaborative. In other cases, customers pointed  out that while the deliverable had not been used as intended, it informed  their office’s decision making and helped to rule out certain technologies  as possibilities. In this regard, the customers felt the R&D was successful,  despite the fact that the deliverable had not or was not being used.
For example, customers within  S&T BMD officials explained that some of its older projects did not have  identifiable customers because its former process for selecting projects  created the potential to engage in R&D without a clear commitment from  the customer. In February 2012, S&T issued a new project management  guide that requires project managers to specify the customer by office  and name, and to describe customer support for the project, including  how the customer has demonstrated commitment for and support of the  project. S&T officials said they believed this new process would prevent  future R&D funding from going towards projects without a clear customer.
Additionally, we reported that from fiscal year 2010 through fiscal year  2012, DNDO produced 42 deliverables at a cost of $115.9 million, which  included 6 discontinued projects and 36 projects that were either  transitioned to the next phase of R&D or were completed. DNDO R&D is  different from the R&D of S&T for many reasons. For one, a DNDO  project may start at a basic research level, and may end up being merged  into other similar efforts in order to achieve a higher project goal. In these  cases, the R&D customers are DNDO project managers rather than  another DHS customer, such as CBP. We discussed 5 DNDO R&D  deliverables at various R&D phases with DNDO officials—4 of which were  deliverables from ongoing or completed projects and 1 of which was a  discontinued project. Two of the 5 projects we discussed had moved from  early-stage R&D into other projects further along in DNDO’s project  management process. Two of the 5 projects were completed, with 1  project that was reported to have provided information that informed  furthered DNDO decision-making and the other project resulting in a  commercialized product. With regard to the 1 discontinued project, DNDO  officials said that the particular project’s technology was determined to be  too expensive to continue pursuing.
section_title: S&T Did Not Gather and Evaluate Feedback
paragraphs: We reported that although S&T project managers sought feedback from  their customers during the execution of projects, S&T did not gather and  evaluate feedback from its customers to determine the impact of its  completed R&D efforts and deliverables, making it difficult to determine if  the R&D met customer needs. Further, in some cases, the customer of  S&T’s R&D was not clear or the results of the R&D were unknown. For  example, a CBP customer identified by S&T was aware of two R&D  deliverables that S&T said were transitioned to his office, but the official  was unable to provide additional information on the project’s impact.  According to S&T officials, since they deal with multiple DHS components  and are not within the same agencies as its customers, it is sometimes  difficult to identify who the customer of the R&D is and also difficult to  determine what the impact of the R&D was. S&T officials also stated that  in S&T’s 2012 update to its project management guide, in its project  closeout process, S&T had included a step to collect feedback from all  relevant customers and a template for collecting this feedback.
While we found in September 2013 that S&T had developed a process  and template to collect feedback at the end of each project and  incorporated this into its project management plan, we also found that it  did not plan to survey customers each time it provides a deliverable to the  customer. This is relevant because S&T projects are often conducted  over several years before they are concluded and these projects also  often produce multiple deliverables for a customer over many years that  are designed to meet a specific operational need. For example, a Ground  Based Technologies project began in fiscal year 2006 and was slated to  continue through fiscal year 2018. During this period, S&T provided  multiple R&D deliverables to CBP—including test results comparing  different ground based radar systems. The National Academy of Sciences  has stated that feedback from both R&D failures and successes may be  communicated to stakeholders and used to modify future investments.At the time of our report, S&T had not established timeframes and  milestones for collecting and evaluating feedback from its customers on  the extent to which the deliverables it provides were meeting its  customer’s needs.
As a result, we recommended that S&T establish timeframes and  milestones for collecting and evaluating feedback from its customers to  determine the usefulness and impact of both its R&D projects and project  deliverables, and use it to make better-informed decisions regarding  future work. S&T officials concurred with the recommendation at the time  of our review, and reported that it was developing R&D strategies with  DHS components, which would include a strategic assessment of  components’ R&D needs and updated annually on the basis of customer  feedback. As of July 2014, S&T has completed strategic plans with  Border Patrol, the Transportation Security Administration (TSA), and the  Secret Service. Further, at the time of our review, S&T reported that it  was developing a new project management guide to improve R&D  management at all stages of development, and that the guide would  include a template for project managers to use to gather customer  feedback on a more consistent basis. In November 2013, S&T finalized  its guide which includes a customer survey template to obtain feedback  on the quality, timeliness, and relevance of a deliverable, as well as  detailed descriptions of actions project managers should take throughout  a project to ensure the R&D is aligned with customer needs. We will  continue to review the implementation of these actions and to determine  whether they fully address the intent of our recommendation.
section_title: DHS Border and Maritime R&D Agencies Have Taken Action to Improve Internal and External R&D Coordination
paragraphs: In September 2013, we also reported that S&T’s BMD, the Coast Guard,  and DNDO reported taking a range of actions to coordinate with one  another and their customers to ensure that R&D is addressing high  priority needs. Officials from BMD identified several ways in which it  coordinated R&D activities with its customers, which are primarily offices  within CBP. For example, BMD officials reported having a person detailed  to CBP’s Office of Technology Innovation and Acquisition and identified  its integrated product teams, such as its cross border tunnel threat team,  and jointly funded projects as ways in which the division worked to ensure  its R&D efforts were coordinated with CBP.  We also found that  opportunities existed for DHS to enhance coordination with universities  conducting R&D on its behalf. Specifically, we reported that the S&T  Office of University Programs could help ensure that the approximately $3  million to $4 million a year dedicated to each university center is used  more effectively by more carefully considering data needs, potential  access issues, and potential data limitations with its federal partners  before approving projects. We recommended that S&T ensure design  limitations with regard to data reliability, accessibility, and availability are  reviewed and understood before approving Center of Excellence R&D  projects. S&T Office of University Programs officials concurred with the  recommendation and discussed the variety of ways in which centers and  DHS components collaborate and share information. Office of University  Programs officials stated that the office’s process for soliciting research  topics and evaluating proposals is good and that it keeps the centers  flexible. However, officials from DHS’s primary land border security  Center of Excellence reported challenges with respect to a lack of clarity  regarding protocols for access to DHS information when conducting R&D.  Specifically, officials from this center reported that they have been  regularly unable to obtain data from CBP to complete research it was  conducting on CBP’s behalf, which resulted in delays and terminated  R&D projects.
Given the challenges raised by officials from universities leading the R&D  for land border security, we recommended that S&T conduct a more  rigorous review of potential data-related challenges and limitations at the  start of a project in order to help R&D customers (such as CBP) identify  data requirements and potential limitations up front so that money is not  allocated to projects that potentially cannot be completed. In concurring  with our recommendation, S&T Office of University Programs officials  agreed that making sure their clients take additional steps to identify data  requirements up front could help address these challenges and following  our review had started taking steps to address the recommendation. For  instance, in September 2013, the Office of University Programs reported  that it was working to develop standard guidelines and protocols that  would apply to all of its centers of excellence. These protocols were to  describe how data sets must be modified to enable their use in open- source research formats. In March 2014, the Office of University  Programs and the National Center for Border Security and Immigration, a  DHS S&T Center of Excellence, co-hosted a workshop to identify  common problems the centers have in accessing data from DHS,  understand DHS constraints in sharing data, and develop best practices  for requesting and sharing data between the centers of excellence and  DHS. We believe this is a step in the right direction and should move S&T  closer toward meeting the intention of our recommendation. We will  continue to monitor DHS’s efforts in this area.
Chairman Bucshon, Chairman Broun, Ranking Member Lipinski, Ranking  Member Maffei, and members of the committee, this completes my  prepared statement. I would be happy to respond to any questions you  may have at this time.
section_title: GAO Contact and Staff Acknowledgements
paragraphs: If you or your staff members have any questions about this testimony,  please contact me at (202) 512-9627 or Maurerd@gao.gov. Contact  points for our Offices of Congressional Relations and Public Affairs may  be found on the last page of this statement. Key contributors to this  statement include Adam Hoffman, Assistant Director; Aditi Archer, and  Charlotte Gamble. Francis Cook, Michele Fejfar, Emily Gunn, Richard  Hung, Gary Malavenda, and Linda Miller also made contributions to this  testimony.
This is a work of the U.S. government and is not subject to copyright protection in the  United States. The published product may be reproduced and distributed in its entirety  without further permission from GAO. However, because this work may contain  copyrighted images or other material, permission from the copyright holder may be  necessary if you wish to reproduce this material separately.