Homeland Security Acquisitions: DHS Should Better Define Oversight Roles and Improve Program Reporting to Congress
Published: 2015-03-12T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: In fiscal year 2014, DHS reported it planned to spend approximately $10.7 billion on its major acquisition programs. DHS acquires systems to reduce the probability of a terrorist attack, protect against disease, mitigate natural hazards, and secure borders. Partially in response to GAO recommendations, the department has taken steps to improve acquisition management in recent years, but has not yet implemented many of these recommendations.
GAO was asked to review DHS's oversight of its major acquisition programs. This report addresses (1) steps DHS has taken to improve oversight and gaps that exist, if any, and (2) whether the data PARM provides to DHS and congressional decision makers are accurate and up-to-date.
GAO reviewed DHS policies and procedures and interviewed oversight and acquisition officials from all nine DHS components with at least one major acquisition program with a life-cycle cost estimate exceeding $1 billion. From these components, GAO selected a non-generalizable sample of nine major acquisition programs with a variety of characteristics to compare PARM oversight activities and review program data.
section_title: What GAO Found
paragraphs: The Department of Homeland Security (DHS) has taken steps to improve oversight of major acquisition programs, but it lacks written guidance for a consistent approach to day-to-day oversight. Federal Standards for Internal Control call for organizations to define and document key areas of responsibility in order to effectively plan, direct, and control operations to achieve agency objectives. DHS has defined the role of the Component Acquisition Executive, the senior acquisition official within each component, and established monthly meetings to discuss programs that require management attention. However, DHS has not defined all of the roles and responsibilities of the Office of Program Accountability and Risk Management (PARM)—the lead body responsible for overseeing the acquisition process and assessing the status of acquisition programs—and other headquarters organizations. GAO also found that officials' involvement and relationships with components varied significantly. DHS does not have a structure in place for overseeing the costs of 42 programs in sustainment (that is, programs that have been fielded and are operational) for which acquisition documentation requirements were waived in 2013. Sustainment costs can account for more than 80 percent of total costs, and all but one of these programs lack an approved cost estimate. GAO also previously reported that cost estimates are necessary to support decisions about program funding and resources.
The most recent data that PARM provided to DHS and congressional decision makers for oversight were not consistently accurate and up-to-date. Specifically, PARM's fiscal year 2014 Comprehensive Acquisition Status Report (CASR), which was based on fiscal year 2013 data, contained inaccurate information on DHS acquisition programs. To develop the CASR, PARM drew from DHS's official system for acquisition program reporting, the Next Generation Periodic Reporting System (nPRS); however, the system is hampered by data issues, including inconsistent participation by program officials responsible for entering the data. Further, DHS has not provided useful information for certain CASR reporting requirements. DHS interpreted one requirement in a way that eliminated the need to report cost, schedule, or performance changes for almost half of the programs in the CASR. Holding programs accountable for maintaining their data in nPRS and providing decision makers with more in-depth information would enhance future acquisition reports and render the CASR a more effective instrument for DHS and congressional oversight.
GAO Assessment of the DHS Comprehensive Acquisition Status Report Development Process
section_title: What GAO Recommends
paragraphs: GAO recommends that DHS take a number of actions including developing written guidance for a consistent approach to oversight, addressing programs in sustainment, and enhancing data quality and reports to Congress. DHS concurred with GAO's recommendations.

section_title: Letter
paragraphs: The Department of Homeland Security (DHS) invests extensively in  acquisition programs to help execute its many critical missions. DHS and  its underlying components are acquiring systems to help reduce the  probability of a terrorist attack, protect against infectious diseases,  mitigate natural hazards, secure air and land borders, and execute a wide  variety of other operations. In fiscal year 2014, DHS reported it planned to  spend approximately $10.7 billion on its major acquisition programs,  including acquisition, planning, maintenance, and investment support  costs. In 2011, DHS told Congress that it planned to invest a total of  $167 billion in its major acquisition programs. However, DHS officials  were unable to verify this number or provide an update, during the course  of this review, since the agency lacks key cost information for its  programs. We have highlighted DHS acquisition management issues on  our high-risk list since 2005 and made numerous recommendations to   In recent years, DHS has  improve acquisition management practices.taken steps to improve acquisition management by dedicating additional  resources to oversight and documenting major acquisition decisions in a  more transparent and consistent manner. However, many of our  recommendations have not yet been implemented, including that DHS  ensure all major acquisition programs fully comply with DHS acquisition  policy. Within DHS, the Office of Program Accountability and Risk  Management (PARM) is the lead body responsible for overseeing the  acquisition process and assessing the status of acquisition programs,  although other DHS offices also have oversight roles. Nearly all of DHS’s  program management offices are located within 13 department  organizations, including components such as the Transportation Security  Administration, U.S. Coast Guard, and U.S. Customs and Border  Protection.
You requested that we review DHS’s oversight of its major acquisition  programs. This review focuses on PARM’s day-to-day program oversight,  rather than oversight at key points in the acquisition life cycle as defined  in policy, such as acquisition decision events that can occur years apart.  We have previously reviewed the key points in the acquisition life cycle  and have found that DHS’s acquisition policy reflects many key program  management practices. Specifically, this report addresses (1) steps DHS  has taken to improve oversight and gaps that exist, if any, and (2)  whether the data PARM provides to DHS and congressional decision  makers to carry out their oversight responsibilities are accurate and up-to- date.
To conduct our work, we identified organizations within DHS, in addition  to PARM, that are responsible for oversight of major acquisitions and  determined their roles and responsibilities by analyzing DHS policies and  procedures, reviewing organizational charts, and interviewing policy,  budget, and acquisition oversight officials at the headquarters level. To  determine how PARM coordinates with the components to conduct  oversight, we selected nine DHS components with responsibility for at  least one Level 1 acquisition—a program with a reported life-cycle cost  estimate exceeding $1 billion—and interviewed their Component  Acquisition Executives (CAE) or designees. We reviewed component- specific policies and procedures and charters for program governance  groups, as well as other relevant documentation. We also selected a non- generalizable sample of one major acquisition program from each of the  nine components to collect examples, from program office officials, of  PARM’s oversight and coordination activities. We selected as our case  study programs those that included a variety of characteristics, such as  PARM identification as high visibility for oversight purposes, high and low  “risk scores” as reported by PARM, information technology (IT) and non- IT programs, and a range of life-cycle cost estimates. For the nine case  studies, we reviewed relevant documentation, such as acquisition  decision memorandums, and interviewed program officials. We selected  the following programs as case studies:
National Protection and Programs Directorate—Next Generation
Office of the Chief Information Officer (OCIO)—National Capital
Science and Technology Directorate—National Bio and Agro-Defense
Transportation Security Administration—Technology Infrastructure
U.S. Citizenship and Immigration Services—Verification
U.S. Coast Guard—Fast Response Cutter
U. S. Customs and Border Protection—Strategic Air and Marine
U.S. Immigration and Customs Enforcement—Electronic Health  In addition, we collected program data for each of the nine case study  programs from the information system that PARM uses to track program  performance, the Next Generation Periodic Reporting System (nPRS).  We assessed the data reliability of nPRS by comparing this data to the  information contained in the Comprehensive Acquisition Status Report, a  report to Congress, and noting discrepancies between the system data  and the issued data. Based on the discrepancies we found, we evaluated  the effectiveness of the acquisition report as a tool for DHS management  and congressional oversight. We determined that the nPRS data were not  sufficiently reliable for our purposes; however, we present the data for  illustrative purposes only. See appendix I for a more complete description  of our scope and methodology.
We conducted this performance audit from March 2014 to March 2015 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: DHS invests in major acquisition programs to develop capabilities  intended to improve its ability to execute its mission. DHS policy defines  acquisition programs as follows:
Level 1 major acquisition programs are expected to cost $1 billion or  more over their life cycles.
Level 2 major acquisition programs are expected to cost at least $300  million over their life cycles.
Special interest programs, without regard to the established dollar  thresholds, are designated as Level 1 or Level 2 programs. For  example, a program may be raised to a higher acquisition level if  its importance to DHS’s strategic and performance plans is  disproportionate to its size or it has high executive visibility.
Level 3 programs are those with a life-cycle cost estimate less than  $300 million and are considered non-major.
DHS’s Acquisition Management Directive 102-01 (MD 102) and DHS  Instruction Manual 102-01-001 (Guidebook), which includes 12  appendices, establish the framework for the department’s policies and  processes for managing these acquisition programs. MD 102 establishes  that DHS’s Chief Acquisition Officer—the Under Secretary for  Management (USM)—is responsible for the management and oversight  of the department’s acquisition policies and procedures. The Deputy  Secretary, USM, and CAE are the acquisition decision authorities for  DHS’s acquisition programs, depending on the level.
The acquisition decision authority is responsible for reviewing and  approving the movement of DHS’s major acquisition programs through  four phases of the acquisition life cycle at a series of five acquisition  decision events. These acquisition decision events, which can be more  than one year apart, provide the acquisition decision authority an  opportunity to assess whether a major program is ready to proceed  through the life-cycle phases. Following are the four phases of the  acquisition life cycle, as established in DHS acquisition policy:  1.  Need: Department officials identify that there is a need, consistent  with DHS’s strategic plan, justifying an investment in a new capability  and the establishment of an acquisition program to produce that  capability;  2.  Analyze/Select: A designated program manager reviews alternative  approaches to meeting the need and recommends a best option to  the acquisition decision authority;  3.  Obtain: The program manager develops, tests, and evaluates the  selected option. During this phase, programs may proceed through  acquisition decision event 2B, which focuses on the cost, schedule,  and performance parameters; and acquisition decision event 2C,  which focuses on low rate initial production; and  4.  Produce/Deploy/Support: DHS delivers the new capability to its  operators, and maintains the capability until it is retired. This phase  includes sustainment, which begins when a capability has been  fielded for operational use; sustainment involves the supportability of  fielded systems through disposal, including maintenance.
Figure 1 depicts the four phases of the acquisition life cycle and the  associated acquisition decision events.
An important aspect of these acquisition decision events is the review and  approval of key acquisition documents critical to establishing the need for  a major program, its operational requirements, an acquisition baseline,  and testing and support plans. Examples of key DHS acquisition  documents include: a life-cycle cost estimate, which provides an exhaustive and  structured accounting of all resources and associated cost  elements required to develop, produce, deploy, and sustain a  program; and an acquisition program baseline, which establishes a program’s  critical baseline cost, schedule, and performance parameters.
We are also conducting a separate review that assesses the extent to  which select DHS major acquisition programs are on track to meet their  cost estimates, schedules, and capability requirements.
PARM is designated by MD 102 as the lead body responsible for  overseeing the acquisition process of major acquisition programs. PARM  was established in October 2011 to develop and update program  management policies and practices, oversee the acquisition workforce,  and collect program performance data. PARM is led by an executive  director who reports directly to the USM. In addition to its role of  overseeing major acquisitions, PARM provides support and assistance to  CAEs and program managers at each of DHS’s 13 components during  the acquisition process. Within these components, CAEs are responsible  for establishing acquisition processes and overseeing the execution of  their respective portfolios. Also within the components, program  management offices are responsible for planning and executing DHS’s  individual programs within cost, schedule, and performance goals and  preparing required acquisition documents for acquisition decision events,  which help facilitate the governance process. Table 1 lists elements at  the headquarters, component, and program level that contribute to  oversight of DHS major acquisition programs.
The Fiscal Year 2012 DHS Appropriations Act required the USM to  submit a Comprehensive Acquisition Status Report (CASR) with the  President’s budget proposal for fiscal year 2013, and an associated  conference report contained the specific information to be included in the  CASR. The requirement for the CASR has been continued in  subsequent appropriations acts, and DHS is currently working on the next  iteration of the CASR, assuming DHS will again be required to produce  this report. The legislation required DHS to provide to Congressional  appropriations committees programmatic data and evaluative information,  such as a program’s current acquisition phase, life-cycle cost, and a  rating of cost, schedule, and technical risks. DHS is to include this  information for each major acquisition on the Master Acquisition Oversight  List (MAOL)—a list of DHS acquisitions that is broken down into  categories defining the differing oversight requirements across programs.  The legislation established the following CASR requirements for major  acquisition programs:  1.  A narrative description including current gaps and shortfalls, the  capabilities to be fielded, and the number of planned increments  and/or units;  2.  Acquisition Review Board status of each acquisition, including the  current acquisition phase, the date of the last review, and a listing of  the required documents that have been reviewed and/or approved;  3.  The most current approved acquisition program baseline, including  project schedules and events;  4.  A comparison of the original and current acquisition program baseline,  and the current estimate;  5.  Whether or not an independent verification and validation has been  implemented, with an explanation for the decision and a summary of  any findings;  6.  A rating of cost risk, schedule risk, and technical risk associated with  the program, including narrative descriptions and mitigation actions;  7.  Contract status, including earned value management data, as  8.  A life-cycle cost of the acquisition, and time basis for the estimate;  9.  A planned procurement schedule, including the best estimate of the  annual cost and increments/units to be procured annually;  10. A table delineated by appropriation that provides the actual or  estimated appropriations, obligations, unobligated authority, and  planned expenditures;  11. The reason for any significant changes from the previous CASR in  acquisition quantity, cost, or schedule;  12. Key events or milestones from the prior fiscal year; and  13. Key events or milestones for the current fiscal year.
section_title: DHS Has Taken Steps to Improve Oversight of Major Acquisition Programs, but Lacks Written Guidance and Cost Oversight Mechanism for Some Programs
paragraphs: Although DHS has taken steps to improve oversight of major acquisition  programs, such as clarifying the role of the CAEs, it lacks written  guidance for a consistent approach to oversight. Specifically, there is no  guidance to define the roles and responsibilities of PARM and other DHS  headquarters organizations in providing day-to-day support and oversight  to programs during the acquisition process. PARM started conducting  monthly high visibility meetings to discuss programs that require  immediate or additional management attention. PARM also maintains a  list of programs subject to oversight, the MAOL. The process for creating  this list has fluctuated over time; PARM recently made revisions to the  MAOL and plans to make further changes to it in the future. Finally, DHS  has not established a structure for overseeing the costs of 42 programs in  sustainment whose acquisition documentation requirements were waived  by the USM in 2013. Sustainment costs can account for more than 80  percent of total costs, and all but one of these programs lack an approved  cost estimate.
subsections: section_title: DHS Lacks Written Guidance for a Consistent Approach to Ongoing Oversight
paragraphs: While DHS has made progress in defining and documenting roles and  responsibilities in the oversight of major acquisitions, such as issuing  guidance describing the roles of CAEs, the roles and responsibilities of  PARM and other DHS headquarters organizations are not clear. Without  defined roles and responsibilities, DHS cannot ensure it is providing the  appropriate level of oversight or receiving the right information to conduct  oversight. PARM has made efforts to expand its oversight and support  roles through its component leads, PARM’s liaisons to the components;  however, roles and responsibilities for these positions are not defined. In  addition, there was no guidance to define the differences in the role of  PARM and OCIO-Enterprise Business Management Office (EBMO) in the  oversight of major IT acquisitions, and we found potential overlap in the  roles of these entities.
Figure 2 illustrates PARM’s interactions with DHS headquarters,  component, and program-level offices and officials with acquisition  oversight responsibility. Some of these interactions are set forth in policy  while others are not.
PARM provides ongoing oversight and support to programs in a number  of ways, such as consulting with program officials to prepare required  documents prior to an Acquisition Review Board and providing training to  components on various aspects of program management. One of  PARM’s key mechanisms for providing day-to-day oversight and support  to programs between acquisition decision events is through its staff of 10  component leads, but their roles and responsibilities are not defined in  DHS acquisition policy. According to PARM officials, the component leads  provide day-to-day oversight and support to acquisition programs for a  specific component and are intended to be a key source of  communication and coordination between PARM and the programs. In  turn, component leads provide program information to PARM’s executive  director, which may be used in high visibility meetings with the USM.
PARM component leads told us they interact directly with the CAEs and  program offices to ensure that programs are adhering to the acquisition  process, along with meeting acquisition milestones and reporting  requirements. While PARM’s component leads play an important role in  the coordination with components, their roles and responsibilities are not  defined in DHS acquisition policy. We found that their involvement and  relationships with components varies significantly. For example, PARM’s  component lead for U.S. Citizenship and Immigration Services is involved  in the day-to-day management of programs. This component lead  regularly attends component and program-level meetings and organizes  training workshops to educate component and program-level staff. In  another example, the PARM component lead for the National Protection  and Programs Directorate provided additional guidance and attention to  the directorate’s programs while the acting CAE was learning his role. In  contrast, a U.S. Coast Guard official told us that while there is informal,  almost daily communication, their component lead does not have direct  access to program level data and relies on the input of the CAE to  schedule and prioritize department-level acquisition milestone meetings.  Such differences in the PARM component leads’ involvement with  programs may be appropriate depending on the type of program or  experience of component and program office staff; however, without  defined roles and responsibilities, PARM cannot ensure it is providing the  appropriate level of oversight or receiving the right information to conduct  oversight.
GAO, Auditing and Financial Management: Standards for Internal Control in the Federal  Government, GAO/AIMD-00-21.3.1 (Washington, D.C.: Nov. 1, 1999). leads are not defined in DHS acquisition policy and it is a challenge that  they are trying to determine how to address.
Further, while PARM is the lead office responsible for overseeing all  major acquisition programs, we found confusion among component  officials related to PARM’s role in IT acquisitions, where OCIO-EBMO  also has oversight responsibility. Of the 72 Level 1 and Level 2  acquisition or service programs listed on the 2014 MAOL, 57 are  designated as IT programs. Per the DHS acquisition policy, the OCIO is  responsible for establishing IT policies and procedures and ensuring that  approved IT acquisitions comply with technical requirements and  departmental management processes, such as Agile development, which  calls for producing software in small, short increments. Within OCIO,  EBMO has been given primary responsibility for ensuring that the  department’s IT investments align with its missions and objectives.  However, while DHS acquisition policy outlines responsibilities for PARM  and OCIO, there is no guidance that defines how the role of PARM differs  from the role of EBMO in the oversight of IT acquisition programs.
GAO, Auditing and Financial Management: Framework for Assessing the Acquisition  Function At Federal Agencies, GAO-05-218G (Washington, D.C.: Sept.1, 2005). the oversight of major acquisitions could improve coordination, limit  overlap of responsibilities, and reduce duplicative efforts at the  component level.
subsections: section_title: DHS Issued Guidance to Clarify the Role of the CAE
paragraphs: In September 2014, the USM issued a policy memorandum clarifying the  responsibilities of the CAEs, who have an important role in acquisition  oversight. To strengthen acquisition oversight within the department, the  USM intends to standardize these officials’ acquisition authorities and  experience levels. The memo also sets forth oversight responsibilities of  the CAEs, particularly for the Level 3 programs for which they are the  acquisition decision authority. The memorandum additionally clarifies that  for the purposes of acquisition oversight, program managers report to  their CAE and the CAEs report to the USM. This clarification is useful, as  CAEs we spoke with prior to the issuance of the memorandum noted  differences in the roles and responsibilities of the CAEs across  components. For example, at U.S. Immigration and Customs  Enforcement, it was the component OCIO, rather than the CAE, who was  responsible for the execution of acquisitions, and program managers  reported directly to the component OCIO. At the U.S. Coast Guard, the  CAE is currently the Vice Commandant, who oversees all of the  component’s operations and mission support functions, including human  resources, budget, and acquisitions. Within mission support is the  Assistant Commandant for Acquisitions, who has more direct oversight of  the U.S. Coast Guard’s acquisition programs. Given the new  requirements for CAE experience levels, PARM’s executive director  anticipates that there may be changes in CAE assignments for at least  one component.
The memo directs PARM to create and provide executive-level  acquisition training to the CAEs. The memorandum further outlines the  CAEs’ responsibilities for complementing PARM’s oversight activities,  such as responding in a timely manner to requests for information. As of  March 2014, PARM began working with CAEs to hold monthly forums to  discuss topics such as the MAOL, staffing plans, and the CASR.
section_title: PARM Highlights Some Programs for Oversight through High Visibility Meetings, but DHS Lacks a Cost Oversight Mechanism for Programs In Sustainment
paragraphs: PARM established monthly high visibility meetings to discuss programs  that require immediate or additional management attention. In addition, to  identify programs for which PARM has oversight responsibility, PARM  maintains a list of DHS’s acquisition programs on the MAOL, which is  broken down into categories that describe each program’s reporting  characteristics. However, DHS has not established a structure for  overseeing the costs of 42 programs in sustainment whose acquisition  documentation requirements were waived by the USM in 2013.  Sustainment costs can account for more than 80 percent of total costs,  and all but one of these programs lack an approved cost estimate.
subsections: section_title: PARM High Visibility Meetings Highlight Programs for Management Attention
paragraphs: PARM’s executive director established high visibility meetings in  December 2013 to discuss any acquisition programs that require more  immediate attention from DHS management. PARM’s executive director  uses these meetings as a management tool. He identifies the programs to  be discussed in consultation with component leads. PARM’s executive  director told us that the purpose of these meetings is to make sure that  senior leadership—including the USM, Chief Financial Officer, Chief  Information Officer, Chief Readiness Support Officer, Chief Procurement  Officer, and General Counsel—have a common understanding of the  acquisition programs’ status and key issues. According to PARM officials,  the high visibility meetings have provided better focus through greater  senior level involvement and led to a reinvigoration of preparation for  Acquisition Review Boards. As of November 2014, 33 programs have  been discussed in the high visibility meetings.
PARM officials put programs on the meeting agenda based on a variety  of considerations: programs with an upcoming Acquisition Review Board  meeting, programs with concerns or issues, and programs that PARM is  monitoring closely. Officials told us that the last two categories may  include programs under GAO or Inspector General review, programs  involved in a bid protest, and programs that have experienced schedule  slips or a cost increase. For example, PARM officials told us about a  program that changed its acquisition strategy to incorporate information  technology, but did not involve the Chief Information Officer. PARM  included this program in a high visibility meeting to ensure that officials  were informed of the change in strategy and were involved as  appropriate. In another case, PARM officials told us that they used high  visibility meetings to raise early awareness about concerns with a  program, which resulted in multiple follow-on meetings among high level  headquarters and component officials. The USM directed the component  to pause the program and issued an acquisition decision memorandum  that described the path forward.
section_title: PARM is Taking Steps to Improve the MAOL
paragraphs: DHS acquisition policy provides the overall structure for acquisition  management that programs are required to follow. The policy requires  PARM to create a list of major acquisition programs, the MAOL, a  document approved by the USM. PARM uses the MAOL to identify  programs for which it has oversight responsibility and to determine which  programs they include in the CASR, an annual report to Congress. In  2014, PARM updated and expanded the MAOL by listing programs in six  categories that detail the characteristics of programs. Five categories  specifically address acquisition programs (see table 2). In addition, there  is one category for a non-acquisition activity that is required to submit an  Office of Management and Budget business case.
PARM officials stated that they updated the list to more clearly  incorporate input of all headquarters organizations, thereby making it a  more useful oversight tool. The list has evolved over time as more  headquarters organizations have added programs to the MAOL. PARM  officials have drafted updates to DHS acquisition policy that include a  section on requirements for the MAOL. Specifically, the planned updates  will include which headquarters organizations will be involved in the  development of the list, and establish a process for removing programs.  PARM officials also told us they recently began a process for updating the  list more regularly. The updates also provide additional information about  the development and use of the MAOL. The draft updates describe the  process for determining whether or not a program belongs on the MAOL,  which follows a decision tree. PARM officials said that the new process  for developing the MAOL more effectively coordinates and tracks the  input from other DHS headquarters organizations, like EBMO and the  Office of the Chief Financial Officer, as well as CAEs. The draft updates  also describe justifications for removal from the MAOL. For example, a  program might be removed if it is merged with another program, or if it is  no longer considered special interest—meaning that a program was  elevated to a higher acquisition level without regard to dollar threshold.  Officials were unsure when the draft updates would be approved by DHS  management. In addition to the draft policy updates, PARM officials told  us that they recently instituted a governing board of officials who will  determine changes to the MAOL on a quarterly basis, given its potential  to provide important information to department decision makers. PARM  officials told us that the next MAOL, expected in February 2015, will use  the new process described in draft guidance.
section_title: DHS Lacks Cost Oversight Mechanism for Programs in Sustainment
paragraphs: DHS does not have a structure in place for overseeing the costs of 42  programs whose acquisition documentation requirements were waived   This waiver  through a memorandum issued by the USM in May 2013.covered certain programs in sustainment, meaning that these acquisition  programs have been developed and delivered and they are being  operated and maintained through the disposal phase. Because these  programs were in sustainment when MD 102 was instituted in 2008, the  USM determined that it would be cost prohibitive and inefficient to  recreate documentation for previous phases. However, we found that only  one of the 42 waived programs has an approved life-cycle cost estimate,  which would include acquisition costs as well as the costs to operate and  maintain the system once it is in sustainment. PARM officials could not  provide us estimates of the value of the sustainment programs. PARM’s  executive director stated that these programs should produce operations  and maintenance cost estimates. These estimates would account for the  remainder of their life cycles through disposal, but the programs are not  currently required to do so, given the 2013 waiver.
Further, in the 2014 MAOL, PARM included seven additional programs in  sustainment and also noted that documentation was waived for these  programs. The 42 programs in sustainment from the USM’s  memorandum and the seven programs listed on the MAOL are listed in  appendix II. The Office of Management and Budget stated in 2014 that  the sustainment phase can account for more than 80 percent of program  life-cycle costs, which demonstrates the need for oversight of these  programs’ costs. We have previously reported that cost estimates are  necessary to support decisions about program funding, develop annual  budget requests, and evaluate resource requirements. Furthermore, the  management of a cost estimate involves continually updating the estimate  with actual data as they become available, revising the estimate to reflect  changes, and analyzing differences between the estimated and actual  costs. Without knowing the operations and maintenance cost estimates  for these programs, DHS will not be able to fully plan for and manage  funding requirements across its major acquisition programs.
The 2013 waiver did not define which DHS office is responsible for  oversight of the sustainment programs. CAEs are responsible for Level 3  programs and PARM officials stated that this also applies to programs in  sustainment. A PARM official further told us it is difficult to know who is  responsible for oversight of Level 1 and Level 2 programs in sustainment.  PARM officials expressed concerns about the lack of oversight of these  programs. Specifically, officials noted DHS may decide, on a case-by- case basis, which organization should most appropriately provide  oversight to programs in sustainment, which may include more than one  office.
section_title: Program Data PARM Provided to DHS and Congressional Decision Makers Were Not Consistently Accurate and Up-to-Date
paragraphs: PARM’s fiscal year 2014 CASR, a report mandated by Congress,  provided the status of 82 DHS major acquisition programs but contained  data that were inaccurate and out-of-date. PARM primarily drew  information for the CASR from nPRS, DHS’s official system of record for  acquisition program reporting. However, data issues—including  inconsistent participation among the programs responsible for entering  data—have led to inaccurate information in nPRS. For example, our  analysis found discrepancies between the CASR and nPRS for life-cycle  cost estimate data even after efforts to update or fix the data inaccuracies  through an extensive adjudication process. Therefore, it was unclear  whether congressional CASR recipients received accurate program  information. PARM officials have acknowledged ongoing issues with the  data reported in both nPRS and the CASR, and noted that they are  working to improve the data quality. Officials stated that information in the  CASR did not provide a complete picture of program life-cycle costs,  which was the result of both incorrect data that programs had reported  and limitations in using the nPRS system. Further, component and  program officials have also stated that the CASR did not accurately reflect  program risks. Finally, DHS provided insufficient information to address  certain CASR reporting requirements. For example, the CASR did not  include annual planned procurement schedules containing estimates of  the units and/or increments for each program, although it was required to  do so.
subsections: section_title: Data in DHS’s Acquisition Program Reporting System Were Not Consistently Accurate
paragraphs: For the nine programs in our review, we found that program offices did  not consistently enter and verify their data in nPRS. DHS established  nPRS as the system of record for acquisition program reporting in 2008,  and in 2012 the USM issued a memorandum to CAEs stating that  programs should make every effort to ensure that their data in nPRS is  complete, accurate, and valid on a monthly basis. According to the  memorandum, nPRS was intended to be a key tool for acquisition  program management, and help provide the capability to efficiently  assess the department’s acquisition portfolio. DHS components have the  responsibility to ensure that their respective programs enter the data in  nPRS as required, and CAEs are required to ensure that the data is  validated and submitted in timely manner. However, we found that this  was not done consistently for the nine major acquisition programs in our  review, which are overseen by nine different DHS components.
We examined nPRS data for the nine programs at two key points:  September 2013, the closing date for data for fiscal year 2013, and March  2014, the date when PARM issued its fiscal year 2014 CASR, which was  based on fiscal year 2013 program data. We found a number of problems  with the data. For example, as of September 2013, three programs we  reviewed did not enter expenditure data, the amount the programs  actually spent, in nPRS for fiscal year 2013 as required, and two of these  programs did not enter historical expenditure data at all.
When we compared programs’ entries of expenditure data over time, we  found additional discrepancies. As an example, the U.S. Coast Guard’s  Fast Response Cutter program’s expenditure entries in nPRS increased  by more than $340 million from September 2013 to March 2014, even  though both of these entries were supposed to reflect fiscal year 2013  expenditures. A U.S. Coast Guard official stated that this increase was  due to a correction in the program’s reported expenditures, to account for  all funds spent in fiscal year 2013 regardless of when those funds were  received. The official noted that the program’s entry from September  2013 reflected only funds received in fiscal year 2013. However, the  reason for this change was not documented in nPRS. In another  example, the U.S. Citizenship and Immigration Service’s Verification  Modernization program’s entries for total historical expenditures through  fiscal year 2012 decreased by almost $240 million when comparing these  data from September 2013 and March 2014. Figure 3 shows the  differences in reported expenditure data in nPRS for the Fast Response  Cutter and Verification Modernization programs.
Large nPRS discrepancies such as these call into question the reliability  of the underlying data and whether DHS management has the information  it needs to provide oversight of major acquisition programs. PARM  officials have acknowledged ongoing issues with the data reported in  nPRS and noted that they are working to improve its quality. For example,  PARM provides a working group to the components to express their  views on the nPRS system and its processes. However, PARM officials  stated they do not have a mechanism to hold the programs accountable  for updating their data. Component and program officials told us that they  do not use nPRS for program management—even though that was one  intended purpose of the system—because the system is difficult to use  and does not meet their needs. For example, the OCIO National Capital  Region Infrastructure Operations program manager stated that his  program does not work with nPRS. For both September 2013 and March  2014, nPRS data fields for this program that were to be used to populate  the fiscal year 2014 CASR, such as the program description and last  acquisition review board date, were blank. Component and program  officials also stated that they use other internal tools, such as  spreadsheets, presentations, or project software for program  management purposes and to maintain current information. PARM has  not undertaken an effort to ascertain the root causes of why program  managers are not populating nPRS as required. As we have previously  reported, to be useful, performance information must meet users’ needs  for completeness, accuracy, consistency, timeliness, validity, and ease of  use. Unless DHS program managers consider nPRS to be a useful tool  for their own program management purposes, as intended, the problems  we found with inaccurate data are likely to persist.
section_title: Extensive Adjudication Process Did Not Correct Inaccuracies for Reported Life-Cycle Cost Estimates
paragraphs: To further understand the reasons for program data inaccuracies in the  fiscal year 2014 CASR, we analyzed the steps that PARM undertakes as  part of an extensive adjudication process with the components regarding  the underlying nPRS data. We found that PARM’s adjudication process  did not rectify key inaccuracies in the fiscal year 2014 CASR, specifically  regarding programs’ life-cycle cost estimates. As a result, Congress may  not have received accurate program information. Prior to its release,  PARM conducts an extenstive adjudication process for the CASR  information, including reviews with program and various DHS  headquarters offices, in order to identify and address potential data  inconsistencies between the sources and draft report. This process began  in October after the close of the fiscal year and ended when the report  was published in March. We reviewed nPRS data from March 2014, the  date when PARM issued its fiscal year 2014 CASR (which is comprised  of fiscal year 2013 program data) to compare these data to what was  presented to Congress in the CASR. Figure 4 shows select elements of  the CASR development and adjudication process, along with our  assessment of those elements.
The fiscal year 2014 CASR reported on a total of 82 major acquisition  programs. For four of the nine major acquisition programs in our review,  we found discrepancies between the CASR and nPRS for life-cycle cost  estimate data after the adjudication process, which should have  reconciled such inconsistencies. As an example of these discrepancies,  both across nPRS and between nPRS and the CASR, life-cycle cost  estimates for two programs differed even though the estimates were  associated with the same source and date. Another program, the  Electronic Health Record System, had three different life-cycle values  ranging from approximately $60 million to $80 million, a difference of over  35 percent, with two of those values presented in the CASR. Our analysis  of the data inconsistencies indicated that it was unclear whether  congressional CASR recipients received accurate program cost  information because there was no way to confirm which estimate was  correct or what the different estimates represented. PARM officials stated  that they are changing their process for the development of the next  CASR, which they expect to issue with the President’s fiscal year 2016  budget submission, in an effort to more effectively match their reported  data to nPRS. Table 3 highlights discrepancies between the CASR and  nPRS for life-cycle cost estimate data for the four programs that we  reviewed.
Further, PARM may have incorrectly included or excluded certain  programs in the CASR based on DHS’s incomplete information on  program life-cycle costs. As we reported in 2014, unreliable cost  estimates have been an enduring challenge for DHS. PARM included  Level 1 and 2 programs from the MAOL in the CASR, as required, using  program life-cycle costs to determine program status. However, PARM  officials acknowledged that some programs’ acquisition category levels  were incorrect in the CASR, and due to the lack of DHS approved life- cycle cost estimates, they would not know the scope of this issue. For  example, the CASR included program life-cycle cost estimate figures, but  did not indicate who approved these estimates (i.e., if they were approved  at the component or department level), or if anyone approved the  estimates at all. Because these cost figures may not accurately reflect the  actual life-cycle costs, programs may have been inappropriately included  or excluded from the CASR and ultimately not receive the appropriate  level of congressional oversight.
We also found areas where additional explanation in the CASR would  have been helpful. For example, PARM listed the Electronic Health  Record System program in the CASR as a Level 2 program, while its  reported life-cycle cost estimate of approximately $70 million would  designate it a Level 3 program. PARM is not required to include Level 3  programs in the CASR. Electronic Health Record System officials  explained that the program was listed as Level 2 because DHS  management designated it as a “special interest” program in the MAOL,  which did make it eligible for inclusion in the CASR, but PARM did not  include this rationale in the CASR.
In addition, program officials stated that the inflexibility of nPRS may  prevent the department from providing accurate program information in  the CASR. For example, officials from the National Protection and  Programs Directorate’s Next Generation Network-Priority Service noted  difficulties in accurately reporting data on their program’s increments in  nPRS. The officials stated that the program has multiple increments, each  with its own set of acquisition decision events. However, the program  reported one overall life-cycle cost estimate in nPRS, even though it has  estimates for each increment, because the system does not allow for the  inclusion of multiple estimates. Officials ultimately provided explanatory  comments in nPRS that noted the increment 1 estimate included only  acquisition costs while not specifying who approved the estimate, and  that PARM approved the increment 2 estimate in July 2013. Because  these incremental estimates were at different stages in their development,  combining them into a single estimate in nPRS, which PARM ultimately  reported in the CASR, did not provide Congress with an accurate picture  of the program’s costs.
section_title: DHS Did Not Provide Most Useful Information on Program Risks and Other Requirements for the CASR
paragraphs: Of the 13 CASR reporting requirements, DHS provided insufficient  information for in-depth oversight for four of them, and in one case, DHS  did not comply with a reporting requirement. The first reporting  requirement was a rating of cost risk, schedule risk, and technical risk  associated with the program. PARM fulfilled this requirement by reporting  programs’ top-five cost, schedule, and technical risks, instead of separate  ratings for each. However, this reporting was inconsistent. Programs  submitted these risks through DHS’s Investment Management System,  but this system can contain more than five risks. As a result, program  officials stated that they did not know how PARM selected their top-five  risks for inclusion in the CASR. For example, the Strategic Air and Marine  Program listed nine risks in the system, but PARM only reported two risks  in the CASR. Further, the risk reporting in this section varied across the  programs in our review. For example:
The Verification Modernization program had five risks, all of which  were technical.
The Technology Infrastructure Modernization program used four  of its own risk categories, which do not track to the required cost,  schedule, and technical risks: reliability of systems; dependencies  and interoperability between this investment and others; security;  and business.
The inconsistent risk reporting in this section prevents Congress from  making cost, schedule, and performance comparisons across DHS  programs.
In addition, as part of a separate requirement for independent verification  and validation, PARM evaluated the overall risk of each program,  assigned each a numerical risk score, and published these scores in the  CASR. PARM officials computed these risk scores using a set of  weighted criteria. However, component and program officials told us that  they did not know how PARM evaluated the risk scores for their  respective programs. PARM, component, and program officials have  acknowledged that the CASR did not accurately reflect the cost,  schedule, and performance risks associated with the programs. PARM  officials were unable to provide us with the supporting data used to  generate the scores because they did not store this information in nPRS.  As a result, we were unable assess how PARM computed these scores  or determine the extent to which PARM’s evaluation accurately reflected  program risks.
Due to PARM’s inability to provide us with supporting data, we reviewed  the Science and Technology Directorate’s National Bio and Agro-Defense  Facility program’s risk assessment in the CASR and found it lacked  details that could be useful to Congress and DHS management. PARM  gave the program a high risk score, in part, due to the lack of DHS  approval for the program’s acquisition documents. A program official  stated, however, that the documents could not go forward for DHS  approval due to the program’s lack of funding. The CASR did not contain  any clarifying explanation for the documents not being approved. Further  complicating the issue, the program’s nPRS data in September 2013  conflicts with its CASR entry, and showed program documents approved  by DHS as early as 2009. Discrepancies such as this call into question  the value of the information DHS is providing to Congress in the CASR.  To gain more visibility into the reasons for these inconsistencies, we  reviewed the source documents for the National Bio and Agro-Defense  Facility program, provided to us by PARM, and found that the approval  dates for five out of six documents do not match what was listed in either  nPRS or the CASR. Table 4 compares the National Bio and Agro- Defense Facility program’s reported document status according to nPRS  as of September 2013 and the CASR issued in March 2014, and the  source documents provided by PARM.
In another example, in PARM’s risk assessment of the Federal  Emergency Management Agency’s Risk Mapping, Assessment and  Planning program, the CASR stated that the program was covered by the  USM’s waiver of acquisition documents requirements and thus did not  include certain program data. However, an examination of nPRS showed  that, according to the system, the program had key documents, including  a mission needs statement and an acquisition program baseline approved  by DHS, which PARM did not list in the CASR and which could have  provided further information to decision makers.
A second CASR reporting requirement was a program’s planned  procurement schedule, including an estimate of the quantity to be  procured annually until completion. PARM did not comply with this  requirement. Instead, PARM provided the top-five contracts by dollar  value for each program, but these entries did not include procurement  quantity information for the programs. Some programs did report total  procurement quantities, but did not link these units to a schedule. PARM  officials noted that certain programs are not well-suited for reporting  procurement quantities, such as IT programs. However, in such cases,  the CASR should explain why no procurement quantities were listed.
A third requirement was the reason for any significant changes in a  program’s acquisition cost, quantity, or schedule from the prior annual  CASR. PARM officials interpreted these changes to only be those that  resulted in the submission of a new acquisition program baseline.  According to DHS acquisition policy, programs need to submit new  baselines when they breach defined cost, schedule, and performance  parameters defined in their original baseline. However, programs can  experience cost, quantity, or schedule changes that do not require a new  baseline. For example, the National Bio and Agro-Defense Facility  program experienced a delay in its construction schedule. While this  program’s baseline remains in place, the construction delay could impact  on-time delivery of the facility and is an example of a significant change  that could be reported in this section. In addition, the CASR included 38  programs without an approved acquisition program baseline; according to  PARM’s guidance, the CASR would not include any cost, schedule, or  performance changes for these programs. By defining programs’  significant changes as those that resulted in new baselines, PARM  eliminated the need to report on any cost, schedule, or performance  changes for almost half of the programs in the CASR, thereby limiting the  information available to Congress.
Finally, we found that PARM did not include certain key program events  in the CASR, such as acquisition decision events or full operating  capability schedules. Such data, in addition to the acquisition program  baseline approval dates that PARM currently reports, would have  provided Congress with more robust information about the program  status. Table 5 lists these four CASR requirements and our assessment  of the information reported.
section_title: Conclusions
paragraphs: Effective, on-going oversight of DHS’s broad portfolio of programs is  essential to ensure that programs are accountable for their performance  and that Congress and DHS decision makers receive useful, accurate,  and up-to-date information. DHS has improved aspects of its acquisition  management in recent years, including dedicating additional resources to  acquisition oversight and clarifying the roles of CAEs.
DHS could further enhance its oversight efforts by providing written roles  and responsibilities to oversight officials within PARM and among  headquarters organizations. Furthermore, a consistent, defined approach  to oversight could limit overlap of responsibilities and give DHS more  insight into whether its acquisition programs are executing according to  cost, schedule, and performance goals.
Likewise, as DHS acquisition programs move into the sustainment phase,  their costs continue to require monitoring. The USM’s waiving of  documentation requirements for the 42 programs in sustainment in 2013  resulted in a lack of oversight of costs for these programs. As of yet, no  DHS office has been designated to take over monitoring those programs’  operations and maintenance costs. Without an identified oversight body,  DHS lacks insight into those programs’ performance and the execution of  their funding, which could potentially be billions of dollars. This is  particularly of concern given that only one program had an approved cost  estimate at the time of the waiver.
Finally, DHS has not effectively communicated program status to  Congress through the CASR because it has provided out-of-date and  inaccurate information. Programs do not consistently report their own  data in nPRS, and components are not validating the information.  Although PARM’s adjudication process may address some data issues,  data are not corrected in the source systems before being published in  the CASR. Further, while PARM has some flexibility in the implementation  of the CASR reporting requirements, in one case the requirement was not  met. In other cases, such as PARM’s assessment of program risks, there  are opportunities for more transparency and clarity in the information  being transmitted to Congress. Holding programs accountable for  maintaining their cost, schedule, and performance data, and presenting  contextual information would help make the CASR a more effective  instrument for DHS and congressional oversight.
section_title: Recommendations for Executive Action
paragraphs: In order to help ensure consistent, effective oversight of DHS’s acquisition  programs, we recommend the Secretary of DHS take the following five  actions:
Direct PARM to develop written guidance that defines roles and  responsibilities of its component leads.
Direct the USM to:
Develop written guidance to clarify roles and responsibilities of  PARM and OCIO-EBMO for conducting oversight of major  acquisition programs.
Produce operations and maintenance cost estimates for programs  in sustainment and establish responsibility for tracking  sustainment programs’ adherence to those estimates.
Determine mechanisms to hold programs accountable for entering  data in nPRS consistently and accurately and to hold CAEs  accountable for validating the information. Also, evaluate the root  causes of why programs are not using nPRS as intended.
To make the CASR more useful, starting with the report reflecting  fiscal year 2015 program data, adjust the CASR to do the  following:
Report an individual rating for each program’s cost, schedule,
Report a best estimate of procurement quantities or indicate  why this is not applicable, as appropriate;
Report all programs’ significant changes in acquisition cost,  quantity, or schedule from the previous CASR report by  determining a means to account for programs that lack  acquisition program baselines;
Report major program events that are included in acquisition  program baselines, such as scheduled acquisition decision  events; and
Report the level at which the program’s life-cycle cost  estimate was approved.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this product to DHS for comment. In its written  comments, reproduced in appendix III, DHS concurred with all five of our  recommendations and provided plans of action and estimated completion  dates for four of them. Regarding the remaining recommendation, that the  Secretary of DHS direct PARM to develop written guidance that defines  roles and responsibilities for component leads, DHS provided evidence  that is has complied with the recommendation, and we agree.  Specifically, DHS provided a Component Lead Handbook, signed on  February 13, 2015, while our report was out for comment, that provides  oversight roles and responsibilities and other guidance to PARM  component leads in their job to oversee component programs.
DHS also provided technical comments that we incorporated into the  report as appropriate.
As agreed with your offices, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 30 days from the  report date. At that time, we will send copies to the Secretary of DHS. In  addition, the report will be available at no charge on GAO’s website at  http://www.gao.gov.
If you or your staff have questions about this report, please contact me at  (202) 512-4841 or mackinm@gao.gov. Contact points for our Offices of  Congressional Relations and Public Affairs may be found on the last page  of this report. GAO staff who made major contributions to this report are  listed in appendix IV.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: The objective of this review was to assess the Department of Homeland  Security’s (DHS) oversight of its major acquisition programs. Specifically,  this review focused on DHS’s Office of Program Accountability and Risk  Management (PARM) and its day-to-day program oversight, rather than  the oversight it conducts at key points in the acquisition life cycle as  defined in policy. We assessed (1) steps DHS has taken to improve  oversight and what gaps, if any, exist and (2) whether the data PARM  provides to DHS and congressional decision makers to carry out their  oversight responsibilities on program cost, schedule, and performance  are accurate and up-to-date.
To answer these questions, we identified organizations within DHS, in  addition to PARM, that are responsible for oversight of major acquisitions  and determined their roles and responsibilities by analyzing DHS policies  and procedures, reviewing organizational charts, and interviewing policy,  budget, and acquisition oversight officials at the headquarters level.  Specifically, we reviewed DHS Acquisition Management Directive 102-01  (MD 102) and its associated guidebook—DHS Instruction Manual 102-01- 001—and the guidebook’s 12 appendices. We reviewed draft updates to  DHS acquisition policy, as well as draft updates to departmental  instructions, such as Agile Development and Delivery for Information  Technology and the Systems Engineering Life Cycle Guidebook. We also  reviewed DHS acquisition memorandums, including the Secretary’s April  2014 Unity of Effort memorandum; the Office of the Chief Procurement  Officer’s Strategic Plan; information technology (IT) policies and  guidance, such as DHS Directive 102-04 on IT portfolio management and  the Office of the Chief Information Officer (OCIO) Portfolio Governance  Concept of Operations. At the department level, we interviewed officials  from PARM, OCIO–Enterprise Business Management Office (EBMO),  Office of the Chief Procurement Officer, Office of Policy, and Office of the  Chief Financial Officer–Office of Program Analysis and Evaluation and  Cost Analysis Division. In addition, we reviewed relevant GAO and DHS  Inspector General reports to provide context for all of our objectives.
To address our first objective, we selected nine DHS components with  responsibility for at least one Level 1 acquisition—a program with a  reported life-cycle cost estimate exceeding $1 billion—and interviewed  their Component Acquisition Executives (CAE) or designees. We  reviewed component-specific policies and procedures and charters for  program governance groups, as well as other relevant documentation.  The 2014 Master Acquisition Oversight List (MAOL) identifies Level 1  acquisition programs for the following nine components:
Federal Emergency Management Agency
National Protection and Programs Directorate
OCIO
Science and Technology Directorate
Transportation Security Administration
U.S. Citizenship and Immigration Services
U.S. Coast Guard
U. S. Customs and Border Protection
U.S. Immigration and Customs Enforcement  To collect examples of PARM’s oversight and coordination activities at  the program level, we selected a non-generalizable sample of major  acquisition programs from each of the nine components. Table 6 lists the  nine programs we selected as case studies.
We selected programs that were included in the fiscal year 2014  Comprehensive Acquisition Status Report (CASR), which PARM  submitted to the House and Senate Appropriations Committees to provide  information on DHS major acquisition programs. To the extent possible,  we chose programs that have been identified as having “concerns/issues”  or as being “monitored closely” in PARM’s high visibility meetings, which  include DHS senior leadership. In order to assess acquisition oversight  across the spectrum of DHS programs, we selected case study programs  with a variety of characteristics. We chose a mix of Level 1 and Level 2  programs, as defined in the fiscal year 2014 CASR, and included both IT  and non-IT programs. One of our nine case study programs, the  Electronic Health Record System, was classified as a Level 2 program in  the fiscal year 2014 CASR, but listed as a Level 3 program on the 2014  MAOL. We chose this program in order to examine the reasons for the  change and to determine the extent to which oversight varies for major  and non-major acquisition programs. Another factor used to select the  case studies was program risk, as measured by CASR risk scores. Risk  scores are included in the CASR’s independent verification and validation  section and are derived from PARM’s rating of program risk using a  standard set of criteria. We chose programs to include a mix of both high  and low CASR risk scores. For these case studies, we reviewed relevant  program documentation, such as acquisition decision memorandums, and  interviewed program officials.
For the second objective, we collected and reviewed data from DHS’s  official system of record for its acquisition programs, the Next Generation  Periodic Reporting System (nPRS), and compared that data to the fiscal  year 2014 CASR. All major programs on DHS’s MAOL are required to  report in nPRS. PARM then uses the program data in nPRS to help  generate its CASR.
In order to assess the data reliability of nPRS, we reviewed select  acquisition program data from nPRS and compared this data to the  information contained in the CASR. Specifically, we used nPRS reports  for each of the nine case study programs from the end of fiscal year 2013,  when PARM pulled the program data from the system to begin generating  the fiscal year 2014 CASR. We then compared the data from those  reports to the issued CASR, as well as to the nPRS program reports from  March 2014—the date that PARM released the CASR. The comparison of  those three sets of information allowed us to note discrepancies, including  missing data or outliers, between the system data and the issued data, as  well as if corrections were made to the system data following the release  of the report.
We assessed various data elements from the nPRS program reports that  are used to generate the information contained in the CASR. We  reviewed data across a range of tabs contained in the nPRS program  reports such as general information, Acquisition Review Board history,  program status, budget and funding, acquisition program baseline  milestones, risk, and key documents. For each of the nine case study  programs, we assessed reports from the end of fiscal year 2013, as well  as March 2014, for a total of eighteen program reports. In addition, we  reviewed documents, such as the nPRS user manual and policies related  to data entry, and interviewed agency officials responsible for inputting  and reviewing the nPRS data.
We determined that the nPRS data were not sufficiently reliable for our  purposes; however, we present the data for illustrative purposes only. For  example, for certain programs in our review, current and historical  expenditure data was missing. Another example from our analysis  showed differences in the life-cycle cost estimates for certain programs in  nPRS compared to those reported in the CASR. While the Under  Secretary for Management (USM) issued a memorandum that programs  are to update their nPRS data monthly, PARM officials recognized that  this does not happen consistently, and they acknowledged that there are  data accuracy issues with nPRS. In addition, when officials made updates  or changes to program-reported information due to the pending release of  the CASR, the programs were responsible for entering these updates or  changes into nPRS. Our analysis confirmed that certain programs in our  review did not update nPRS after going through the CASR reporting  process.
In order to evaluate the effectiveness of nPRS and the CASR as tools for  DHS management and congressional oversight, we first reviewed the  Department of Homeland Security Appropriations Act, 2014, which  established the provision for the USM to submit the CASR with the  President’s budget proposal for fiscal year 2015. In addition, we reviewed  Conference Report 112-331 for the Consolidated Appropriations Act,  2012, which contained the information requirements for inclusion in the  CASR. We then compared the data from nPRS for the nine case study  programs to that presented in the fiscal year 2014 CASR to determine  discrepancies between the two for certain data elements. We further  compared the CASR information to PARM policies and procedures, such  as the MAOL and MD 102. In order to review the CASR’s independent  verification and validation requirement, we asked PARM officials for  supporting documentation for how they generated their independent  verification and validation evaluations, but they were unable to provide  the documentation because they did not store it in nPRS. Finally, we  assessed the information the CASR either did or did not provide  compared to its congressional reporting requirements. We conducted this  assessment based on the CASR’s congressional reporting requirements,  DHS policies, and acquisition practices.
We conducted this performance audit from March 2014 to March 2015 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
section_title: Appendix II: Department of Homeland Security Acquisition Programs in Sustainment Exempt From Documentation Requirements
paragraphs: Table 7 below identifies 42 acquisition programs for which the  Department of Homeland Security’s (DHS) Under Secretary for  Management waived documentation requirements in a May 2013  memorandum. These programs were already in sustainment prior to  2008, meaning that they were in the last phase of their acquisition life  cycle, when DHS issued MD 102. Programs in sustainment have been  developed and delivered to their respective components for operation and  maintenance through disposal. The memorandum stated that it would be  cost prohibitive and inefficient for these programs to recreate the  documentation called for under the directive for their previous acquisition  life-cycle phases.
Table 8 below identifies seven additional major acquisition programs in  sustainment for which DHS waived documentation requirements on the  2014 Master Acquisition Oversight List. The Master Acquisition Oversight  List stated that having programs provide documents for their previous  acquisition life-cycle phases would be costly and provide no positive  performance impact for systems already delivered.
section_title: Appendix III: Comments from the Department of Homeland Security
section_title: Appendix IV: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Katherine Trimble, Assistant  Director; Leigh Ann Haydon, Analyst-in-Charge; Stephen V. Marchesani;  Alexis Olson; Sarah Marie Martin; and Daniel Hilger made key  contributions to this report. Peter W. Anderson, Jean L. McSween, Ozzy  Trevino, and Alyssa Weir also provided assistance.