Homeland Security Acquisitions: DHS Should Better Define Oversight Roles and Improve Program Reporting to Congress
Released: 2015-03-16T00:00:00

Letter

Background

DHS invests in major acquisition programs to develop capabilities intended to improve its ability to execute its mission. DHS policy defines acquisition programs as follows:

Level 1 major acquisition programs are expected to cost $1 billion or more over their life cycles.

Level 2 major acquisition programs are expected to cost at least $300 million over their life cycles.

Special interest programs, without regard to the established dollar thresholds, are designated as Level 1 or Level 2 programs. For example, a program may be raised to a higher acquisition level if its importance to DHS’s strategic and performance plans is disproportionate to its size or it has high executive visibility.

Level 3 programs are those with a life-cycle cost estimate less than $300 million and are considered non-major.

DHS’s Acquisition Management Directive 102-01 (MD 102) and DHS Instruction Manual 102-01-001 (Guidebook), which includes 12 appendices, establish the framework for the department’s policies and processes for managing these acquisition programs. MD 102 establishes that DHS’s Chief Acquisition Officer—the Under Secretary for Management (USM)—is responsible for the management and oversight of the department’s acquisition policies and procedures. The Deputy Secretary, USM, and CAE are the acquisition decision authorities for DHS’s acquisition programs, depending on the level.

The acquisition decision authority is responsible for reviewing and approving the movement of DHS’s major acquisition programs through four phases of the acquisition life cycle at a series of five acquisition decision events. These acquisition decision events, which can be more than one year apart, provide the acquisition decision authority an opportunity to assess whether a major program is ready to proceed through the life-cycle phases. Following are the four phases of the acquisition life cycle, as established in DHS acquisition policy: 1. Need: Department officials identify that there is a need, consistent with DHS’s strategic plan, justifying an investment in a new capability and the establishment of an acquisition program to produce that capability; 2. Analyze/Select: A designated program manager reviews alternative approaches to meeting the need and recommends a best option to the acquisition decision authority; 3. Obtain: The program manager develops, tests, and evaluates the selected option. During this phase, programs may proceed through acquisition decision event 2B, which focuses on the cost, schedule, and performance parameters; and acquisition decision event 2C, which focuses on low rate initial production; and 4. Produce/Deploy/Support: DHS delivers the new capability to its operators, and maintains the capability until it is retired. This phase includes sustainment, which begins when a capability has been fielded for operational use; sustainment involves the supportability of fielded systems through disposal, including maintenance.

Figure 1 depicts the four phases of the acquisition life cycle and the associated acquisition decision events.

An important aspect of these acquisition decision events is the review and approval of key acquisition documents critical to establishing the need for a major program, its operational requirements, an acquisition baseline, and testing and support plans. Examples of key DHS acquisition documents include: a life-cycle cost estimate, which provides an exhaustive and structured accounting of all resources and associated cost elements required to develop, produce, deploy, and sustain a program; and an acquisition program baseline, which establishes a program’s critical baseline cost, schedule, and performance parameters.

We are also conducting a separate review that assesses the extent to which select DHS major acquisition programs are on track to meet their cost estimates, schedules, and capability requirements.

PARM is designated by MD 102 as the lead body responsible for overseeing the acquisition process of major acquisition programs. PARM was established in October 2011 to develop and update program management policies and practices, oversee the acquisition workforce, and collect program performance data. PARM is led by an executive director who reports directly to the USM. In addition to its role of overseeing major acquisitions, PARM provides support and assistance to CAEs and program managers at each of DHS’s 13 components during the acquisition process. Within these components, CAEs are responsible for establishing acquisition processes and overseeing the execution of their respective portfolios. Also within the components, program management offices are responsible for planning and executing DHS’s individual programs within cost, schedule, and performance goals and preparing required acquisition documents for acquisition decision events, which help facilitate the governance process. Table 1 lists elements at the headquarters, component, and program level that contribute to oversight of DHS major acquisition programs.

The Fiscal Year 2012 DHS Appropriations Act required the USM to submit a Comprehensive Acquisition Status Report (CASR) with the President’s budget proposal for fiscal year 2013, and an associated conference report contained the specific information to be included in the CASR. The requirement for the CASR has been continued in subsequent appropriations acts, and DHS is currently working on the next iteration of the CASR, assuming DHS will again be required to produce this report. The legislation required DHS to provide to Congressional appropriations committees programmatic data and evaluative information, such as a program’s current acquisition phase, life-cycle cost, and a rating of cost, schedule, and technical risks. DHS is to include this information for each major acquisition on the Master Acquisition Oversight List (MAOL)—a list of DHS acquisitions that is broken down into categories defining the differing oversight requirements across programs. The legislation established the following CASR requirements for major acquisition programs: 1. A narrative description including current gaps and shortfalls, the capabilities to be fielded, and the number of planned increments and/or units; 2. Acquisition Review Board status of each acquisition, including the current acquisition phase, the date of the last review, and a listing of the required documents that have been reviewed and/or approved; 3. The most current approved acquisition program baseline, including project schedules and events; 4. A comparison of the original and current acquisition program baseline, and the current estimate; 5. Whether or not an independent verification and validation has been implemented, with an explanation for the decision and a summary of any findings; 6. A rating of cost risk, schedule risk, and technical risk associated with the program, including narrative descriptions and mitigation actions; 7. Contract status, including earned value management data, as 8. A life-cycle cost of the acquisition, and time basis for the estimate; 9. A planned procurement schedule, including the best estimate of the annual cost and increments/units to be procured annually; 10. A table delineated by appropriation that provides the actual or estimated appropriations, obligations, unobligated authority, and planned expenditures; 11. The reason for any significant changes from the previous CASR in acquisition quantity, cost, or schedule; 12. Key events or milestones from the prior fiscal year; and 13. Key events or milestones for the current fiscal year.

DHS Has Taken Steps to Improve Oversight of Major Acquisition Programs, but Lacks Written Guidance and Cost Oversight Mechanism for Some Programs

Although DHS has taken steps to improve oversight of major acquisition programs, such as clarifying the role of the CAEs, it lacks written guidance for a consistent approach to oversight. Specifically, there is no guidance to define the roles and responsibilities of PARM and other DHS headquarters organizations in providing day-to-day support and oversight to programs during the acquisition process. PARM started conducting monthly high visibility meetings to discuss programs that require immediate or additional management attention. PARM also maintains a list of programs subject to oversight, the MAOL. The process for creating this list has fluctuated over time; PARM recently made revisions to the MAOL and plans to make further changes to it in the future. Finally, DHS has not established a structure for overseeing the costs of 42 programs in sustainment whose acquisition documentation requirements were waived by the USM in 2013. Sustainment costs can account for more than 80 percent of total costs, and all but one of these programs lack an approved cost estimate.

DHS Lacks Written Guidance for a Consistent Approach to Ongoing Oversight

While DHS has made progress in defining and documenting roles and responsibilities in the oversight of major acquisitions, such as issuing guidance describing the roles of CAEs, the roles and responsibilities of PARM and other DHS headquarters organizations are not clear. Without defined roles and responsibilities, DHS cannot ensure it is providing the appropriate level of oversight or receiving the right information to conduct oversight. PARM has made efforts to expand its oversight and support roles through its component leads, PARM’s liaisons to the components; however, roles and responsibilities for these positions are not defined. In addition, there was no guidance to define the differences in the role of PARM and OCIO-Enterprise Business Management Office (EBMO) in the oversight of major IT acquisitions, and we found potential overlap in the roles of these entities.

Figure 2 illustrates PARM’s interactions with DHS headquarters, component, and program-level offices and officials with acquisition oversight responsibility. Some of these interactions are set forth in policy while others are not.

PARM provides ongoing oversight and support to programs in a number of ways, such as consulting with program officials to prepare required documents prior to an Acquisition Review Board and providing training to components on various aspects of program management. One of PARM’s key mechanisms for providing day-to-day oversight and support to programs between acquisition decision events is through its staff of 10 component leads, but their roles and responsibilities are not defined in DHS acquisition policy. According to PARM officials, the component leads provide day-to-day oversight and support to acquisition programs for a specific component and are intended to be a key source of communication and coordination between PARM and the programs. In turn, component leads provide program information to PARM’s executive director, which may be used in high visibility meetings with the USM.

PARM component leads told us they interact directly with the CAEs and program offices to ensure that programs are adhering to the acquisition process, along with meeting acquisition milestones and reporting requirements. While PARM’s component leads play an important role in the coordination with components, their roles and responsibilities are not defined in DHS acquisition policy. We found that their involvement and relationships with components varies significantly. For example, PARM’s component lead for U.S. Citizenship and Immigration Services is involved in the day-to-day management of programs. This component lead regularly attends component and program-level meetings and organizes training workshops to educate component and program-level staff. In another example, the PARM component lead for the National Protection and Programs Directorate provided additional guidance and attention to the directorate’s programs while the acting CAE was learning his role. In contrast, a U.S. Coast Guard official told us that while there is informal, almost daily communication, their component lead does not have direct access to program level data and relies on the input of the CAE to schedule and prioritize department-level acquisition milestone meetings. Such differences in the PARM component leads’ involvement with programs may be appropriate depending on the type of program or experience of component and program office staff; however, without defined roles and responsibilities, PARM cannot ensure it is providing the appropriate level of oversight or receiving the right information to conduct oversight.

GAO, Auditing and Financial Management: Standards for Internal Control in the Federal Government, GAO/AIMD-00-21.3.1 (Washington, D.C.: Nov. 1, 1999). leads are not defined in DHS acquisition policy and it is a challenge that they are trying to determine how to address.

Further, while PARM is the lead office responsible for overseeing all major acquisition programs, we found confusion among component officials related to PARM’s role in IT acquisitions, where OCIO-EBMO also has oversight responsibility. Of the 72 Level 1 and Level 2 acquisition or service programs listed on the 2014 MAOL, 57 are designated as IT programs. Per the DHS acquisition policy, the OCIO is responsible for establishing IT policies and procedures and ensuring that approved IT acquisitions comply with technical requirements and departmental management processes, such as Agile development, which calls for producing software in small, short increments. Within OCIO, EBMO has been given primary responsibility for ensuring that the department’s IT investments align with its missions and objectives. However, while DHS acquisition policy outlines responsibilities for PARM and OCIO, there is no guidance that defines how the role of PARM differs from the role of EBMO in the oversight of IT acquisition programs.

GAO, Auditing and Financial Management: Framework for Assessing the Acquisition Function At Federal Agencies, GAO-05-218G (Washington, D.C.: Sept.1, 2005). the oversight of major acquisitions could improve coordination, limit overlap of responsibilities, and reduce duplicative efforts at the component level.

DHS Issued Guidance to Clarify the Role of the CAE

In September 2014, the USM issued a policy memorandum clarifying the responsibilities of the CAEs, who have an important role in acquisition oversight. To strengthen acquisition oversight within the department, the USM intends to standardize these officials’ acquisition authorities and experience levels. The memo also sets forth oversight responsibilities of the CAEs, particularly for the Level 3 programs for which they are the acquisition decision authority. The memorandum additionally clarifies that for the purposes of acquisition oversight, program managers report to their CAE and the CAEs report to the USM. This clarification is useful, as CAEs we spoke with prior to the issuance of the memorandum noted differences in the roles and responsibilities of the CAEs across components. For example, at U.S. Immigration and Customs Enforcement, it was the component OCIO, rather than the CAE, who was responsible for the execution of acquisitions, and program managers reported directly to the component OCIO. At the U.S. Coast Guard, the CAE is currently the Vice Commandant, who oversees all of the component’s operations and mission support functions, including human resources, budget, and acquisitions. Within mission support is the Assistant Commandant for Acquisitions, who has more direct oversight of the U.S. Coast Guard’s acquisition programs. Given the new requirements for CAE experience levels, PARM’s executive director anticipates that there may be changes in CAE assignments for at least one component.

The memo directs PARM to create and provide executive-level acquisition training to the CAEs. The memorandum further outlines the CAEs’ responsibilities for complementing PARM’s oversight activities, such as responding in a timely manner to requests for information. As of March 2014, PARM began working with CAEs to hold monthly forums to discuss topics such as the MAOL, staffing plans, and the CASR.

PARM Highlights Some Programs for Oversight through High Visibility Meetings, but DHS Lacks a Cost Oversight Mechanism for Programs In Sustainment

PARM established monthly high visibility meetings to discuss programs that require immediate or additional management attention. In addition, to identify programs for which PARM has oversight responsibility, PARM maintains a list of DHS’s acquisition programs on the MAOL, which is broken down into categories that describe each program’s reporting characteristics. However, DHS has not established a structure for overseeing the costs of 42 programs in sustainment whose acquisition documentation requirements were waived by the USM in 2013. Sustainment costs can account for more than 80 percent of total costs, and all but one of these programs lack an approved cost estimate.

PARM High Visibility Meetings Highlight Programs for Management Attention

PARM’s executive director established high visibility meetings in December 2013 to discuss any acquisition programs that require more immediate attention from DHS management. PARM’s executive director uses these meetings as a management tool. He identifies the programs to be discussed in consultation with component leads. PARM’s executive director told us that the purpose of these meetings is to make sure that senior leadership—including the USM, Chief Financial Officer, Chief Information Officer, Chief Readiness Support Officer, Chief Procurement Officer, and General Counsel—have a common understanding of the acquisition programs’ status and key issues. According to PARM officials, the high visibility meetings have provided better focus through greater senior level involvement and led to a reinvigoration of preparation for Acquisition Review Boards. As of November 2014, 33 programs have been discussed in the high visibility meetings.

PARM officials put programs on the meeting agenda based on a variety of considerations: programs with an upcoming Acquisition Review Board meeting, programs with concerns or issues, and programs that PARM is monitoring closely. Officials told us that the last two categories may include programs under GAO or Inspector General review, programs involved in a bid protest, and programs that have experienced schedule slips or a cost increase. For example, PARM officials told us about a program that changed its acquisition strategy to incorporate information technology, but did not involve the Chief Information Officer. PARM included this program in a high visibility meeting to ensure that officials were informed of the change in strategy and were involved as appropriate. In another case, PARM officials told us that they used high visibility meetings to raise early awareness about concerns with a program, which resulted in multiple follow-on meetings among high level headquarters and component officials. The USM directed the component to pause the program and issued an acquisition decision memorandum that described the path forward.

PARM is Taking Steps to Improve the MAOL

DHS acquisition policy provides the overall structure for acquisition management that programs are required to follow. The policy requires PARM to create a list of major acquisition programs, the MAOL, a document approved by the USM. PARM uses the MAOL to identify programs for which it has oversight responsibility and to determine which programs they include in the CASR, an annual report to Congress. In 2014, PARM updated and expanded the MAOL by listing programs in six categories that detail the characteristics of programs. Five categories specifically address acquisition programs (see table 2). In addition, there is one category for a non-acquisition activity that is required to submit an Office of Management and Budget business case.

PARM officials stated that they updated the list to more clearly incorporate input of all headquarters organizations, thereby making it a more useful oversight tool. The list has evolved over time as more headquarters organizations have added programs to the MAOL. PARM officials have drafted updates to DHS acquisition policy that include a section on requirements for the MAOL. Specifically, the planned updates will include which headquarters organizations will be involved in the development of the list, and establish a process for removing programs. PARM officials also told us they recently began a process for updating the list more regularly. The updates also provide additional information about the development and use of the MAOL. The draft updates describe the process for determining whether or not a program belongs on the MAOL, which follows a decision tree. PARM officials said that the new process for developing the MAOL more effectively coordinates and tracks the input from other DHS headquarters organizations, like EBMO and the Office of the Chief Financial Officer, as well as CAEs. The draft updates also describe justifications for removal from the MAOL. For example, a program might be removed if it is merged with another program, or if it is no longer considered special interest—meaning that a program was elevated to a higher acquisition level without regard to dollar threshold. Officials were unsure when the draft updates would be approved by DHS management. In addition to the draft policy updates, PARM officials told us that they recently instituted a governing board of officials who will determine changes to the MAOL on a quarterly basis, given its potential to provide important information to department decision makers. PARM officials told us that the next MAOL, expected in February 2015, will use the new process described in draft guidance.

DHS Lacks Cost Oversight Mechanism for Programs in Sustainment

DHS does not have a structure in place for overseeing the costs of 42 programs whose acquisition documentation requirements were waived This waiver through a memorandum issued by the USM in May 2013.covered certain programs in sustainment, meaning that these acquisition programs have been developed and delivered and they are being operated and maintained through the disposal phase. Because these programs were in sustainment when MD 102 was instituted in 2008, the USM determined that it would be cost prohibitive and inefficient to recreate documentation for previous phases. However, we found that only one of the 42 waived programs has an approved life-cycle cost estimate, which would include acquisition costs as well as the costs to operate and maintain the system once it is in sustainment. PARM officials could not provide us estimates of the value of the sustainment programs. PARM’s executive director stated that these programs should produce operations and maintenance cost estimates. These estimates would account for the remainder of their life cycles through disposal, but the programs are not currently required to do so, given the 2013 waiver.

Further, in the 2014 MAOL, PARM included seven additional programs in sustainment and also noted that documentation was waived for these programs. The 42 programs in sustainment from the USM’s memorandum and the seven programs listed on the MAOL are listed in appendix II. The Office of Management and Budget stated in 2014 that the sustainment phase can account for more than 80 percent of program life-cycle costs, which demonstrates the need for oversight of these programs’ costs. We have previously reported that cost estimates are necessary to support decisions about program funding, develop annual budget requests, and evaluate resource requirements. Furthermore, the management of a cost estimate involves continually updating the estimate with actual data as they become available, revising the estimate to reflect changes, and analyzing differences between the estimated and actual costs. Without knowing the operations and maintenance cost estimates for these programs, DHS will not be able to fully plan for and manage funding requirements across its major acquisition programs.

The 2013 waiver did not define which DHS office is responsible for oversight of the sustainment programs. CAEs are responsible for Level 3 programs and PARM officials stated that this also applies to programs in sustainment. A PARM official further told us it is difficult to know who is responsible for oversight of Level 1 and Level 2 programs in sustainment. PARM officials expressed concerns about the lack of oversight of these programs. Specifically, officials noted DHS may decide, on a case-by- case basis, which organization should most appropriately provide oversight to programs in sustainment, which may include more than one office.

Program Data PARM Provided to DHS and Congressional Decision Makers Were Not Consistently Accurate and Up-to-Date

PARM’s fiscal year 2014 CASR, a report mandated by Congress, provided the status of 82 DHS major acquisition programs but contained data that were inaccurate and out-of-date. PARM primarily drew information for the CASR from nPRS, DHS’s official system of record for acquisition program reporting. However, data issues—including inconsistent participation among the programs responsible for entering data—have led to inaccurate information in nPRS. For example, our analysis found discrepancies between the CASR and nPRS for life-cycle cost estimate data even after efforts to update or fix the data inaccuracies through an extensive adjudication process. Therefore, it was unclear whether congressional CASR recipients received accurate program information. PARM officials have acknowledged ongoing issues with the data reported in both nPRS and the CASR, and noted that they are working to improve the data quality. Officials stated that information in the CASR did not provide a complete picture of program life-cycle costs, which was the result of both incorrect data that programs had reported and limitations in using the nPRS system. Further, component and program officials have also stated that the CASR did not accurately reflect program risks. Finally, DHS provided insufficient information to address certain CASR reporting requirements. For example, the CASR did not include annual planned procurement schedules containing estimates of the units and/or increments for each program, although it was required to do so.

Data in DHS’s Acquisition Program Reporting System Were Not Consistently Accurate

For the nine programs in our review, we found that program offices did not consistently enter and verify their data in nPRS. DHS established nPRS as the system of record for acquisition program reporting in 2008, and in 2012 the USM issued a memorandum to CAEs stating that programs should make every effort to ensure that their data in nPRS is complete, accurate, and valid on a monthly basis. According to the memorandum, nPRS was intended to be a key tool for acquisition program management, and help provide the capability to efficiently assess the department’s acquisition portfolio. DHS components have the responsibility to ensure that their respective programs enter the data in nPRS as required, and CAEs are required to ensure that the data is validated and submitted in timely manner. However, we found that this was not done consistently for the nine major acquisition programs in our review, which are overseen by nine different DHS components.

We examined nPRS data for the nine programs at two key points: September 2013, the closing date for data for fiscal year 2013, and March 2014, the date when PARM issued its fiscal year 2014 CASR, which was based on fiscal year 2013 program data. We found a number of problems with the data. For example, as of September 2013, three programs we reviewed did not enter expenditure data, the amount the programs actually spent, in nPRS for fiscal year 2013 as required, and two of these programs did not enter historical expenditure data at all.

When we compared programs’ entries of expenditure data over time, we found additional discrepancies. As an example, the U.S. Coast Guard’s Fast Response Cutter program’s expenditure entries in nPRS increased by more than $340 million from September 2013 to March 2014, even though both of these entries were supposed to reflect fiscal year 2013 expenditures. A U.S. Coast Guard official stated that this increase was due to a correction in the program’s reported expenditures, to account for all funds spent in fiscal year 2013 regardless of when those funds were received. The official noted that the program’s entry from September 2013 reflected only funds received in fiscal year 2013. However, the reason for this change was not documented in nPRS. In another example, the U.S. Citizenship and Immigration Service’s Verification Modernization program’s entries for total historical expenditures through fiscal year 2012 decreased by almost $240 million when comparing these data from September 2013 and March 2014. Figure 3 shows the differences in reported expenditure data in nPRS for the Fast Response Cutter and Verification Modernization programs.

Large nPRS discrepancies such as these call into question the reliability of the underlying data and whether DHS management has the information it needs to provide oversight of major acquisition programs. PARM officials have acknowledged ongoing issues with the data reported in nPRS and noted that they are working to improve its quality. For example, PARM provides a working group to the components to express their views on the nPRS system and its processes. However, PARM officials stated they do not have a mechanism to hold the programs accountable for updating their data. Component and program officials told us that they do not use nPRS for program management—even though that was one intended purpose of the system—because the system is difficult to use and does not meet their needs. For example, the OCIO National Capital Region Infrastructure Operations program manager stated that his program does not work with nPRS. For both September 2013 and March 2014, nPRS data fields for this program that were to be used to populate the fiscal year 2014 CASR, such as the program description and last acquisition review board date, were blank. Component and program officials also stated that they use other internal tools, such as spreadsheets, presentations, or project software for program management purposes and to maintain current information. PARM has not undertaken an effort to ascertain the root causes of why program managers are not populating nPRS as required. As we have previously reported, to be useful, performance information must meet users’ needs for completeness, accuracy, consistency, timeliness, validity, and ease of use. Unless DHS program managers consider nPRS to be a useful tool for their own program management purposes, as intended, the problems we found with inaccurate data are likely to persist.

Extensive Adjudication Process Did Not Correct Inaccuracies for Reported Life-Cycle Cost Estimates

To further understand the reasons for program data inaccuracies in the fiscal year 2014 CASR, we analyzed the steps that PARM undertakes as part of an extensive adjudication process with the components regarding the underlying nPRS data. We found that PARM’s adjudication process did not rectify key inaccuracies in the fiscal year 2014 CASR, specifically regarding programs’ life-cycle cost estimates. As a result, Congress may not have received accurate program information. Prior to its release, PARM conducts an extenstive adjudication process for the CASR information, including reviews with program and various DHS headquarters offices, in order to identify and address potential data inconsistencies between the sources and draft report. This process began in October after the close of the fiscal year and ended when the report was published in March. We reviewed nPRS data from March 2014, the date when PARM issued its fiscal year 2014 CASR (which is comprised of fiscal year 2013 program data) to compare these data to what was presented to Congress in the CASR. Figure 4 shows select elements of the CASR development and adjudication process, along with our assessment of those elements.

The fiscal year 2014 CASR reported on a total of 82 major acquisition programs. For four of the nine major acquisition programs in our review, we found discrepancies between the CASR and nPRS for life-cycle cost estimate data after the adjudication process, which should have reconciled such inconsistencies. As an example of these discrepancies, both across nPRS and between nPRS and the CASR, life-cycle cost estimates for two programs differed even though the estimates were associated with the same source and date. Another program, the Electronic Health Record System, had three different life-cycle values ranging from approximately $60 million to $80 million, a difference of over 35 percent, with two of those values presented in the CASR. Our analysis of the data inconsistencies indicated that it was unclear whether congressional CASR recipients received accurate program cost information because there was no way to confirm which estimate was correct or what the different estimates represented. PARM officials stated that they are changing their process for the development of the next CASR, which they expect to issue with the President’s fiscal year 2016 budget submission, in an effort to more effectively match their reported data to nPRS. Table 3 highlights discrepancies between the CASR and nPRS for life-cycle cost estimate data for the four programs that we reviewed.

Further, PARM may have incorrectly included or excluded certain programs in the CASR based on DHS’s incomplete information on program life-cycle costs. As we reported in 2014, unreliable cost estimates have been an enduring challenge for DHS. PARM included Level 1 and 2 programs from the MAOL in the CASR, as required, using program life-cycle costs to determine program status. However, PARM officials acknowledged that some programs’ acquisition category levels were incorrect in the CASR, and due to the lack of DHS approved life- cycle cost estimates, they would not know the scope of this issue. For example, the CASR included program life-cycle cost estimate figures, but did not indicate who approved these estimates (i.e., if they were approved at the component or department level), or if anyone approved the estimates at all. Because these cost figures may not accurately reflect the actual life-cycle costs, programs may have been inappropriately included or excluded from the CASR and ultimately not receive the appropriate level of congressional oversight.

We also found areas where additional explanation in the CASR would have been helpful. For example, PARM listed the Electronic Health Record System program in the CASR as a Level 2 program, while its reported life-cycle cost estimate of approximately $70 million would designate it a Level 3 program. PARM is not required to include Level 3 programs in the CASR. Electronic Health Record System officials explained that the program was listed as Level 2 because DHS management designated it as a “special interest” program in the MAOL, which did make it eligible for inclusion in the CASR, but PARM did not include this rationale in the CASR.

In addition, program officials stated that the inflexibility of nPRS may prevent the department from providing accurate program information in the CASR. For example, officials from the National Protection and Programs Directorate’s Next Generation Network-Priority Service noted difficulties in accurately reporting data on their program’s increments in nPRS. The officials stated that the program has multiple increments, each with its own set of acquisition decision events. However, the program reported one overall life-cycle cost estimate in nPRS, even though it has estimates for each increment, because the system does not allow for the inclusion of multiple estimates. Officials ultimately provided explanatory comments in nPRS that noted the increment 1 estimate included only acquisition costs while not specifying who approved the estimate, and that PARM approved the increment 2 estimate in July 2013. Because these incremental estimates were at different stages in their development, combining them into a single estimate in nPRS, which PARM ultimately reported in the CASR, did not provide Congress with an accurate picture of the program’s costs.

DHS Did Not Provide Most Useful Information on Program Risks and Other Requirements for the CASR

Of the 13 CASR reporting requirements, DHS provided insufficient information for in-depth oversight for four of them, and in one case, DHS did not comply with a reporting requirement. The first reporting requirement was a rating of cost risk, schedule risk, and technical risk associated with the program. PARM fulfilled this requirement by reporting programs’ top-five cost, schedule, and technical risks, instead of separate ratings for each. However, this reporting was inconsistent. Programs submitted these risks through DHS’s Investment Management System, but this system can contain more than five risks. As a result, program officials stated that they did not know how PARM selected their top-five risks for inclusion in the CASR. For example, the Strategic Air and Marine Program listed nine risks in the system, but PARM only reported two risks in the CASR. Further, the risk reporting in this section varied across the programs in our review. For example:

The Verification Modernization program had five risks, all of which were technical.

The Technology Infrastructure Modernization program used four of its own risk categories, which do not track to the required cost, schedule, and technical risks: reliability of systems; dependencies and interoperability between this investment and others; security; and business.

The inconsistent risk reporting in this section prevents Congress from making cost, schedule, and performance comparisons across DHS programs.

In addition, as part of a separate requirement for independent verification and validation, PARM evaluated the overall risk of each program, assigned each a numerical risk score, and published these scores in the CASR. PARM officials computed these risk scores using a set of weighted criteria. However, component and program officials told us that they did not know how PARM evaluated the risk scores for their respective programs. PARM, component, and program officials have acknowledged that the CASR did not accurately reflect the cost, schedule, and performance risks associated with the programs. PARM officials were unable to provide us with the supporting data used to generate the scores because they did not store this information in nPRS. As a result, we were unable assess how PARM computed these scores or determine the extent to which PARM’s evaluation accurately reflected program risks.

Due to PARM’s inability to provide us with supporting data, we reviewed the Science and Technology Directorate’s National Bio and Agro-Defense Facility program’s risk assessment in the CASR and found it lacked details that could be useful to Congress and DHS management. PARM gave the program a high risk score, in part, due to the lack of DHS approval for the program’s acquisition documents. A program official stated, however, that the documents could not go forward for DHS approval due to the program’s lack of funding. The CASR did not contain any clarifying explanation for the documents not being approved. Further complicating the issue, the program’s nPRS data in September 2013 conflicts with its CASR entry, and showed program documents approved by DHS as early as 2009. Discrepancies such as this call into question the value of the information DHS is providing to Congress in the CASR. To gain more visibility into the reasons for these inconsistencies, we reviewed the source documents for the National Bio and Agro-Defense Facility program, provided to us by PARM, and found that the approval dates for five out of six documents do not match what was listed in either nPRS or the CASR. Table 4 compares the National Bio and Agro- Defense Facility program’s reported document status according to nPRS as of September 2013 and the CASR issued in March 2014, and the source documents provided by PARM.

In another example, in PARM’s risk assessment of the Federal Emergency Management Agency’s Risk Mapping, Assessment and Planning program, the CASR stated that the program was covered by the USM’s waiver of acquisition documents requirements and thus did not include certain program data. However, an examination of nPRS showed that, according to the system, the program had key documents, including a mission needs statement and an acquisition program baseline approved by DHS, which PARM did not list in the CASR and which could have provided further information to decision makers.

A second CASR reporting requirement was a program’s planned procurement schedule, including an estimate of the quantity to be procured annually until completion. PARM did not comply with this requirement. Instead, PARM provided the top-five contracts by dollar value for each program, but these entries did not include procurement quantity information for the programs. Some programs did report total procurement quantities, but did not link these units to a schedule. PARM officials noted that certain programs are not well-suited for reporting procurement quantities, such as IT programs. However, in such cases, the CASR should explain why no procurement quantities were listed.

A third requirement was the reason for any significant changes in a program’s acquisition cost, quantity, or schedule from the prior annual CASR. PARM officials interpreted these changes to only be those that resulted in the submission of a new acquisition program baseline. According to DHS acquisition policy, programs need to submit new baselines when they breach defined cost, schedule, and performance parameters defined in their original baseline. However, programs can experience cost, quantity, or schedule changes that do not require a new baseline. For example, the National Bio and Agro-Defense Facility program experienced a delay in its construction schedule. While this program’s baseline remains in place, the construction delay could impact on-time delivery of the facility and is an example of a significant change that could be reported in this section. In addition, the CASR included 38 programs without an approved acquisition program baseline; according to PARM’s guidance, the CASR would not include any cost, schedule, or performance changes for these programs. By defining programs’ significant changes as those that resulted in new baselines, PARM eliminated the need to report on any cost, schedule, or performance changes for almost half of the programs in the CASR, thereby limiting the information available to Congress.

Finally, we found that PARM did not include certain key program events in the CASR, such as acquisition decision events or full operating capability schedules. Such data, in addition to the acquisition program baseline approval dates that PARM currently reports, would have provided Congress with more robust information about the program status. Table 5 lists these four CASR requirements and our assessment of the information reported.

Conclusions

Effective, on-going oversight of DHS’s broad portfolio of programs is essential to ensure that programs are accountable for their performance and that Congress and DHS decision makers receive useful, accurate, and up-to-date information. DHS has improved aspects of its acquisition management in recent years, including dedicating additional resources to acquisition oversight and clarifying the roles of CAEs.

DHS could further enhance its oversight efforts by providing written roles and responsibilities to oversight officials within PARM and among headquarters organizations. Furthermore, a consistent, defined approach to oversight could limit overlap of responsibilities and give DHS more insight into whether its acquisition programs are executing according to cost, schedule, and performance goals.

Likewise, as DHS acquisition programs move into the sustainment phase, their costs continue to require monitoring. The USM’s waiving of documentation requirements for the 42 programs in sustainment in 2013 resulted in a lack of oversight of costs for these programs. As of yet, no DHS office has been designated to take over monitoring those programs’ operations and maintenance costs. Without an identified oversight body, DHS lacks insight into those programs’ performance and the execution of their funding, which could potentially be billions of dollars. This is particularly of concern given that only one program had an approved cost estimate at the time of the waiver.

Finally, DHS has not effectively communicated program status to Congress through the CASR because it has provided out-of-date and inaccurate information. Programs do not consistently report their own data in nPRS, and components are not validating the information. Although PARM’s adjudication process may address some data issues, data are not corrected in the source systems before being published in the CASR. Further, while PARM has some flexibility in the implementation of the CASR reporting requirements, in one case the requirement was not met. In other cases, such as PARM’s assessment of program risks, there are opportunities for more transparency and clarity in the information being transmitted to Congress. Holding programs accountable for maintaining their cost, schedule, and performance data, and presenting contextual information would help make the CASR a more effective instrument for DHS and congressional oversight.

Recommendations for Executive Action

In order to help ensure consistent, effective oversight of DHS’s acquisition programs, we recommend the Secretary of DHS take the following five actions:

Direct PARM to develop written guidance that defines roles and responsibilities of its component leads.

Direct the USM to:

Develop written guidance to clarify roles and responsibilities of PARM and OCIO-EBMO for conducting oversight of major acquisition programs.

Produce operations and maintenance cost estimates for programs in sustainment and establish responsibility for tracking sustainment programs’ adherence to those estimates.

Determine mechanisms to hold programs accountable for entering data in nPRS consistently and accurately and to hold CAEs accountable for validating the information. Also, evaluate the root causes of why programs are not using nPRS as intended.

To make the CASR more useful, starting with the report reflecting fiscal year 2015 program data, adjust the CASR to do the following:

Report an individual rating for each program’s cost, schedule,

Report a best estimate of procurement quantities or indicate why this is not applicable, as appropriate;

Report all programs’ significant changes in acquisition cost, quantity, or schedule from the previous CASR report by determining a means to account for programs that lack acquisition program baselines;

Report major program events that are included in acquisition program baselines, such as scheduled acquisition decision events; and

Report the level at which the program’s life-cycle cost estimate was approved.

Agency Comments and Our Evaluation

We provided a draft of this product to DHS for comment. In its written comments, reproduced in appendix III, DHS concurred with all five of our recommendations and provided plans of action and estimated completion dates for four of them. Regarding the remaining recommendation, that the Secretary of DHS direct PARM to develop written guidance that defines roles and responsibilities for component leads, DHS provided evidence that is has complied with the recommendation, and we agree. Specifically, DHS provided a Component Lead Handbook, signed on February 13, 2015, while our report was out for comment, that provides oversight roles and responsibilities and other guidance to PARM component leads in their job to oversee component programs.

DHS also provided technical comments that we incorporated into the report as appropriate.

As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the Secretary of DHS. In addition, the report will be available at no charge on GAO’s website at http://www.gao.gov.

If you or your staff have questions about this report, please contact me at (202) 512-4841 or mackinm@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix IV.

Appendix I: Objectives, Scope, and Methodology

The objective of this review was to assess the Department of Homeland Security’s (DHS) oversight of its major acquisition programs. Specifically, this review focused on DHS’s Office of Program Accountability and Risk Management (PARM) and its day-to-day program oversight, rather than the oversight it conducts at key points in the acquisition life cycle as defined in policy. We assessed (1) steps DHS has taken to improve oversight and what gaps, if any, exist and (2) whether the data PARM provides to DHS and congressional decision makers to carry out their oversight responsibilities on program cost, schedule, and performance are accurate and up-to-date.

To answer these questions, we identified organizations within DHS, in addition to PARM, that are responsible for oversight of major acquisitions and determined their roles and responsibilities by analyzing DHS policies and procedures, reviewing organizational charts, and interviewing policy, budget, and acquisition oversight officials at the headquarters level. Specifically, we reviewed DHS Acquisition Management Directive 102-01 (MD 102) and its associated guidebook—DHS Instruction Manual 102-01- 001—and the guidebook’s 12 appendices. We reviewed draft updates to DHS acquisition policy, as well as draft updates to departmental instructions, such as Agile Development and Delivery for Information Technology and the Systems Engineering Life Cycle Guidebook. We also reviewed DHS acquisition memorandums, including the Secretary’s April 2014 Unity of Effort memorandum; the Office of the Chief Procurement Officer’s Strategic Plan; information technology (IT) policies and guidance, such as DHS Directive 102-04 on IT portfolio management and the Office of the Chief Information Officer (OCIO) Portfolio Governance Concept of Operations. At the department level, we interviewed officials from PARM, OCIO–Enterprise Business Management Office (EBMO), Office of the Chief Procurement Officer, Office of Policy, and Office of the Chief Financial Officer–Office of Program Analysis and Evaluation and Cost Analysis Division. In addition, we reviewed relevant GAO and DHS Inspector General reports to provide context for all of our objectives.

To address our first objective, we selected nine DHS components with responsibility for at least one Level 1 acquisition—a program with a reported life-cycle cost estimate exceeding $1 billion—and interviewed their Component Acquisition Executives (CAE) or designees. We reviewed component-specific policies and procedures and charters for program governance groups, as well as other relevant documentation. The 2014 Master Acquisition Oversight List (MAOL) identifies Level 1 acquisition programs for the following nine components:

Federal Emergency Management Agency

National Protection and Programs Directorate

OCIO

Science and Technology Directorate

Transportation Security Administration

U.S. Citizenship and Immigration Services

U.S. Coast Guard

U. S. Customs and Border Protection

U.S. Immigration and Customs Enforcement To collect examples of PARM’s oversight and coordination activities at the program level, we selected a non-generalizable sample of major acquisition programs from each of the nine components. Table 6 lists the nine programs we selected as case studies.

We selected programs that were included in the fiscal year 2014 Comprehensive Acquisition Status Report (CASR), which PARM submitted to the House and Senate Appropriations Committees to provide information on DHS major acquisition programs. To the extent possible, we chose programs that have been identified as having “concerns/issues” or as being “monitored closely” in PARM’s high visibility meetings, which include DHS senior leadership. In order to assess acquisition oversight across the spectrum of DHS programs, we selected case study programs with a variety of characteristics. We chose a mix of Level 1 and Level 2 programs, as defined in the fiscal year 2014 CASR, and included both IT and non-IT programs. One of our nine case study programs, the Electronic Health Record System, was classified as a Level 2 program in the fiscal year 2014 CASR, but listed as a Level 3 program on the 2014 MAOL. We chose this program in order to examine the reasons for the change and to determine the extent to which oversight varies for major and non-major acquisition programs. Another factor used to select the case studies was program risk, as measured by CASR risk scores. Risk scores are included in the CASR’s independent verification and validation section and are derived from PARM’s rating of program risk using a standard set of criteria. We chose programs to include a mix of both high and low CASR risk scores. For these case studies, we reviewed relevant program documentation, such as acquisition decision memorandums, and interviewed program officials.

For the second objective, we collected and reviewed data from DHS’s official system of record for its acquisition programs, the Next Generation Periodic Reporting System (nPRS), and compared that data to the fiscal year 2014 CASR. All major programs on DHS’s MAOL are required to report in nPRS. PARM then uses the program data in nPRS to help generate its CASR.

In order to assess the data reliability of nPRS, we reviewed select acquisition program data from nPRS and compared this data to the information contained in the CASR. Specifically, we used nPRS reports for each of the nine case study programs from the end of fiscal year 2013, when PARM pulled the program data from the system to begin generating the fiscal year 2014 CASR. We then compared the data from those reports to the issued CASR, as well as to the nPRS program reports from March 2014—the date that PARM released the CASR. The comparison of those three sets of information allowed us to note discrepancies, including missing data or outliers, between the system data and the issued data, as well as if corrections were made to the system data following the release of the report.

We assessed various data elements from the nPRS program reports that are used to generate the information contained in the CASR. We reviewed data across a range of tabs contained in the nPRS program reports such as general information, Acquisition Review Board history, program status, budget and funding, acquisition program baseline milestones, risk, and key documents. For each of the nine case study programs, we assessed reports from the end of fiscal year 2013, as well as March 2014, for a total of eighteen program reports. In addition, we reviewed documents, such as the nPRS user manual and policies related to data entry, and interviewed agency officials responsible for inputting and reviewing the nPRS data.

We determined that the nPRS data were not sufficiently reliable for our purposes; however, we present the data for illustrative purposes only. For example, for certain programs in our review, current and historical expenditure data was missing. Another example from our analysis showed differences in the life-cycle cost estimates for certain programs in nPRS compared to those reported in the CASR. While the Under Secretary for Management (USM) issued a memorandum that programs are to update their nPRS data monthly, PARM officials recognized that this does not happen consistently, and they acknowledged that there are data accuracy issues with nPRS. In addition, when officials made updates or changes to program-reported information due to the pending release of the CASR, the programs were responsible for entering these updates or changes into nPRS. Our analysis confirmed that certain programs in our review did not update nPRS after going through the CASR reporting process.

In order to evaluate the effectiveness of nPRS and the CASR as tools for DHS management and congressional oversight, we first reviewed the Department of Homeland Security Appropriations Act, 2014, which established the provision for the USM to submit the CASR with the President’s budget proposal for fiscal year 2015. In addition, we reviewed Conference Report 112-331 for the Consolidated Appropriations Act, 2012, which contained the information requirements for inclusion in the CASR. We then compared the data from nPRS for the nine case study programs to that presented in the fiscal year 2014 CASR to determine discrepancies between the two for certain data elements. We further compared the CASR information to PARM policies and procedures, such as the MAOL and MD 102. In order to review the CASR’s independent verification and validation requirement, we asked PARM officials for supporting documentation for how they generated their independent verification and validation evaluations, but they were unable to provide the documentation because they did not store it in nPRS. Finally, we assessed the information the CASR either did or did not provide compared to its congressional reporting requirements. We conducted this assessment based on the CASR’s congressional reporting requirements, DHS policies, and acquisition practices.

We conducted this performance audit from March 2014 to March 2015 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Department of Homeland Security Acquisition Programs in Sustainment Exempt From Documentation Requirements

Table 7 below identifies 42 acquisition programs for which the Department of Homeland Security’s (DHS) Under Secretary for Management waived documentation requirements in a May 2013 memorandum. These programs were already in sustainment prior to 2008, meaning that they were in the last phase of their acquisition life cycle, when DHS issued MD 102. Programs in sustainment have been developed and delivered to their respective components for operation and maintenance through disposal. The memorandum stated that it would be cost prohibitive and inefficient for these programs to recreate the documentation called for under the directive for their previous acquisition life-cycle phases.

Table 8 below identifies seven additional major acquisition programs in sustainment for which DHS waived documentation requirements on the 2014 Master Acquisition Oversight List. The Master Acquisition Oversight List stated that having programs provide documents for their previous acquisition life-cycle phases would be costly and provide no positive performance impact for systems already delivered.

Appendix III: Comments from the Department of Homeland Security

Appendix IV: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, Katherine Trimble, Assistant Director; Leigh Ann Haydon, Analyst-in-Charge; Stephen V. Marchesani; Alexis Olson; Sarah Marie Martin; and Daniel Hilger made key contributions to this report. Peter W. Anderson, Jean L. McSween, Ozzy Trevino, and Alyssa Weir also provided assistance.