Homeland Security: DHS Requires More Disciplined Investment Management to Help Meet Mission Needs
Published: 2012-09-18T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: DHS invests extensively in major acquisition programs to develop new systems that help the department execute its many critical missions. In 2011, DHS reported to Congress that it planned to invest $167 billion in these major acquisition programs. We previously found that DHS had not managed its investments effectively, and its acquisition management activities have been on GAOs High Risk List since 2005. This report addresses the extent to which (1) major DHS acquisition programs face key challenges; (2) DHS has policies and processes to effectively manage individual acquisition programs; (3) DHS has policies and processes to effectively manage its portfolio of acquisition programs as a whole; and (4) DHS has taken actions to address the high-risk acquisition management issues GAO has identified in previous reports. GAO surveyed all 77 major program offices DHS identified in 2011 (92 percent response rate), reviewed available documentation of acquisition decisions from November 2008 to April 2012, and interviewed officials at DHS headquarters and components.
section_title: What GAO Found
paragraphs: Nearly all of the Department of Homeland Security (DHS) program managers GAO surveyed reported their programs had experienced significant challenges. Sixty-eight of the 71 respondents reported they experienced funding instability, faced workforce shortfalls, or their planned capabilities changed after initiation, and most survey respondents reported a combination of these challenges. DHS lacks the data needed to accurately measure program performance, but GAO was able to use survey results, information DHS provided to Congress, and an internal DHS review from March 2012 to identify 42 programs that experienced cost growth, schedule slips, or both. GAO gained insight into the magnitude of the cost growth for 16 of the 42 programs, which increased from $19.7 billion in 2008 to $52.2 billion in 2011, an aggregate increase of 166 percent.
DHS acquisition policy reflects many key program management practices that could help mitigate program risks. It requires programs to develop documents demonstrating critical knowledge that would help leaders make better informed investment decisions when managing individual programs. However, DHS has not consistently met these requirements. The department has only verified that four programs documented all of the critical knowledge the policy requires to proceed with acquisition activities. Officials explained that DHSs culture has emphasized the need to rapidly execute missions more than sound acquisition management practices. Most major programs lack reliable cost estimates, realistic schedules, and agreed-upon baseline objectives, limiting DHS leaderships ability to effectively manage those programs and provide information to Congress. DHS recognizes the need to implement its acquisition policy more consistently, but significant work remains.
DHS acquisition policy does not fully reflect several key portfolio management practices, such as allocating resources strategically, and DHS has not yet re-established an oversight board to manage its investment portfolio across the department. As a result, DHS has largely made investment decisions on a program-by-program and component-by-component basis. The widespread risk of poorly understood cost growth, coupled with the fiscal challenges facing the federal government, makes it essential that DHS allocate resources to its major programs in a deliberate manner. DHS plans to develop stronger portfolio-management policies and processes, but until it does so, DHS programs are more likely to experience additional funding instability, which will increase the risk of further cost growth and schedule slips. These outcomes, combined with a tighter budget, could prevent DHS from developing needed capabilities.
DHS has introduced seven initiatives that could improve acquisition management by addressing longstanding challenges GAO and DHS survey respondents have identified, such as funding instability and acquisition workforce shortfalls. Implementation plans are still being developed, and DHS is still working to address critical issues. Because of this, it is too early to determine whether the DHS initiatives will be effective, as GAO has previously established that agencies must sustain progress over time to address management challenges. DHS is also pursuing a tiered-governance structure, but it must reduce risks and improve program outcomes before regularly delegating major milestone decision authority.
section_title: What GAO Recommends
paragraphs: GAO recommends that DHS modify its policy to better reflect key program and portfolio management practices, ensure acquisition programs fully comply with DHS acquisition policy, prioritize major acquisition programs departmentwide and account for anticipated resource constraints, and document prerequisites for delegating major milestone decision authority. DHS concurred with all of GAOs recommendations, and noted its progress on a number of fronts, which is accounted for in the report.

section_title: Letter
paragraphs: The Department of Homeland Security (DHS) invests extensively in  acquisition programs to develop new systems that help the department  execute its many critical missions. DHS is acquiring systems to help  secure the border, facilitate trade, screen travelers, enhance cyber  security, improve disaster response, and execute a wide variety of other  operations. Several of DHS’s major acquisition programs—the  department’s most expensive and critical investments—existed prior to  the creation of DHS and were managed by one of the 22 separate  agencies that merged to form the department. These acquisition  programs are now managed by senior officials at DHS headquarters and  12 component agencies. In 2011, DHS reported to Congress that it  planned to ultimately invest $167 billion in its major acquisition programs.  In fiscal year 2012, DHS reported it was investing more than $18 billion in  the department’s acquisition programs.
DHS acquisition management issues have been highlighted in our High- Risk List since 2005. Over the past several years, our work has identified  significant shortcomings in the department’s ability to manage an  expanding portfolio of complex acquisitions. In 2008, DHS revised its  acquisition review process to include more detailed guidance for key  acquisition decision events, documentation requirements, and the roles  and responsibilities of DHS decision makers. In January 2010, DHS again  updated its acquisition policy, but later that year, we found that the  department still was not effectively carrying out its acquisition  management responsibilities. We have previously established that a  program must have a sound business case that includes firm  requirements, a knowledge-based acquisition strategy, and realistic cost  estimates in order to reduce program challenges.provide a program a reasonable chance of overcoming challenges yet  delivering on time and within budget.
Because DHS invests significant resources developing capabilities to  support the department’s mission, you asked us to assess the extent to  which (1) DHS’s major acquisition programs face challenges that increase  the risk of poor outcomes; (2) DHS has policies and processes in place to  effectively manage individual acquisition programs; (3) DHS has policies  and processes in place to effectively manage its portfolio of acquisition  programs as a whole; and (4) DHS has taken actions to address the high- risk acquisition management issues we have identified in previous  reports. In addition to this report, we are also issuing a report focused on  the performance of DHS’s major information technology (IT)  investments.
To determine the extent to which major acquisition programs identified by  DHS in 2011 face challenges, we surveyed all 77 major program offices  from January to March 2012, and achieved a 92 percent response rate.
DHS originally identified 82 major acquisition programs in the 2011 major acquisition  oversight list, but five of those programs were subsequently cancelled in 2011. Seventy- one program managers responded to the survey. See appendix IV. risks, management challenges, and data limitations—particularly data  limitations regarding program performance. Further, we reviewed  resource plans and DHS performance reports to establish the extent to  which major acquisition programs are achieving their cost, schedule and  capability objectives. To determine the extent to which DHS has policies  in place to effectively manage individual acquisition programs, as well as  the department’s acquisition portfolio as a whole, we compared our key  acquisition management practices to DHS acquisition policy, and  identified the extent to which DHS has implemented its policy. We also  met with DHS officials to discuss our analysis, identify relevant sections of  the policy that we had not yet accounted for, and solicit their thoughts on  those key practices that were not reflected in the policy. To determine the  extent to which DHS has taken actions to address the high-risk  acquisition management issues we have identified in previous reports, we  analyzed the department’s recently proposed efforts to address high-risk  acquisition management challenges, including the department’s progress  in implementing new initiatives and any challenges DHS must overcome  moving forward.
We conducted this performance audit from August 2011 to September  2012 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: DHS invests in major acquisition programs to develop capabilities  intended to improve its ability to execute its mission. DHS generally  defines major programs as those expected to cost at least $300 million  over their respective life cycles, and many are expected to cost more than  $1 billion. DHS Acquisition Management Directive 102-01 (AD 102) and  DHS Instruction Manual 102-01-001 (Guidebook), which includes 12  appendixes, establish the department’s policies and processes for  managing these major acquisition programs. DHS issued the initial  version of AD 102 in 2008 in an effort to establish an acquisition  management system that effectively provides required capability to  operators in support of the department’s missions. AD 102 establishes  that DHS’s Chief Acquisition Officer—currently the Under Secretary for  Management (USM)—is responsible for the management and oversight  of the department’s acquisition policies and procedures. The USM,  Deputy Secretary, and Component Acquisition Executives (CAE) are the  Acquisition Decision Authorities for DHS’s major acquisition programs.  Table 1 identifies how DHS categorizes the 77 major acquisition  programs it identified in 2011.
The Acquisition Decision Authority is responsible for reviewing and  approving the movement of DHS’s major acquisition programs through  four phases of the acquisition life cycle at a series of five predetermined  Acquisition Decision Events. These five Acquisition Decision Events  provide the Acquisition Decision Authority an opportunity to assess  whether a major program is ready to proceed through the life-cycle  phases. The four phases of the acquisition life cycle, as established in AD  102, are:  1.  Need phase: Department officials identify that there is a need,  consistent with DHS’s strategic plan, justifying an investment in a new  capability and the establishment of an acquisition program to produce  that capability;  2.  Analyze/Select phase: the Acquisition Decision Authority designates a  qualified official to manage the program, and this program manager  subsequently reviews alternative approaches to meeting the need,  and recommends a best option to the Acquisition Decision Authority;  3.  Obtain phase: The program manager develops, tests, and evaluates  the selected option; during this phase, programs may proceed through  ADE 2B, which focuses on the cost, schedule, and performance  parameters for each of the program’s projects; and ADE 2C, which  focuses on low rate initial production issues; and  4.  Produce/Deploy/Support phase: DHS delivers the new capability to its  operators, and maintains the capability until it is retired; this phase  includes sustainment, which begins when a capability has been  fielded for operational use; sustainment involves the supportability of  fielded systems through disposal, including maintenance and the  identification of cost reduction opportunities; this phase tends to  account for up to 70 percent of life-cycle costs.
Figure 1 depicts the acquisition life cycle.
An important aspect of the Acquisition Decision Events is the review and  approval of key acquisition documents critical to establishing the need for  a major program, its operational requirements, an acquisition baseline,  and testing and support plans. AD 102—and the associated DHS  Instruction Manual 102-01-001 and appendixes—provide more detailed  guidance for preparing these documents than DHS’s predecessor policy.  See table 2 for descriptions of the key acquisition documents requiring  department-level approval before a program moves to the next acquisition  phase.
Level 2 programs’ life cycle cost estimates do not require department-level approval.
Chief Financial Officer,
Chief Procurement Officer,
Chief Information Officer,
Chief Human Capital Officer,
Chief Administrative Services Officer,
Chief Security Officer,
CAE responsible for the program being reviewed, and
User representatives from component(s) sponsoring the capability.
The Office of Program Accountability and Risk Management (PARM) is  responsible for DHS’s overall acquisition governance process, supports  the IRB, and reports directly to the USM. PARM, which is led by an  executive director, develops and updates program management policies  and practices, oversees the acquisition workforce, provides support to  program managers, and collects program performance data. In March  2012, PARM issued its first Quarterly Program Accountability Report,  which provided an independent evaluation of major programs’ health and  risks.
The department’s program management offices are responsible for  planning and executing DHS’s individual programs within cost, schedule,  and performance goals. The program managers provide the IRB key  information by preparing required acquisition documents that contain  critical knowledge about their respective programs, facilitating the  governance process. Nearly all of DHS’s program management offices  are located within 12 of the department’s component agencies, such as  the Transportation Security Administration, or U.S. Customs and Border  Protection. Within these components, CAEs are responsible for  establishing acquisition processes and overseeing the execution of their  respective portfolios. Additionally, under AD 102, the USM can delegate  Acquisition Decision Authority to CAEs for programs with life-cycle cost  estimates between $300 million and $1 billion. Figure 2 depicts the  relationship between acquisition managers at the department,  component, and program level.
The Office of Program Analysis and Evaluation (PA&E), within the Office  of the Chief Financial Officer (OCFO), is responsible for advising the  USM, among others, on resource allocation issues. PA&E coordinates  with DHS’s Office of Policy on the department’s long-term strategic  planning efforts, analyzing budget submissions, cost estimates, and  resource constraints. PA&E also oversees the development of the Future  Years Homeland Security Program (FYHSP). DHS is required to submit  the FYHSP to Congress annually with each budget request. The FYHSP  is DHS’s 5-year funding plan for programs approved by the Secretary that  are to support the DHS strategic plan. The FYHSP provides a detailed  account of time-phased resource requirements for each component, as  well as programs’ cost estimates, milestones, and performance  measures.
section_title: Nearly All Program Managers Surveyed Reported Significant Challenges
paragraphs: Nearly all of the program managers we surveyed reported their programs  had experienced significant challenges increasing the risk of poor  outcomes, particularly cost growth and schedule slips. Sixty-eight of the  71 programs that responded to our survey reported that they experienced  funding instability, faced workforce shortfalls, or their planned capabilities  changed after initiation. Most program managers reported a combination  of these challenges, as illustrated in figure 3.
We have previously reported that these challenges increase the likelihood  acquisition programs will cost more and take longer to deliver capabilities  than expected.realistic schedules needed to accurately measure program performance,   Although DHS lacks the reliable cost estimates and  it has submitted some cost information to Congress, and PARM  conducted an internal review of its major acquisition programs in March  2012. We used this information and our survey results to identify 42  programs that experienced cost growth, schedule slips, or both. Cost  information DHS submitted to Congress provides insight into the  magnitude of the cost growth for 16 of the 42 programs. Using this  information, we found total project costs increased from $19.7 billion in  2008 to $52.2 billion in 2011, an aggregate increase of 166 percent. See  figure 4.
We have previously reported that cost growth and schedule slips can lead  to reduced capabilities, decreasing the value provided to the operator—as  well as the value of the resources invested in the programs. This poor  performance threatens the department’s ability to successfully field the  capabilities it is pursuing.
subsections: section_title: Most Major Programs Reported Their Planned Capabilities Changed after Initiation
paragraphs: Prior to entering the Obtain phase, programs are to establish the specific  capabilities they plan to develop to improve DHS’s ability to execute its  mission. Forty-three survey respondents reported that their programs  changed planned capabilities after the initiation of design and  development activities, which occurs between ADE 2B and testing. We  have previously found that both increases and decreases in planned  capabilities are associated with cost growth and schedule slips. We  have found that increasing planned capabilities can lead to cost growth or  schedule slips because programs are more costly to change after they  begin development activities. Alternatively, we have stated that programs  may choose to decrease their planned capabilities in response to cost  growth or schedule slips in an effort to maintain affordability or deliver  certain capabilities when needed. At DHS, we found that more than half  of the 43 programs that reported changing their capabilities had  experienced cost growth or schedule slips, regardless of whether their  planned capabilities increased, decreased, or both. See figure 5.
The 43 survey respondents that reported their planned capabilities  changed identified five key reasons for the changes. Nineteen of the 43  survey respondents reported more than one reason. See figure 6.
Survey respondents identified operator input as the most common reason  for increasing planned capabilities after the initiation of development  efforts, even though officials at the department, component, and program  levels all said operator input at the initiation of design and development is  very useful. For example, in 2011, we reported that the U.S. Citizenship  and Immigration Services’s Transformation program did not fully define its  planned capabilities before it awarded a contract to develop a new  system to enhance the adjudication of applications. After the contract was  awarded, the program office worked with those officials most familiar with  adjudication operations and discovered that the functions were more  complex than expected. As a result, the program office revised the  requirements, and the deployment date for key capabilities slipped from  April 2011 to October 2012.
Alternatively, DHS program managers identified funding availability as the  most common reason for decreasing planned capabilities after the  initiation of development efforts. In the past, we have stated that agencies  may reduce planned capabilities in this manner when their programs  experience cost growth. Decreasing planned capabilities in response to  affordability concerns may be fiscally responsible, but as a result,  operators may not receive the capability originally agreed upon to  address existing capability gaps.
section_title: Most Major Programs Reported They Experienced Funding Instability
paragraphs: DHS is required to establish out-year funding levels for programs annually  in the FYHSP. Changes to planned out-year funding levels create funding  instability, which we have previously found increases the risk of cost  growth, schedule slips, and capability shortfalls. Sixty-one survey  respondents reported that their programs have experienced funding  instability, and we found that 44 of the 61 programs had also realized cost  growth, schedule slips, or capability reductions. Additionally, 29 survey  respondents reported that their programs had to resequence the delivery  of certain capabilities. For example, Coast Guard officials told us they  deferred some of the HH-60 helicopter’s capabilities because of funding  constraints across their portfolio of programs. The Coast Guard delayed  delivery of dedicated radar to search the surface of the water in order to  replace critical components, such as main rotor blades, as planned.  Figure 7 identifies how program managers reported funding instability has  affected their programs.
Forty-five of the 61 survey respondents that reported their programs  experienced funding instability also reported reasons for the funding  instability. Twenty-two survey respondents reported more than one  reason. See figure 8.
Eighteen survey respondents reported that their program experienced a  funding decrease because of another program’s funding needs. We have  previously reported that agencies often change funding levels in this  manner when they commit to more programs than they can afford. A  PA&E official told us that DHS’s resource requirements exceed the  department’s funding levels, and that the department has allowed major  acquisition programs to advance through the acquisition life cycle without  identifying how they will be funded. Furthermore, a PA&E official stated  that DHS has not been able to determine the magnitude of its forthcoming  funding gap because cost estimates are unreliable. The director of the  department’s cost analysis division determined that only 12 major  acquisition programs met most of DHS’s criteria for reliable cost  estimates when it reviewed the components’ fiscal year 2013 budget  submissions. In 2010, we reported that DHS officials had difficulty  managing major programs because they lacked accurate cost  estimates.
Given the fiscal challenges facing the federal government, funding  shortfalls may become an increasingly common challenge at DHS,  leading to further cost growth that widens the gap between resource  requirements and available funding.
section_title: Most Major Programs Reported They Experienced Workforce Shortfalls
paragraphs: DHS acquisition policy establishes that each program office should be  staffed with personnel who have appropriate qualifications and  experience in key disciplines, such as systems engineering, logistics, and  financial management. Fifty-one survey respondents reported that their  programs had experienced workforce shortfalls—specifically a lack of  government personnel—increasing the likelihood their programs will  perform poorly in the future. We have previously reported that a lack of  adequate staff in DHS program offices—both in terms of skill and staffing  levels—increased the risk of insufficient program planning and contractor  oversight, which is often associated with cost growth and schedule slips.Figure 9 below identifies the functional areas where DHS acquisition  programs reported workforce shortfalls.
We found that 29 of the 51 DHS programs that identified workforce  shortfalls had also experienced cost growth or schedule slips.workforce shortfalls have led to insufficient program planning, hindering  the development of key acquisition documents intended to inform senior- level decision making. For example, CAEs and program managers said  that workforce shortfalls limited program management offices’ interaction  with stakeholders and operators, and delayed or degraded test plans and  cost estimates. In addition, a PARM official explained that DHS has had  to rely on contractors to produce cost estimates because of workforce  shortfalls, and the quality of these cost estimates has varied.
The  The USM has stated that properly staffing programs is one of DHS’s  biggest challenges, and we have previously reported that the capacity of  the federal government’s acquisition workforce has not kept pace with  increased spending for increasingly complex purchases. PARM officials  told us that the IRB’s program reviews include assessments of the  program office workforce, but that the IRB considers staffing issues a  relatively low priority, and we found the IRB has formally documented  workforce-related challenges for only 11 programs.
section_title: Programs Proceed without Meeting Sound DHS Acquisition Policy
paragraphs: DHS acquisition policy reflects many key program management  practices.critical knowledge that would help leaders make better informed  investment decisions when managing individual programs. This  knowledge would help DHS mitigate the risks of cost growth and  schedule slips resulting from funding instability, workforce shortfalls, and  planned-capability changes. However, as of April 2012, the department  had only verified that four programs documented all of the critical  knowledge required to progress through the acquisition life cycle. In most  instances, DHS leadership has allowed programs it has reviewed to  proceed with acquisition activities without meeting these requirements.  Officials explained that DHS’s culture has emphasized the need to rapidly  execute missions more than sound acquisition management practices,  and we have found that most of the department’s major programs are at  risk of cost growth and schedule slips as a result. In addition, they lack  the reliable cost estimates, realistic schedules, and agreed-upon baseline  objectives that DHS acknowledges are needed to accurately track  program performance, limiting DHS leadership’s ability to effectively  manage those programs and provide information to Congress. DHS  recognizes the need to implement its acquisition policy more consistently,  but significant work remains.
subsections: section_title: DHS Acquisition Policy Generally Reflects Key Program Management Practices
paragraphs: In 2005, we reported that DHS established an investment review process  that adopted many practices to reduce risk and increase the chances for  successful outcomes. In 2010, we reported that AD 102 provided more  detailed guidance for preparing key acquisition documents than the  department’s predecessor policy. In October 2011, DHS updated the  Guidebook and its appendixes, and we have found that it establishes a  knowledge-based acquisition policy for program management that is  largely consistent with key practices.
A knowledge-based approach to capability development allows  developers to be reasonably certain, at critical points in the acquisition life  cycle, that their products are likely to meet established cost, schedule,  and performance objectives. information needed to make sound investment decisions, and it would  help DHS address the significant challenges we identified across its  acquisition programs: funding instability, workforce shortfalls, and  planned-capability changes. Over the past several years, our work has  emphasized the importance of obtaining key knowledge at critical points  in major system acquisitions and, based on this work, we have identified  eight key practice areas for program management. These key practice  areas are summarized in table 3, along with our assessment of DHS’s  acquisition policy.
In our past work examining weapon acquisition issues and best practices for product  development, we have found that leading commercial firms pursue an acquisition  approach that is anchored in knowledge, whereby high levels of product knowledge are  demonstrated by critical points in the acquisition process. See GAO-11-233SP.
Legend:  DHS policy reflects key practices; ◕ DHS policy substantially reflects key practices;   ◑ DHS policy partially reflects key practices.
We found that DHS’s acquisition policy generally reflects key program- management practices, including some intended to help develop  knowledge at critical points in the acquisition life cycle. Furthermore, the  revised policy the department issued in October 2011 better reflects two  key practice areas by bolstering exit criteria and taking steps to establish  an adequate acquisition workforce. Specifically, the revised Guidebook  and its appendixes require that refined cost estimates be reviewed at  major milestones after the program baseline has been established, and  used to determine whether a program has developed appropriate  knowledge to move forward in the acquisition life cycle. These reviews  can help reduce risk and the potential for unexpected cost and schedule  growth. Additionally, the revised policy establishes that major program  offices should be staffed with personnel with appropriate qualifications  and experience in key acquisition disciplines. We have previously  identified that the magnitude and complexity of the DHS acquisition  portfolio demands a capable and properly trained workforce and that  workforce shortfalls increase the risk of poor acquisition outcomes. The  policy revisions could help mitigate this risk.
However, there are three areas where DHS could further enhance  acquisition oversight:
The policy requires that DHS test technologies and manufacturing  processes, but it does not require that 1) programs demonstrate  technologies in a realistic environment prior to initiating development  activities at the outset of the Obtain phase, or 2) manufacturing  processes be tested prior to production. These practices decrease the  risk that rework will be required, which can lead to additional cost  growth and schedule slips.
The policy requires that DHS establish exit criteria for programs  moving to the next acquisition phase, and standardizes document  requirements across all major programs, but it does not require that 1)  exit criteria be quantifiable to the extent possible, or 2) consistent  information be used across programs when approving progress within  the Obtain phase, specifically at ADE 2B and 2C. These practices  decrease the risk that a program will make an avoidable error  because management lacks information needed to leverage lessons  learned across multiple program reviews.
The policy requires that program managers be certified at an  appropriate level, but it does not state that they should remain with  their programs until the next major milestone when possible. This  practice decreases the risk that program managers will not be held  accountable for their decisions, such as proceeding without reliable  cost estimates or realistic schedules.
PARM officials generally acknowledged DHS has opportunities to  strengthen its program-management guidance. Officials reported that  they are currently in the process of updating AD 102, which they plan to  complete by the end of fiscal year 2012. They also plan to issue revisions  to the associated guidebook and appendixes in phases. PARM officials  told us that they plan to structure the revised acquisition policy by  function, consolidating guidance for financial management, systems  engineering, reporting requirements, and so forth. PARM officials  anticipate that this organization will make it easier for users to identify  relevant information as well as streamline the internal review process for  future updates.
section_title: DHS Has Approved Few Programs’ Key Acquisition Documents
paragraphs: DHS acquisition policy establishes several key program-management  practices through document requirements. AD 102 requires that major  acquisition programs provide the IRB documents demonstrating the  critical knowledge needed to support effective decision making before  progressing through the acquisition life cycle. For example, programs  must document that they have assessed alternatives to select the most  appropriate solution through a formal Analysis of Alternatives report,  which must be approved by component-level leadership. Figure10  identifies acquisition documents that must be approved at the department  level and their corresponding key practice areas.
DHS acquisition policy requires these documents, but the department  generally has not implemented its acquisition policy as intended, and in  practice the department has not adhered to key program management  practices. DHS’s efforts to implement the department’s acquisition policy  have been complicated by the large number of legacy programs initiated  before the department was created, including 11 programs that PARM  officials told us were in sustainment when AD 102 was signed. We  found that the department has only approved four programs’ required  documents in accordance with DHS policy: the National Cybersecurity  and Protection System, the Next Generation Network, the Offshore Patrol  Cutter, and the Passenger Screening Program. Additionally, we found  that 32 programs had none of the required documents approved by the  department. See figure 11.
Since 2008, DHS leadership—through the IRB or its predecessor body  the Acquisition Review Board—has formally reviewed 49 of the 71 major  programs that responded to our survey. It permitted 43 of those programs  to proceed with acquisition activities without verifying the programs had  developed the knowledge required for AD 102’s key acquisition  documents. See figure 12.
Officials from half of the CAE offices we spoke to reported that DHS’s  culture has emphasized the need to rapidly execute missions more than  sound acquisition management practices. PARM officials agreed,  explaining that DHS has permitted programs to advance without  department-approved acquisition documents because DHS had an  operational need for the promised capabilities, but the department could  not approve the documents in a timely manner. PARM officials explained  that, in certain instances, programs were not capable of documenting  knowledge, while in others, PARM lacked the capacity to validate that the  documented knowledge was adequate. In 2008 and 2010, we reported  that several programs were permitted to proceed with acquisition  activities on the condition they complete key action items in the future.However, PARM officials told us that many of these action items were not  addressed in a timely manner. Additionally, program managers reported  that there has been miscommunication between DHS headquarters and  program offices regarding implementation of the acquisition policy, and  we found that DHS headquarters and program managers often had a  different understanding of whether their programs were in compliance  with AD 102. For example, DHS headquarters officials told us that 19 of  the 40 programs that reported through our survey they had department- approved acquisition program baselines (APB) in fact did not.
Because DHS has not generally implemented its acquisition policy, senior  leaders lack the critical knowledge needed to accurately track program  performance: (1) department-approved APBs, (2) reliable cost estimates,  and (3) realistic schedules. Specifically, at the beginning of 2012, DHS  leadership had approved APBs for less than one-third of the 63 programs  we reviewed that are required to have one based on their progression  through the acquisition life cycle. Additionally, we found that none of the  programs with a department-approved APB also met DHS’s criteria for  both reliable cost estimates and realistic schedules, which are key  components of the APB. This raises questions about the quality of those  APBs that have been approved, as well as the value of the DHS review  process in practice. Figure 13 identifies how many programs currently  have department-approved APBs, reliable cost estimates, and realistic  schedules.
The APB is a critical tool for managing an acquisition program. According  to DHS’s acquisition Guidebook, the program baseline is the agreement  between program, component, and department level officials, establishing  how systems will perform, when they will be delivered, and what they will  cost. In practice, when the Acquisition Decision Authority approves a  program’s APB, among other things, it is concurring that the proposed  capability is worth the estimated cost. However, we found that DHS plans  to spend more than $105 billion on programs lacking current, department- approved APBs. Specifically, when DHS submitted the FYHSP to  Congress in 2011, it reported that 34 of the 43 programs lacking  department-approved APBs were expected to cost $108.8 billion over  their acquisition life cycles. DHS did not provide cost estimates for the  other 9 programs because the data were unreliable.
In addition to overall cost, schedule, and performance goals, the APB  also contains intermediate metrics to measure a program’s progress in  achieving those goals. These intermediate metrics allow managers to  take corrective actions earlier in the acquisition life cycle. DHS’s lack of  APBs, PARM officials explained, makes it more difficult to manage  program performance. In March 2012, PARM reported that 32 programs  had experienced significant cost growth or schedule slips in its internal  Quarterly Program Accountability Report. However, DHS has only  formally established that 8 of its programs have fallen short of their cost,  schedule, or performance goals, because approximately three-quarters of  the programs PARM identified lack the current, department-approved  APBs needed to authoritatively measure performance.
To accurately assess a program’s performance, managers need accurate  cost and schedule information. However, DHS acquisition programs  generally do not have reliable cost estimates and realistic schedules, as  required by DHS policy. In June 2012, the department reported to GAO  that its senior leaders lacked confidence in the performance data they  receive, hindering their efforts to manage risk and allocate resources.
GAO-10-588SP. approved APBs. Additionally, only 12 program offices reported that they  fully adhered to DHS’s scheduling guidance, which requires that  programs sequence all activities, examine the effects of any delays,  update schedules to ensure validity, and so forth. Eight of these programs  lacked department-approved APBs.
DHS’s lack of reliable performance data not only hinders its internal  acquisition management efforts, but also limits Congressional oversight.  Congress mandated the department submit the Comprehensive  Acquisition Status Report (CASR) to the Senate and House Committees  on Appropriations as part of the President’s fiscal year 2013 budget,  which was submitted in February 2012. However, DHS told us that it did  not do so until August 2012. Congress mandated DHS produce the CASR  in order to obtain information necessary for in-depth congressional  oversight, including life-cycle cost estimates, schedules, risk ratings, and  out-year funding levels for all major programs. The CASR has the  potential to greatly enhance oversight efforts by establishing a common  understanding of the status of all major programs.
section_title: DHS Recognizes the Need to Implement Its Acquisition Policy More Consistently, but Significant Work Remains
paragraphs: In April 2012, PARM officials told us that DHS had begun to implement its  acquisition policy in a more disciplined manner. They told us that they had  adequate capacity to review programs, and would no longer advance  programs through the acquisition life cycle until DHS leadership verified  the programs had developed critical knowledge. For example, in February  2012, the IRB denied a request from the BioWatch Gen 3 program— which is developing a capability to detect airborne biological agents—to  solicit proposals from contractors because its draft APB was not valid.  PARM officials said they are using a risk-based approach to prioritize the  approval of the department’s APBs. Specifically, they explained that one  of their fiscal year 2011 initiatives was to attain department-level approval  of APBs for all Level 1 programs in the Obtain phase of the acquisition life  cycle. However, we found only 8 of the 19 programs PARM said fell into  this category had current, department-approved APBs as of September  2012.
In an effort to improve the consistency of performance data reported by  program managers, PARM officials stated that they are establishing  scorecards to assess cost estimates and standard work breakdown  structures for IT programs. The PARM officials also explained that CAE’s  performance evaluations now include an assessment of the  completeness and accuracy of performance data reported for their  respective programs. However, DHS must overcome significant  challenges in order to improve the reliability of performance data and  meet key requirements in the department’s acquisition policy. For  example, department and component-level officials told us that program  managers do not report on their programs in a consistent manner.  Additionally, DHS officials told us that they lack cost estimating capacity  throughout the department and that they must rely heavily on contractors,  which do not consistently provide high-quality deliverables. In August  2012, a PARM official stated that DHS was currently in the process of  hiring eight additional government cost estimators to support programs.
section_title: DHS Needs Policy and Process Enhancements to Effectively Manage Its Portfolio of Investments
paragraphs: DHS acquisition policy does not fully reflect several key portfolio- management practices, such as allocating resources strategically, and  DHS has not yet reestablished an oversight board to manage its  investment portfolio across the department. As a result, DHS has largely  made investment decisions on a program-by-program and component-by- component basis. The widespread risk of poorly understood cost growth,  coupled with the fiscal challenges facing the federal government, makes it  essential that DHS allocate resources to its major programs in a  deliberate manner. DHS plans to develop stronger portfolio-management  policies and processes, but until it does so, DHS programs are more likely  to experience additional funding instability in the future, which will  increase the risk of further cost growth and schedule slips. These  outcomes, combined with a tighter budget, could prevent DHS from  developing needed capabilities.
subsections: section_title: DHS Acquisition Policy Shortfalls Hinder Portfolio Management Efforts
paragraphs: In our past work, we have found that successful commercial companies  use a disciplined and integrated approach to prioritize needs and allocate  resources. As a result, they can avoid pursuing more projects than their  resources can support, and better optimize the return on their investment.  This approach, known as portfolio management, requires companies to  view each of their investments as contributing to a collective whole, rather  than as independent and unrelated. With this enterprise perspective,  companies can effectively (1) identify and prioritize opportunities, and (2)  allocate available resources to support the highest priority—or most  promising—opportunities. Over the past several years, we have  examined the practices that private and public sector entities use to  achieve a balanced mix of new projects, and based on this work, we have  identified four key practice areas for portfolio management, summarized  in table 4, along with our assessment of DHS acquisition policy.
We found that DHS’s acquisition policy reflects some key portfolio- management practices. DHS has not designated individual portfolio  managers, but it requires that the department’s Chief Acquisition Officer— currently the USM—be supported by the IRB, which includes officials  representing key functional areas, such as budget, procurement, IT, and  human capital. DHS’s acquisition policy also establishes that  requirements, acquisition, and budget processes should be connected to  promote stability.
However, as acknowledged by DHS officials, the policy does not reflect  several other key portfolio-management practices:
The policy does not empower portfolio managers to decide how best  to invest resources. This practice increases the likelihood resources  will be invested effectively, and that portfolio managers will be held  accountable for outcomes.
The policy does not establish that investments should be ranked and  selected using a disciplined process. This practice increases the  likelihood the portfolio will be balanced with risk spread across  products.
The policy does not establish that (1) resource allocations should  align with strategic goals, or (2) the investment review policy should  use long-range planning. These practices increase the likelihood that  the right amount of funds will be delivered to the right projects,  maximizing return on investments.
The policy does not require portfolio reviews (1) annually to consider  proposed changes, (2) as new opportunities are identified, or (3)  whenever a program breaches its objectives. These practices provide  opportunities for leaders to increase the value of investments,  determine whether or not the investments are still relevant and  affordable, and help keep programs within cost and schedule targets.
PARM officials acknowledge that the department does not currently have  a policy that addresses these key portfolio-management practices.  Further, they told us that there has been less focus on portfolio  management than program management to date because the acquisition  process is still relatively immature. As a result, DHS largely makes  investment decisions on a program-by-program and component-by- component basis. In our work at the Department of Defense, we have  found this approach hinders efforts to achieve a balanced mix of  programs that are affordable and feasible and that provide the greatest  return on investment.
PARM officials anticipate that DHS will improve its portfolio-management  guidance in the future by formalizing its proposed Integrated Investment  Life Cycle Model (IILCM). In January 2011, DHS presented a vision of the  IILCM as a means to better integrate investment management functions,  including requirements development, resource allocation, and program  governance. DHS explained that the IILCM would ensure mission needs  drive investment decisions and establish a common framework for  monitoring and assessing the department’s investments. The IILCM  would be implemented through the creation of several new department- level councils, as illustrated in figure 14, which would identify priorities  and capability gaps.
In 2003, DHS established the Joint Requirements Council (JRC) to  identify crosscutting opportunities and common requirements among DHS  components, and help determine how DHS should use its resources.  However, as we have previously reported, the JRC stopped meeting in  2006. In 2008, we recommended that the JRC be reinstated, or that  DHS establish another joint requirements oversight board. At that time,  DHS officials recognized that strengthening the JRC was a top priority.  The department has proposed the creation of a Capabilities and  Requirements Council (CRC) to serve in a similar role as the JRC, but the  CRC is not yet established.
In the absence of a JRC, or the proposed CRC, DHS budget officials  explained it is difficult to develop a unified strategy to guide trade-offs  between programs because of the diversity of the department’s missions.  Poor program outcomes, coupled with a tighter budget, could prevent  DHS from developing needed capabilities. In our work at the Department  of Defense, we have found that agencies must prioritize investments, or  programs will continually compete for funding by promising more  capabilities than they can deliver while underestimating costs. found that success was measured in terms of keeping a program alive  rather than efficiently delivering the capabilities needed. It appears the  lack of prioritization is affecting DHS in the same way. As discussed  earlier in our assessment of program challenges, 18 of the department’s  programs reported DHS decreased their out-year funding levels because  of another program’s funding needs, and 61 programs reported they  experienced some form of funding instability.
Until recently, the responsibility for balancing portfolios has fallen on  components. However, DHS policy officials noted that component-level  officials have a relatively limited perspective focused on those programs  under their authority, making it more difficult to ensure the alignment of  mission needs to department-level goals. Additionally, component-level  officials can only make trade-offs across the portion of the DHS portfolio  that falls under their purview, limiting opportunities to increase the  department’s return on its investments.
GAO-08-619.
The USM and PARM officials have stated they recognize the value of  portfolio management, and they have taken some steps to fill the gap left  without a functioning JRC or CRC. A PARM official stated that, starting in  2012, PARM is collaborating with the Offices of the Chief Information,  Financial, and Procurement Officers, as well as the Office of Policy, to  conduct portfolio reviews from a functional, cross-component perspective.  In the past, PARM’s portfolio reviews focused on each component  individually. This new functional approach is establishing portfolios based  on departmentwide missions, such as domain awareness or screening,  and PARM officials intend to produce trade-off recommendations for  prioritizing funding across different components. They also intend to use  functional portfolio reviews to provide greater insight into the effects of  funding instability, and the USM has stated that the portfolio reviews will  inform the department’s fiscal year 2014 budget. DHS intends for the  proposed CRC to make trade-offs across the functional portfolios.
PARM’s Quarterly Program Accountability Report (QPAR), issued in  March 2012, also has the potential to inform DHS’s portfolio management  efforts. In developing the QPAR, PARM used a standardized set of five  criteria to measure the value of each program: mission alignment,  architectural maturity, capability gap, mission criticality, and DHS benefit.  This allowed PARM to identify 48 high-value and 13 low-value programs.  However, the QPAR does not recommend using the information to  prioritize resource allocations, which would address a key portfolio  management practice. Further, DHS’s widespread lack of department- approved Mission Need Statements (MNS) undermines efforts to improve  portfolio management and prioritize investments. The MNS links  capability gaps to the acquisitions that will fill those gaps, making it a  critical tool for prioritizing programs. The MNS also provides formal  executive-level acknowledgment that there is a mission need justifying  the allocation of DHS’s limited resources. However, only about 40 percent  of DHS’s major acquisition programs have a department-approved MNS.
section_title: DHS Acquisition Management Initiatives Target Longstanding Challenges, but Key Implementation Issues Remain
paragraphs: DHS has introduced seven initiatives that could improve acquisition  management by addressing longstanding challenges we have identified— such as funding instability and acquisition workforce shortfalls—which  DHS survey respondents also identified in 2012. Implementation plans  are still being developed for all of these initiatives, and DHS is still  working to address critical issues, particularly capacity questions.  Because of this, it is too early to determine whether the DHS initiatives  will be effective, as we have previously established that agencies must  sustain progress over time to address management challenges. DHS is  also pursuing a tiered-governance structure that it has begun to  implement for IT acquisitions. Before the department can regularly  delegate ADE decision authority through this tiered-governance structure,  DHS must successfully implement its seven acquisition management  initiatives and apply its knowledge-based acquisition policy on a more  consistent basis to reduce risks and improve program outcomes.
subsections: section_title: DHS Initiatives Are Intended to Address Longstanding Acquisition Management Challenges
paragraphs: In 2005, we identified acquisition management as a high-risk area at  DHS. Since then, we have issued multiple reports identifying acquisition  management challenges. In 2008, we made several recommendations  intended to help DHS address those challenges, and in September 2010,  we provided DHS a list of specific acquisition management outcomes the  department must achieve to help address the high-risk designation. This  list largely drew from our past recommendations, and stressed that the  department must implement its knowledge-based acquisition policy  consistently. DHS has generally concurred with our recommendations,  but still faces many of the same challenges we have previously identified.
In 2011, DHS began to develop initiatives to address these challenges,  and DHS has continued to evolve these plans in 2012. In January 2011,  DHS produced the initial iteration of its Integrated Strategy for High Risk  Management in order to measure progress in addressing acquisition  management challenges we had identified, as well as financial  management, human capital, IT, and management integration issues.  The department subsequently produced updates in June 2011, December  2011, and June 2012. These updates present the department’s progress  in developing and implementing its initiatives. Additionally, in December  2011, DHS issued the Program Management and Execution Playbook  (Playbook), which expounded on some of those initiatives, and introduced  a vision for a “more mature, agile, and effective process for program  governance and execution.” Figure 15 identifies seven key DHS initiatives  and how they correspond to acquisition management challenges we have  identified.
As envisioned, the DHS initiatives would better position the department to  implement its knowledge-based acquisition policy on a more consistent  basis to reduce risks and ultimately improve individual program  outcomes. The initiatives would also help address challenges identified by  survey respondents in 2012, particularly funding instability and acquisition  workforce shortfalls. Additionally, the IILCM would enhance DHS’s ability  to effectively manage its acquisition portfolio as a whole.
section_title: Some Initiatives Are Taking Longer to Implement Than Originally Envisioned, and Capacity Issues Could Create Further Challenges
paragraphs: DHS has made progress implementing some of the initiatives intended to  address the challenges we have identified. In June 2012, DHS reported  that all of its components had an approved CAE in place and the  Procurement Staffing Model had been completed. In August 2012, DHS  told us that eight Centers of Excellence had been chartered. However,  from January 2011 to June 2012, the schedules for four of the seven  initiatives slipped by at least 6 months, including the schedule for the  IILCM, which slipped by a year. In March 2012, an official responsible for  the IILCM initiative stated that many acquisition officials throughout the  department do not yet understand the intended benefits of the IILCM.  Thirty-two survey respondents reported that they were not at all familiar  with the initiative, as opposed to nine that reported they were very familiar  with the IILCM. Additionally, officials from three CAE offices, including  two CAEs, told us that they were not familiar with the IILCM. Previously,  we have reported that it is important to involve employees and obtain their  ownership when transforming organizations.schedule slips and their causes.
section_title: Conclusions
paragraphs: DHS has a diverse, critical, and challenging mission that requires it to  respond to an ever-evolving range of threats. Given this mission, it is  important that DHS maintain an agile and flexible management approach  in its day-to-day operations. However, DHS must adopt a more  disciplined and systematic approach for managing its major investments,  which are intended to help meet critical mission needs. DHS has taken  some steps to improve investment management, but most of its major  acquisition programs continue to cost more than expected, take longer to  deploy than planned, or deliver less capability than promised. These  outcomes are largely the result of DHS’s lack of adherence to key  knowledge-based program management practices, even though many  are reflected in the department’s own acquisition policy. DHS leadership  has authorized and continued to invest in major acquisition programs  even though the vast majority of those programs lack foundational  documents demonstrating the knowledge needed to help manage risks  and measure performance. This limits DHS’s ability to proactively identify  and address the challenges facing individual programs. Further, although  the department’s acquisition policy contains many key practices that help  reduce risks and increase the chances for successful outcomes, the  policy does not include certain program management practices that could  further enhance acquisition management. For example, the policy does  not require that programs demonstrate technologies in a realistic  environment prior to initiating development activities, or that exit criteria  be quantifiable to the extent possible.
Cost growth and schedule slips at the individual program level complicate  DHS’s efforts to manage its investment portfolio as a whole. When  programs encounter setbacks, the department has often redirected  funding to troubled programs at the expense of others, which in turn are  more likely to struggle. Additionally, DHS acquisition policy does not fully  reflect key portfolio-management practices that would help improve  investment management across the department. For example, the policy  does not empower portfolio managers to invest resources in a disciplined  manner or establish that investments should be ranked and selected  using a disciplined process. DHS acknowledges the importance of having  strong portfolio-management practices. However, DHS does not have a  process to systematically prioritize its major investments to ensure that  the department’s acquisition portfolio is consistent with DHS’s anticipated  resource constraints, which is particularly important because of the  diversity of the department’s missions. Since 2008, we have emphasized  the need for DHS to re-establish an oversight board dedicated to  addressing portfolio management challenges. DHS has produced plans to  establish such a board, but the concept is still under development.
It is essential that DHS take a more disciplined acquisition management  approach moving forward, particularly as the department must adjust to a  period of governmentwide funding constraints. Without greater discipline,  decisionmakers will continue to lack critical information and the  department will likely continue to pay more than expected for less  capability than promised, which will ultimately hinder DHS’s day-to-day  operations and its ability to execute its mission. Further, Congress’s  ability to assess DHS funding requests and conduct oversight will remain  limited. To its credit, DHS has undertaken a variety of initiatives over the  past two years designed to address the department’s longstanding  acquisition management challenges, such as increasing acquisition  management capabilities at the component-level. However, more  disciplined program and portfolio management at the department-level is  needed before DHS can regularly delegate major milestone decision  authority to component-level officials. Widespread challenges—including  funding instability and acquisition workforce shortfalls—cost growth, and  schedule slips indicate how much further DHS must go to improve  acquisition outcomes.
section_title: Recommendations for Executive Action
paragraphs: We recommend that the Secretary of Homeland Security direct the Under  Secretary for Management to take the following five actions to help  mitigate the risk of poor acquisition outcomes and strengthen the  department’s investment management activities:
Modify DHS acquisition policy to more fully reflect the following  program management practices:
Require that (1) programs demonstrate technologies in a realistic  environment prior to initiating development activities, and (2)  manufacturing processes be tested prior to production;
Require that (1) exit criteria be quantifiable to the extent possible,  and (2) consistent information be used across programs at ADE  2B and 2C;
State that program managers should remain with their programs  until the next major milestone when possible;
Modify DHS acquisition policy to more fully reflect the following  portfolio management practices:
Empower portfolio managers to decide how best to invest
Establish that investments should be ranked and selected using a
Establish that (1) resource allocations should align with strategic  goals, and (2) the investment review policy should use long-range  planning; and
Require portfolio reviews (1) annually to consider proposed  changes, (2) as new opportunities are identified, and (3) whenever  a program breaches its objectives;
Ensure all major acquisition programs fully comply with DHS  acquisition policy by obtaining department-level approval for key  acquisition documents before approving their movement through the  acquisition life cycle;
Once the department’s acquisition programs comply with DHS  acquisition policy, prioritize major acquisition programs  departmentwide and ensure that the department’s acquisition portfolio  is consistent with DHS’s anticipated resource constraints; and
Clearly document that department-level officials should not delegate  ADE decision authority to component-level officials for programs  lacking department approved APBs or not meeting agreed-upon cost,  schedule, and performance thresholds.
section_title: Agency Comments and Our Evaluation
paragraphs: DHS provided us with written comments on a draft of this report. In its  comments, DHS concurred with all five of our recommendations and  noted that two should be closed based on actions taken. The  department’s written comments are reprinted in appendix V. DHS also  provided technical comments that we incorporated into the report as  appropriate.
DHS identified specific actions the department would take to address  three of our recommendations. DHS stated that it was in the process of  revising its policy to more fully reflect key program management  practices. Additionally, DHS stated that it would continue to mature and  solidify the portfolio review process over the next few years, and that it  would revise its policy to reflect this process. DHS anticipates that this  effort will also help the department prioritize its major acquisition  programs departmentwide, and help ensure that the department’s  acquisition portfolio is consistent with anticipated resource constraints.
DHS concurred with and requested we close our recommendation that  the department ensure all acquisition programs fully comply with DHS  acquisition policy by obtaining department-level approval for key  acquisition documents before approving their movement through the  acquisition life cycle. DHS stated that, in effect, its executive review board  is approving a program’s documents when it advances the program, thus  satisfying this recommendation. As we noted in our report, DHS officials  told us in April 2012 that the department has begun to implement its  acquisition policy in a more disciplined manner and that it will no longer  advance programs through the acquisition life cycle until DHS leadership  verifies the programs have developed critical knowledge. However, it  would be premature to close this recommendation until DHS  demonstrates, over time, the consistent verification of the critical  knowledge captured in key documents, especially as we found that nearly  all of the department’s major acquisition programs lack at least some of  these acquisition documents.
DHS also concurred with and requested we close our recommendation  that the department clearly document that department-level officials  should not delegate ADE decision authority to component-level officials  for programs lacking department approved APBs or not meeting agreed- upon cost, schedule, and performance thresholds. DHS stated that it  amended AD 102 to clarify that decision authority for any program that  breaches an approved APB’s cost, schedule or performance parameters  will not be delegated to component-level officials, thus satisfying this  recommendation. However, the amendment DHS provided does not  include this language or clearly document the department’s stated  position. For this reason, it would be premature to close this  recommendation at this time.
In addition to commenting on our recommendations, the department  made a number of observations on our draft report. For example, DHS  stated that the report references many practices that occurred prior to the  time period of the audit, and that the department has made measurable  progress on a number of fronts. While we reviewed investment  management activities going back to November 2008 to coincide with the  issuance of AD 102, we also accounted for progress made through  August 2012 by assessing ongoing DHS initiatives intended to address  investment management challenges in the future.
DHS also noted that our survey of 71 programs captured valuable  information, but suggested the survey data cannot be generalized and  expressed concern that it would be used as the basis for a  recommendation. To clarify, none of the recommendations in this report  are based on the survey data. In the absence of reliable program data,  we surveyed program managers to obtain their perspectives on  challenges facing the department’s acquisition programs, and we  obtained responses from 92 percent of the major acquisition programs  DHS identified in 2011. DHS noted that programs can experience cost  growth and schedule slips without a “breach.” We recognize the validity of  this point and our findings are consistent with this position.
DHS incorrectly suggested that our data sources for quantifying cost  growth – the Future Years Homeland Security Programs (FYHSP) issued  in 2008 and 2011 – did not consistently account for costs beyond the  initial five-year period. However, these two FYHSPs aggregated funding  levels for each program to produce a total project cost. To measure total  project cost growth for the 16 programs, as depicted in figure 4, we  compared the total project costs reported in the 2008 FYHSP to the total  project costs reported in the 2011 FYHSP. Thus, we measured changes  in total project costs, not just costs over two different five-year periods.
As agreed with your offices, unless you publicly announce the contents of  this report earlier, we plan no further distribution until September 19,  2012. At that time, we will send copies to the Secretary of Homeland  Security. In addition, the report will be available at no charge on the GAO  website at http://www.gao.gov.
If you or your staff have any questions about this report, please contact  me at (202) 512-4841 or huttonj@gao.gov. Contact points for our Offices  of Congressional Relations and Public Affairs may be found on the last  page of this report. GAO staff who made key contributions to this report  are listed in appendix VI.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: The objectives of this review were to assess the Department of Homeland  Security’s (DHS) acquisition management activities. Specifically, we  assessed the extent to which: (1) DHS’s major acquisition programs face  challenges that increase the risk of poor outcomes; (2) DHS has policies  and processes in place to effectively manage individual acquisition  programs; (3) DHS has policies and processes in place to effectively  manage its portfolio of acquisition programs as a whole; and (4) DHS has  taken actions to resolve the high-risk acquisition management issues we  have identified in previous reports. To answer these questions, we  reviewed 77 of the 82 programs DHS included in its fiscal year 2011  Major Acquisition Oversight List (MAOL), which identified each program  the department designated a major acquisition in 2011. We excluded 5  programs that were canceled in 2011; these are identified in appendix IV.  The 77 selected programs were sponsored by 12 different components  and departmental offices.
To determine the extent to which major DHS acquisition programs face  challenges increasing the risk of poor outcomes, we surveyed the  program managers for all 77 programs, and received usable responses  from 71 programs (92 percent response rate). Appendix III presents the  survey questions we asked, and summarizes the responses we received.  The web-based survey was administered from January 12, 2012, to  March 30, 2012. Respondents were sent an e-mail invitation to complete  the survey on a GAO web server using a unique username and  password. During the data collection period, nonrespondents received a  reminder e-mail and phone call. Because this was not a sample survey, it  has no sampling errors. The practical difficulties of conducting any survey  may also introduce nonsampling errors, such as difficulties interpreting a  particular question, which can introduce unwanted variability into the  survey results. We took steps to minimize nonsampling errors by  pretesting the questionnaire in person with program management officials  for five different programs, each in a different component. We conducted  pretests to make sure that the questions were clear and unbiased, the  data and information were readily obtainable, and that the questionnaire  did not place an undue burden on respondents. Additionally, a senior  methodologist within GAO independently reviewed a draft of the  questionnaire prior to its administration. We made appropriate revisions to  the content and format of the questionnaire after the pretests and  independent review. All data analysis programs used to generate survey  results were independently verified for accuracy.
To determine the extent to which major DHS acquisition programs face  challenges increasing the risk of poor outcomes, we also reviewed the  2008 and 2011 versions of the Future Years Homeland Security Program  (FYHSP), all acquisition decision memoranda documenting DHS  executive review board decisions from November 2008 to April 2012, the  Office of Program Accountability and Risk Management’s (PARM) initial  Quarterly Program Assessment Report (QPAR), issued March 2012, and  other management memos identifying available program-performance  data. The survey results and documentation review allowed us to identify  program performance, and the reasons for any poor performance. We  also interviewed individuals at the component and department-level to  enhance our understanding of common challenges. At the component  level, we interviewed six of the eight Component Acquisition Executives  that had been designated by the USM, and interviewed representatives of  the remaining two. At the department level, we interviewed policy, budget,  and acquisition oversight officials, including the Deputy Assistant  Secretary for the Office of Strategic Plans, the department’s Chief  Information Officer, the Executive Director of PARM, and the Director of  Program Analysis and Evaluation (PA&E). These officials provided a  strategic perspective on program management challenges, and shared  valuable insights regarding the limitations of available program  performance data. Based on their input, we chose to use FYHSP data to  calculate cost growth for individual programs where possible because the  document is provided to Congress and constitutes DHS’s most  authoritative, out-year funding plan.
To determine the extent to which DHS policies and processes are in  place to effectively manage individual acquisition programs, as well as the  department’s acquisition portfolio as a whole, we identified key acquisition  management practices and assessed the extent to which DHS policies  and processes reflected those practices. We identified the key practices  through a review of previous GAO reports, which are listed in appendix II.  We compared DHS Acquisition Directive 102-01 (AD 102), an associated  guidebook—DHS Instruction Manual 102-01-001—and the guidebook’s  12 appendixes to those key practices, and identified the extent to which  they were reflected in the department’s acquisition policy using a basic  scoring system. If the DHS policy reflected a particular key practice, we  assigned the policy a score of 5 for that practice. If the policy did not  reflect the key practice, we assigned it a score of 1. We then took the  average score for all the key practices in a particular area—as identified  in appendix II—to establish an overall score for each key practice area.  We concluded that key practice areas that scored a 5 were reflected in  the policy, scored a 4 were substantially reflected, scored a 3 were  partially reflected, and scored a 2 were minimally reflected. We  subsequently met with PARM officials to discuss our analysis, identify  relevant sections of the policy that we had not yet accounted for, and  solicit their thoughts on those key practices that were not reflected in the  policy. In order to assess DHS’s processes for implementing its policy, we  surveyed program managers, and interviewed component and  department-level officials. We also reviewed DHS’s plans for the  Integrated Investment Life Cycle Model (IILCM), which is being designed  to better integrate the department’s investment management functions.  Further, we reviewed all acquisition decision memoranda documenting  DHS executive review board decisions from November 2008 to April  2012, the March 2012 QPAR, and other management memos identifying  available program-performance data, and any limitations of that data.
To determine the extent to which DHS has taken actions to resolve the  high-risk acquisition management issues we have identified in previous  reports and this audit, we reviewed the first three versions of the DHS  Integrated Strategy for High Risk Management—issued in January, June,  and December 2011. We also reviewed the DHS Program Management  and Execution Playbook, issued in December 2011. We identified  initiatives intended to improve acquisition management, the department’s  progress in implementing those initiatives, and enduring challenges  confronting the department. We also surveyed program managers, and  interviewed component and department-level officials to obtain their  perspectives on the initiatives.
We conducted this performance audit from August 2011 to September  2012 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
section_title: Appendix II: Key Acquisition Management Practices
paragraphs: To determine the extent to which the Department of Homeland Security  (DHS) has policies and processes in place to effectively manage  individual acquisition programs, and the department’s acquisition portfolio  as a whole, we identified key acquisition management practices  established in our previous reports examining DHS, the Department of  Defense, NASA, and private sector organizations. The specific program-  and portfolio-management practices, as well as the reports where we  previously identified the value of those practices, are presented below.
subsections: section_title: Key Program Management Practices
section_title: Key Portfolio Management Practices
paragraphs: The following list identifies several key practices that can improve  outcomes when managing a portfolio of multiple programs.
section_title: Previous Reports Establishing Key Acquisition Management Practices
paragraphs: Information Technology: Critical Factors Underlying Successful Major  Acquisitions. GAO-12-7. Washington, D.C.: October 21, 2011.
Acquisition Planning: Opportunities to Build Strong Foundations for Better  Services Contracts. GAO-11-672. Washington, D.C.: August 9, 2011.
NASA: Assessments of Selected Large-Scale Projects. GAO-11-239SP.  Washington, D.C.: March 3, 2011.
Defense Acquisitions: Assessments of Selected Weapon Programs.  GAO-11-233SP. Washington, D.C.: March 29, 2011.
Defense Acquisitions: Strong Leadership Is Key to Planning and  Executing Stable Weapon Programs. GAO-10-522. Washington, D.C.:  May 6, 2010.
Defense Acquisitions: Assessments of Selected Weapon Programs.  GAO-10-388SP. Washington, D.C.: March 30, 2010.
Defense Acquisitions: Many Analyses of Alternatives Have Not Provided  a Robust Assessment of Weapon Systems Options. GAO-09-665.  Washington, D.C.: September 24, 2009.
Department of Homeland Security: Billions Invested in Major Programs  Lack Appropriate Oversight. GAO-09-29. Washington, D.C.: November  18, 2008.
GAO Cost Estimating and Assessment Guide: Best Practices for  Developing and Managing Capital Program Costs. GAO-09-3SP.  Washington, D.C.: March 2009.
Defense Acquisitions: Sound Business Case Needed to Implement  Missile Defense Agency’s Targets Program. GAO-08-1113. Washington,  D.C.: September 26, 2008.
Defense Acquisitions: A Knowledge-Based Funding Approach Could  Improve Major Weapon System Program Outcomes. GAO-08-619.  Washington, D.C.: July 2, 2008.
Defense Acquisitions: Realistic Business Cases Needed to Execute Navy  Shipbuilding Programs. GAO-07-943T. Washington, D.C.: July 24, 2007.
Best Practices: An Integrated Portfolio Management Approach to Weapon  System Investments Could Improve DOD’s Acquisition Outcomes.   GAO-07-388. Washington, D.C.: March 30, 2007.
Best Practices: Better Support of Weapon System Program Managers  Needed to Improve Outcomes. GAO-06-110. Washington, D.C.:  November 30, 2005.
NASA’s Space Vision: Business Case for Prometheus 1 Needed to  Ensure Requirements Match Available Resources. GAO-05-242.  Washington, D.C.: February 28, 2005.
Information Technology Investment Management: A Framework for  Assessing and Improving Process Maturity. GAO-04-394G. Washington,  D.C.: March 2004.
Executive Guide: Leading Practices in Capital Decision-Making.  GAO/AIMD-99-32. Washington, D.C.: December 1998.
section_title: Appendix III: Program Office Survey Results
paragraphs: To help determine the extent to which major Department of Homeland  Security (DHS) acquisition programs face challenges increasing the risk  of poor outcomes, we surveyed the program managers for all 77  programs, and received usable responses from 71 programs (92 percent  response rate). The web-based survey was administered from January  12, 2012, to March 30, 2012. We present the survey questions we asked  and summarize the responses we received below.
subsections: section_title: Section I: Program Profile
subsections: section_title: Mean 5.6
paragraphs: Number  of  respondents  28  3. In what phase(s) of the DHS Acquisition Directive (AD) 102 acquisition lifecycle is your program currently? (select all that  apply)  1. Need (Prior to Acquisition Decision Event (ADE) 1)  2. Analyze/Select (Between ADE 1 and ADE 2)  4. Production/deploy/support (Post ADE 3)
section_title: Section II: DHS Acquisition Guidance, AD 102
subsections: section_title: 4. How often, if at all, do you refer to each of the following resources to help you manage your acquisition program?
paragraphs: I do not refer  to this source  4  DHS Acquisition Program Management Division (APMD)/ Program Analysis and Risk Management (PARM)
Number of  respondents  26  5. Which of the following DHS-provided opportunities, if any, has your program management office used to understand DHS  acquisition guidance, AD 102; and if used, how useful was the opportunity, if at all?  1. Check box at left if your program is not required to follow AD 102, and then click here to skip to question 13  Training session(s) on AD 102 hosted by DHS headquarters  Did your program management office use this opportunity?
If used, how useful was the opportunity, if at all?
Not at all  useful  1  Manuals and templates for implementing AD 102 provided by DHS headquarters  Did your program management office use this opportunity?
If used, how useful was the opportunity, if at all?
Number of  respondents  50  Direct support for your program from DHS Acquisition Program Management Division (APMD)/ Program Analysis and Risk  Management (PARM)
Did your program management office use this opportunity?
If used, how useful was the opportunity, if at all?
Not at all  useful  2  Did your program management office use this opportunity?
If used, how useful was the opportunity, if at all?
Number of  respondents  16  6. How clear or unclear is the DHS AD 102 acquisition guidance and framework for managing the following types of  acquisitions?
Neither clear  nor unclear  6  unclear  Very unclear  4 4  7. How clear or unclear is DHS AD 102 acquisition guidance, including the guidebook and appendices, regarding each of the  following?
Number of  respondents  22  8. How clear or unclear is DHS AD 102 acquisition guidance, including the guidebook and appendices, on how to develop  each of the following key acquisition documents?
Operational Requirements Document (ORD)
Test and Evaluation Master Plan (TEMP)
Number of  respondents  64  9. How long is the average component and DHS review period for key acquisition documents required by AD 102 (e.g. MNS,  ORD, LCCE and APB)?
More than 1  year  5  10. After an Acquisition Review Board (ARB) review, how adequately does an Acquisition Decision Memo (ADM)  communicate action items?
No opinion   Not applicable  5 6  11. How has the introduction of AD 102 helped or hindered your ability to manage your program's cost and schedule and the  overall acquisition program?
Number of  respondents  65  12. If you would like to elaborate on any of your previous responses regarding the clarity and/or implementation of DHS  acquisition guidance (AD 102) please use the following space.
13. Does your program have a DHS-approved Acquisition Program Baseline (APB)?
No (please  explain  below)  31  13a. If your program does not have a DHS-approved APB, please explain why it does not have one in the box below.  14. How does your program's current projected cost compare against its DHS-approved APB?
Cost exceeds  the APB by 8  percent or  15. How does your program's current projected schedule compare to its DHS-approved APB?
Schedule is  ahead  of the APB  3  16. How do your program's current planned system capabilities compare to its DHS-approved APB?
Number of  respondents  39  17. How frequently, if at all, does your program management office use each of the following performance metrics to monitor  your program's progress?
section_title: Section V: Requirements
subsections: section_title: Yes, approved
paragraphs: Submitted to  DHS  leadership,  not yet  approved  6  Operational Requirements Document (ORD)
Test and Evaluation Master Plan (TEMP)
Submitted to  DHS  leadership,  not yet  approved  7  Submitted to  DHS  leadership,  not yet  approved  9  22. When setting operational requirements, which of the following processes best describes your program's efforts to  consider alternatives at the program level?
No AOA or  trade-off  analysis was  been set.
Neither clear  nor unclear  2  unclear  Very unclear  1 3  26. Which of the following are reasons your program's KPPs have changed or been redefined since development activities  began (ADE 2A)?
Associated capabilities were determined unnecessary  Not a reason  Do not know   2 14  To demonstrate traceability between MNS, ORD, and TEMP  Not a reason  Do not know   3 11  Not a reason  Do not know   1 1  27. Since your program's design and development activities began (ADE 2A), how have each of the following factors affected  your planned capabilities, if at all?
section_title: Section VI: Technology Maturity
subsections: section_title: All critical technologies demonstrated full functionality 25
paragraphs: Number of  respondents  66  29. Prior to the initiation of low-rate initial production (ADE 2C), how many reliability goals were met, if any, by production- representative prototypes demonstrated in the intended environment?
Not  applicable,  the program  will not use  low-rate initial  production  21  30. Has the program used an independent testing authority?
section_title: Section VII: Resource Allocation
subsections: section_title: Yes 21
paragraphs: No   Do not know  0  49  32. Does your program use a five-year funding plan to project resource needs?
No   Do not know  0 3  33. Did your program's funding levels in each of the following budget documents meet your program's required funding  needs as reflected in your APB?  1. Check box at left if your program does not have an APB, and then click here to skip to question 34  Number of  respondents  71  At program start, component's commitment in a Resource Allocation Plan (RAP)
No, funds in  the document  were below  the APB  Do not know   Not applicable  4 6 5  At program start, DHS's commitment in a Resource Allocation Decision (RAD)
Number of  respondents  17  35. If your program has experienced funding instability (e.g. a change in planned out-year funding from one five-year funding  plan to the next five-year funding plan), did it affect your program in each of the following ways?
Number of  respondents  22  36. If a gap existed between FY11 enacted funding and FY11 required funding, how effectively were you, as a program  manager, able to directly communicate the impact on your program to DHS and component leadership?  1. Check box at left if your program did not experience a gap between FY11 enacted and required funding, and then click here to skip  to question 37  Component leadership (Component head, CAE, etc.)
Somewhat  effectively   Not effectively  1  DHS leadership (Deputy Secretary, USM, PARM officials, etc.)
Somewhat  effectively   Not effectively  1 3  37. If you would like to elaborate on how resource allocation (i.e. funding) has affected the program's ability to achieve cost,  schedule and performance goals, please use the following space.  38. Since the program was initially staffed, how many program managers have overseen the program management office  (PMO)?
Number of  respondents  71  39. What is the number of government FTEs in your PMO for each of the following functional areas?
Number of government FTEs staffed at initiation of development activities  Number of government FTEs currently staffed  Number of government FTEs currently identified as a need by the program  Business functions (includes auditing, business, cost estimating, financial management, property management, and purchasing)
Number of  respondents  47  Engineering and technical (includes systems planning, research, development and engineering; life cycle logistics; test and evaluation;  production, quality and manufacturing; and facilities engineering)
section_title: Section IX: Communication to DHS
subsections: section_title: 42. How effective, if at all, are each of the following tools for communicating your program's performance to DHS leadership?
paragraphs: Next-generation Periodic Reporting System (nPRS)
section_title: Section X: DHS Acquisition Management Initiatives
subsections: section_title: 46. How familiar are you with each of the following DHS initiatives?
paragraphs: Establishing the Integrated Investment Life Cycle Model (IILCM)
Not at all  familiar  31  Empowering the Component Acquisition Executives (CAEs)
Not at all  familiar  18  Establishing Functional Coordination Office (e.g. Screening Coordination Office)
Number of  respondents  67  Developing APEX, a decision support tool owned by PARM to capture and synthesize information from nPRS and IMS  Not at all  familiar  37  47. How helpful, if at all, will the following DHS initiatives be in helping you manage your acquisition program?
Establishing the Integrated Investment Life Cycle Model (IILCM)
Not at all  helpful  6  Empowering the Component Acquisition Executives (CAEs)
Not at all  helpful  7  Establishing Functional Coordination Office (e.g. Screening Coordination Office)
Creating Executive Steering Councils for program governance  Not at all  helpful  8  Forming the Capabilities and Requirements Council  Not at all  helpful  8  Developing APEX, a decision support tool owned by PARM to capture and synthesize information from nPRS and IMS  Not at all  helpful  9  48. Please use the following space to describe any additional actions that DHS could implement that would help you better  manage your acquisition program (i.e. improvements for acquisition governance and document development).
49. Please identify any significant challenges affecting your program's ability to achieve program objectives (i.e. cost,  schedule, and capabilities) that have not been adequately addressed above.  50. If you would like, please identify any practices your program has found significantly helpful in managing your program.
Table 6 below identifies the 71 major Department of Homeland Security  (DHS) acquisition programs that responded to our survey. It consists of all  the programs DHS included in its 2011 Major Acquisition Oversight List,  with the exception of the 6 programs that did not respond to our survey  (see table 7), and the 5 programs that were cancelled in 2011 (see table  8). Table 6 also identifies whether each program’s Mission Need  Statement (MNS), Operational Requirements Document (ORD),  Acquisition Program Baseline (APB), Integrated Logistics Support Plan  (ILSP), and Test and Evaluation Master Plan (TEMP) have been  approved at the department level.
Table 7 identifies the programs that were included in DHS’s 2011 Major  Acquisition Oversight List, but did not respond to our survey.
Table 8 identifies the programs that were included in DHS’s 2011 Major  Acquisition Oversight List, but were cancelled in 2011.
section_title: Appendix V: Comments from the Department of Homeland Security
section_title: Appendix VI: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Katherine Trimble (Assistant  Director), Nathan Tranquilli (Analyst-in-Charge), John Crawford, David  Garcia, Jill Lacey, Sylvia Schatz, Rebecca Wilson, Candice Wright, and  Andrea Yohe made key contributions to this report.
section_title: Related GAO Products
paragraphs: Immigration Benefits: Consistent Adherence to DHS’s Acquisition Policy  Could Help Improve Transformation Program Outcomes. GAO-12-66.  Washington, D.C.: November 22, 2011.
Coast Guard: Action Needed As Approved Deepwater Program Remains  Unachievable. GAO-11-743. Washington, D.C.: July 28, 2011.
Opportunities to Reduce Potential Duplication in Government Programs,  Save Tax Dollars, and Enhance Revenue. GAO-11-318SP. Washington,  D.C.: March 1, 2011.
High-Risk Series: An Update. GAO-11-278. Washington, D.C.: February  2011.
Defense Acquisitions: Assessments of Selected Weapon Programs.  GAO-11-233SP. Washington, D.C.: March 29, 2011.
Secure Border Initiative: Controls over Contractor Payment for the  Technology Component Need Improvement. GAO-11-68. Washington,  D.C.: May 25, 2011.
Department of Homeland Security: Assessments of Selected Complex  Acquisitions. GAO-10-588SP. Washington, D.C.: June 30, 2010.
The Office of Management and Budget’s Acquisition Workforce  Development Strategic Plan for Civilian Agencies. GAO-10-459R.  Washington, D.C.: April 23, 2010.
Defense Acquisitions: Measuring the Value of DOD’s Weapon Programs  Requires Starting with Realistic Baselines. GAO-09-543T. Washington,  D.C.: April 1, 2009.
Department of Homeland Security: Billions Invested in Major Programs  Lack Appropriate Oversight. GAO-09-29. Washington, D.C.: November  18, 2008.
Homeland Security: Challenges in Creating an Effective Acquisition  Organization. GAO-06-1012T. Washington, D.C.: July 27, 2006.
Homeland Security: Successes and Challenges in DHS’s Efforts to  Create an Effective Acquisition Organization. GAO-05-179. Washington,  D.C.: March 29, 2005.
Results-Oriented Cultures: Implementation Steps to Assist Mergers and  Organizational Transformations. GAO-03-669. Washington, D.C.: July 2,  2003  Best Practices: Better Matching of Needs and Resources Will Lead to  Better Weapon System Outcomes. GAO-01-288. Washington, D.C.:  March 8, 2001.