Homeland Security: Despite Progress, DHS Continues to Be Challenged in Managing Its Multi-Billion Dollar Annual Investment in Large-Scale Information Technology Systems
Released: 2009-09-15T00:00:00

Letter

Background

DHS’s mission is to lead the unified national effort to secure America by preventing and deterring terrorist attacks and protecting against and responding to threats and hazards to the nation. DHS also is to ensure safe and secure borders, welcome lawful immigrants and visitors, and promote the free flow of commerce.

Created in 2003, DHS assumed control of about 209,000 civilian and military positions from 22 agencies and offices specializing in one or more aspects of homeland security. The intent behind the merger creating DHS and expected transformation was to improve coordination, communication, and information sharing among the multiple federal agencies responsible for protecting the homeland. Not since the creation of the Department of Defense in 1947 has the federal government undertaken a transformation of this magnitude. As we reported before the department was created, such a transformation is critically important and poses significant management and leadership challenges. For these reasons, we designated the implementation of the department and its transformation as high-risk in 2003, and we continue to do so today. In this regard, we have stated that failure to effectively address DHS’s management challenges and program risks could have serious consequences for our national security.

Among DHS’s transformation challenges, we highlighted the formidable hurdle of managing the acquisition and integration of numerous mission- critical and mission support systems and associated IT infrastructure. For the department to overcome this hurdle, we emphasized the need for DHS to establish an effective IT governance framework, including controls aimed at effectively managing system acquisition and IT-related people, processes and tools.

DHS Components and IT Spending

To accomplish its mission, the department is organized into various components, each of which is responsible for specific homeland security missions and for coordinating related efforts with its sibling components, as well as external entities. Figure 1 shows DHS’s organizational structure; table 1 shows DHS’s principal organizations and their missions.

Within the Management Directorate is the Office of the Chief Information Officer (CIO). Among other things, this office is to leverage best available technologies and IT management practices, provide shared services, coordinate acquisition strategies, maintain an enterprise architecture that is fully integrated with other management processes, and advocate and enable business transformation. Other DHS entities also are responsible or share responsibility for IT management activities. For example, DHS’s major organizational components (e.g., directorates, offices, and agencies) have their own CIOs and IT organizations. Under this structure, control over the department’s IT management functions is shared by the DHS CIO and the component CIOs.

Also within the Management Directorate is the Office of the Chief Procurement Officer (CPO). The CPO is the department’s senior procurement executive who has leadership and authority over DHS acquisition and contracting, including major investments. This office’s responsibilities include issuing policies and implementing instructions, overseeing acquisition and contracting functions, and ensuring that a given acquisition’s contracting strategy and plans align with the intent of the Acquisition Review Board, DHS’s highest investment review board. Similar to the department and component CIOs, DHS relies on a structure of dual accountability and collaboration between the CPO and the heads of DHS components to carry out the acquisition function.

To promote coordination across DHS component boundaries, the DHS CIO and CPO have each established management councils. For example, the DHS CIO established the department’s CIO council, which is chaired by the DHS CIO and composed of component-level CIOs. According to its charter, the specific functions of the council include establishing a strategic plan, setting priorities for departmentwide IT, identifying opportunities for sharing resources, coordinating multi-bureau projects and programs, and consolidating activities.

To accomplish their respective missions, DHS and its component agencies rely on and invest heavily in IT systems and supporting infrastructure. For example, in fiscal year 2009, DHS IT-related funding totaled about $6.2 billion. Of DHS’s principal component organizations, Customs and Border Protection (CBP) represents the largest IT investor (about $1.7 billion or 28 percent). The next largest single investment in IT transcends DHS organizations and is for DHS-wide IT infrastructure ($1.5 billion), which includes, among other things, development of a replacement for the system used to share homeland security information with its federal, state, and local partners. The U.S. Citizenship and Immigration Services and the National Protection and Programs Directorate are the next largest investors in IT ($561 and $556 million, respectively). See figure 2 for more information on DHS components and their fiscal year 2009 funding.

According to DHS, the $6.2 billion in funding supports 279 major IT acquisition programs. Examples of these programs are described below.

Automated Commercial Environment (ACE): ACE is a CBP program that was begun in 2001 to modernize trade processing and support border security by, among other things, fully automating commercial import and export data processing and facilitating information sharing among federal agencies with a trade-related mission. ACE capabilities are being delivered in a series of increments, and thus far operational capabilities include screening cargo and conveyances, analyzing data to support targeting of high-risk entities, and processing truck manifests electronically. Future increments are to provide additional screening and combined manifest processing across all types of transportation. Through fiscal year 2009, DHS has been appropriated about $2.7 billion for ACE, and for fiscal year 2010, the department has requested about $268 million.

United States Visitor and Immigrant Status Indicator Technology (US-VISIT): This program dates to 2002 and is within the National Protection and Programs Directorate. It is to enhance the security of our citizens and visitors, ensure the integrity of the U.S. immigration system, protect privacy, and facilitate legitimate trade and travel. The program is to achieve these goals by, among other things, (1) collecting, maintaining, and sharing information on certain foreign nationals who enter and exit the United States; (2) identifying foreign nationals who have overstayed or violated the terms of their visit or who can receive, extend, or adjust their immigration status; (3) detecting fraudulent travel documents, verifying visitor identity, and determining visitor admissibility through the use of biometrics (digital fingerprints and a digital photograph); and (4) facilitating information sharing and coordination within the immigration and border management community.

DHS has delivered US-VISIT capabilities in a series of increments. As a result, a biometrically enabled entry capability has been operating at about 300 air, sea, and land POEs since December 2006 (115 airports, 14 seaports, and 154 of 170 land ports). Since 2004, DHS has evaluated a number of biometric exit solutions, and several exit pilot evaluations are currently underway. However, an exit capability is not yet operational. Through fiscal year 2009, DHS had been appropriated about $2.5 billion for US-VISIT, and for fiscal year 2010, the department has requested about $356 million.

Rescue 21: This is a Coast Guard program to modernize a 30-year-old search and rescue communications system used for missions 20 miles or less from shore, referred to as the National Distress and Response System. Among other things, it is to increase communications coverage area, allow electronic tracking of department vessels and other mobile assets, and enable secure communication with other federal and state entities. As of June 2009, Rescue 21’s initial operating capability has been deployed and accepted at 23 of 42 regions. Additional system capability (e.g., the ability to track vessels) remains to be developed, as does a system to meet the unique needs of the Alaska region. Through fiscal year 2009, DHS has been appropriated about $723 million for Rescue 21, and for fiscal year 2010, the department has requested about $117 million.

Secure Flight: This is a Transportation Security Administration (TSA) program to allow the federal government to assume from airlines the responsibility of prescreening passengers for domestic flights by matching of passenger biographic information against watch lists. Among other things, Secure Flight is to prevent people suspected of posing a threat to aviation from boarding commercial aircraft in the United States, protect passengers’ privacy and civil liberties, and reduce the number of people unnecessarily selected for secondary screening. TSA is currently in the process of phasing in its use of Secure Flight for domestic flights. Through fiscal year 2009, DHS has been appropriated about $326 million for Secure Flight, and for fiscal year 2010, the department has requested about $84.4 million.

SBInet: SBInet is the technology component of a CBP program known as SBI, which is to help secure the nation’s borders and reduce illegal immigration through physical infrastructure (e.g., fencing), surveillance systems, and command, control, communications, and intelligence technologies. As of 2009, a pilot of SBInet capabilities referred to as Project 28 has been deployed and is currently operating along 28 miles of the southwest border in Tucson, Arizona. Through fiscal year 2009, DHS has been appropriated about $3.6 billion for SBI, and for fiscal year 2010, the department has requested about $779 million.

DHS Has Made Uneven Progress in Establishing Institutional Management Controls and Capabilities for Large-Scale IT Acquisitions

The department has continued to work to establish effective corporate IT and acquisition management controls and capabilities, but progress across these disciplines has been uneven, and more remains to be done. Until DHS fully institutionalizes these controls and capabilities, it will be challenged in its ability to effectively and efficiently acquire large-scale IT systems and thereby leverage technology to support transformation and achieve mission goals and results.

Enterprise Architecture Continues to Evolve, But Key Content Still Missing

Leading organizations recognize the importance of having and using an enterprise architecture (EA)—a corporate blueprint that describes—in useful models, diagrams, tables, and narrative—how a given entity operates today and how it plans to operate in the future, and provides a road map for transitioning from today to tomorrow. Our experience with federal agencies has shown that attempting to acquire systems without an EA often results in investments that are duplicative, not well integrated, unnecessarily costly to maintain, and limited in terms of optimizing mission performance.

Since 2003, DHS has issued annual updates to its EA that have improved on prior versions by adding previously missing content. Specifically, we reported in November 2003 that DHS’s initial version of its EA was not sufficiently mature to guide and constrain investments. For example, while the department had established the management foundation for developing, maintaining, and implementing its EA and had issued an initial version of its target architecture, it had yet to develop products that fully described its current and target architectural environments, as well as a plan for transitioning from the current to the target environment.

In August 2004, we reported that the initial version of the department’s architecture provided a useful foundation on which to build a more complete architecture, but that it was still missing important content that limited its utility. For example, the content of this version was not systematically derived from a DHS or national corporate business strategy; rather it was an amalgamation of the existing architectures of the DHS predecessor agencies, along with their portfolios of systems investment projects. To assist DHS in evolving its architecture, we made 41 recommendations aimed at adding needed content.

In May 2007, we reported on the third version of DHS’s EA, concluding that while this version partially addressed each of our prior recommendations, it did not fully address them, and thus important content was still missing. Further, we reported that DHS organizational components were not adequately involved in its development. Accordingly, we made additional recommendations.

To the department’s credit, recent versions of its EA largely address our prior recommendations aimed at adding needed architectural depth and breadth. For example, in response to our prior recommendation that the architecture include a technical reference model (TRM) that describes, among other things, the technical standards to be implemented for each enterprise service, the 2008 version of the EA included a TRM that identified such standards. It also adopted an approach for extending the architecture through segments, which is a “divide and conquer” approach to architecture development advocated by OMB. To implement this approach, OMB guidance states that agencies should define and prioritize enterprise segments, focusing first on those segments that will help it perform its mission most effectively, and that they should first focus on developing architectures for high priority segments. However, while the 2008 EA identified 22 segments, it did not prioritize the segments.

DHS recently issued the latest version of its EA, and this version continues to improve on the prior version. For example, it contains a revised DHS business model that decomposes functional areas into business functions, describes information exchanges that support information sharing across organizational boundaries, and provides updated information security profiles for existing systems. It also updates the transition strategy for migrating to the target architecture by including planned 2010 investments. However, this version still does not contain prioritized segments and does not include OMB required architecture information for each segment (e.g., information exchanges between the critical business processes, conceptual solution architecture for each segment). Instead, the EA states that future versions will include revised segmented architectures within the context of its newly developed functional areas. As we have previously reported, segment architectures serve as a bridge between the corporate frame of reference captured in the EA and each individual system investment. Without well-defined segment architectures, DHS does not have a sufficient basis for investing in IT programs in a manner to ensure that they investments are properly sequenced, well integrated, and not duplicative.

IT Acquisition and Investment Management Improvements Made, But More Needs to be Done

Through effective corporate acquisition and investment management, organizations can make informed decisions when selecting among competing investment options and when controlling them throughout their acquisition life cycles. Based on our research, we issued an IT investment management framework that encompasses, among other things, best practices of successful public and private sector organizations relative to selecting and controlling individual investments as well as portfolios (segments) of investments. During the select phase, organizations are to (1) identify and analyze program/project risks and value before committing significant funds and (2) select those that will best support its mission needs. In the control phase, they are to ensure that programs/projects are meeting cost, schedule, and performance expectations at key milestone events, and that actions are taken to address deviations.

Since 2003, DHS has attempted to define and implement a corporate approach to overseeing its acquisition of major system investments, and we have continued to report limitations in its efforts to do so. Specifically, in August 2004, we reported that DHS had established an investment management process that provided for departmental oversight of major IT programs at key milestones, but that most programs (about 75 percent) had not undergone defined milestone reviews in a timely manner. At that time, DHS attributed this to the newness of the process. Based on our findings, we made recommendations aimed at strengthening the process.

In March 2005, we again reported on the department’s acquisition and investment review process, noting that while it incorporated some best practices and provided for senior management having information required to make well-informed investment decisions at key points in the acquisition life cycle, the process did not require senior management attention and oversight at all key decision points. For example, management reviews were not required prior to investment in a prototype or prior to passing a key acquisition milestone. Accordingly, we made further recommendations to improve the process.

In April 2007, we assessed DHS’s investment management structures, policies, and procedures against our ITIM framework, and concluded that while DHS had established investment decisionmaking bodies (e.g., investment review board) to oversee its IT investments, it had yet to fully define 8 of 11 key policies and procedures associated with selecting investments and controlling their acquisition. For example, procedures for selecting among competing investment options did not cite either the specific criteria or the steps for prioritizing and selecting investments at either the individual program level or the portfolio of programs level. In addition, the department had yet to document a methodology, with explicit criteria, for determining a given investment’s alignment to the EA. Instead, it relied on the undocumented and subjective determinations of individuals. We also reported that DHS had not fully implemented the key practices needed to control programs and portfolios of programs. For example, DHS investment review boards were not conducting regular investment reviews, and while program-specific control activities were sometimes performed, they were not performed consistently and thoroughly across investments.. Accordingly, we made recommendations aimed at establishing and implementing mature investment management processes.

In November 2008, we again reported that DHS was not effectively implementing its acquisition and investment review process. Specifically, while DHS’s review process called for its decision-making bodies to review investments at key points in their life cycles—including program authorization—45 of the 48 major investments that we examined were not reviewed in accordance with this process. In addition, DHS was unable to enforce decisions made by these investment bodies because it did not track whether its component organizations took actions called for in the decisions. Further, many of these major investments lacked basic acquisition documents necessary to inform the investment review process, such as program baselines; and two of nine components—which managed a total of 8 major investments—did not have required component-level investment management processes in place. Moreover, almost a third of the 48 major investments received funding without having validated mission needs and requirements, and two-thirds did not have life cycle cost estimates. Finally, DHS had not conducted regular reviews of its investment portfolios to ensure effective performance and minimize unintended duplication of effort. We concluded that without validated requirements, life cycle cost estimates, and regular portfolio reviews, DHS could not ensure that its investment decisions were appropriate and would ultimately address capability gaps. To address these weaknesses, we made a number of recommendations.

To strengthen its institutional approach to acquisition and IT investment management, DHS established the Acquisition Program Management Division (APMD) within the Office of the CPO, and assigned it responsibility for developing and maintaining the department’s acquisition policy and providing support and assistance to the department’s acquisition workforce. To that end, DHS issued a new departmental directive and related guidance in November 2008, which together provide the framework for departmental management, support, review, and approval of programs, including IT acquisitions.

The directive established a revised acquisition review process, including roles and responsibilities of DHS approving authorities, threshold levels for acquisitions, and acquisition decision events and the corresponding documentation required. Specifically, it established the Acquisition Review Board as the department’s highest review body and charged it with reviewing and approving all programs at key milestone decision points that are above $300 million in life cycle costs. It also described working groups and other boards, such as the Enterprise Architecture Board, and Program Review Board, to provide subject matter expertise to the Acquisition Review Board and DHS executives, and to review and approve investments that meet lower dollar thresholds. Recently established, according to a DHS official, was the DHS Asset Board (to provide lead technical authority on acquisition of real property and acquisition of vehicles). Finally, it is establishing the Joint Requirements Council (to validate the results of the strategic requirements planning process).

DHS has also reinstated regular acquisition review board meetings and acquisition decision memorandums. Specifically, DHS’s acquisition review board reports that it completed 14 acquisition reviews in 2008, and has thus far completed 18 reviews in 2009, including reviews of SBInet, US- VISIT, and Secure Flight. DHS also reports that 7 additional reviews are scheduled to occur by the end of the fiscal year. In addition, DHS components have designated Component Acquisition Executives (CAEs) to serve as the senior acquisition officials within the components and to be responsible for implementation of management and oversight of all component acquisition processes. DHS has also begun to make use of a new system to track program cost, schedule, and performance information, as well as action items that result from acquisition oversight board decisions. To support acquisition oversight, the CPO has identified a need for 58 additional positions. As an initial step, DHS’s fiscal year 2010 budget request included 10 additional full time equivalent positions for acquisition oversight support.

Notwithstanding these actions, the department’s acquisition and investment management processes still do not meet some of the program- and portfolio-level management practices in our ITIM framework, which are based on the investment management requirements in the Clinger- Cohen Act. With respect to program-level practices, DHS has not defined specific criteria for selecting and prioritizing new programs or for reselecting and reprioritizing existing ones. Without such criteria, it is unlikely that investment selection and prioritization decisions will be made consistently and will best support mission needs. Without proper management controls in place, it is unlikely that investment oversight decisions will be made consistently and will best support mission needs.

In addition, DHS has yet to adequately address how it determines and ensures that an investment is aligned with its EA. Specifically, while it has recently chartered its Enterprise Architecture Board and assigned it responsibility for ensuring that each investment is architecturally aligned throughout its life cycle, and while its new acquisition guidance specifies the architecture products that investments are to be aligned with (e.g., the business functions within the EA business model, the data objects in the conceptual data model, and the technical standards in the reference model), it has yet to define a methodology, including explicit criteria, for making a risk-based alignment determination. Also, the new directive and other DHS guidance do not provide for development of action plans for addressing areas of misalignment. DHS, in its comments, stated that they do not believe a methodology for alignment determinations is needed and that having subject matter experts involved in each determination is preferable given the wide range of IT programs at DHS; however, we believe that without such a methodology, it is not possible for the department to ensure that such alignment determinations are made consistently and repeatably. Without such acquisition and investment management controls, architecture alignment assessments will continue to largely be based on subjective and unverifiable judgments, and thus will not provide a sufficient basis for ensuring that systems are not duplicative and are interoperable.

With respect to portfolio-level practices, DHS does not have policies and procedures for evaluating or controlling its investment portfolios. Further, while post-implementation reviews are mentioned in DHS guidance, the guidance lacks specific procedures that would, for example, define roles and responsibilities for conducting these reviews and specify how the lessons learned and results of such reviews would be shared and used. Without such policies and procedures for portfolio management, DHS is at risk of not selecting and controlling the mix of investments in a manner that best supports the department’s mission needs.

We are continuing to monitor DHS’s efforts to more fully define its acquisition and investment management processes, as well as the extent to which acquisition reviews are performed regularly and consistently.

System Life Cycle Management Process Guidance Issued, But Improvements Still Needed

Managing IT projects and programs throughout their life cycles requires applying engineering discipline and rigor when defining, designing, developing, integrating, testing, deploying, and maintaining IT systems and services. Our evaluations and research show that applying such rigorous management practices improves the likelihood of delivering expected capabilities on time and within budget. In other words, the quality of IT systems and services is greatly influenced by the quality of the management processes involved in developing and acquiring them. According to leading practices, institutional system engineering maturity requires life cycle management processes that are clearly defined and applied on a repeatable basis across an organization.

A system life cycle management process normally begins with initial concept development and continues through requirements definition to design, development, various phases of testing, implementation, and maintenance. More specifically, during requirements definition, functional requirements are delineated in terms of system functionality (what the system is to do), performance (how well the system is to execute functions), data (what data are needed by what functions, when, and in what form), interfaces (what interactions with related and dependent systems are needed), and security (what controls are needed to address the assessed level of risk). As part of requirements definition, activities and documentation are produced to ensure that requirements are unambiguous, consistent with one another, linked (that is, traceable from one source level to another), verifiable, understood by stakeholders, and fully documented.

The steps in the life cycle process each have important purposes and they have inherent dependencies among themselves. Thus, if earlier life cycle steps are omitted or not performed effectively, later steps will be affected, potentially resulting in costly and time-consuming rework. For example, a system can be effectively tested to determine whether it meets requirements only if these requirements have been completely and correctly defined. To the extent that interdependent life cycle management steps or activities are not effectively performed, or are performed concurrently, a system acquisition or development program will be at risk of cost, schedule, and performance shortfalls.

Since 2004, we have reported that DHS lacked a standard and repeatable life cycle management process, and instead was relying on the processes that each of its components had in place. In 2008, DHS issued an interim life cycle management guide to introduce a standard system development methodology that can be tailored to specific projects. To the department’s credit, this guide addresses important aspects of effective system acquisition and development. For example, the guide requires that business objectives and systems requirements, as well as baseline performance goals, be defined and used as the measures of success for each program, and it requires that all programs be aligned with the HLS EA. Further, it requires acquisition management oversight and defines the roles and responsibilities of key stakeholders, including component CIOs and DHS IT portfolio managers, and to accomplish this it requires checkpoint reviews (i.e., stage reviews) throughout the program’s life cycle. In addition, it specifies key activities associated with each life cycle stage (planning, requirements definition, design, development, integration and test, implementation, operation and maintenance, and disposition).

However, the interim guide does not address all key activities for each life cycle phase. For example, it does not address key practices associated with acquiring commercial products or services, such as evaluating commercial product and supplier viability and assessing commercial product dependencies/interoperability before purchasing the products. Also, while it does identify a list of work products that are to be created and updated to record the results of the activities performed for each life cycle stage, it does not address the content of all of these work products. For example, it does not provide a sample document or content template for a quality assurance plan, a configuration management plan, or a service reuse plan. Thus, opportunities remain to further define the SDLC. Moreover, it is unclear when and how this SDLC will be implemented. Until addressed, DHS will remain challenged in its ability to acquire and develop systems in a defined and repeatable manner.

Acquisition and IT Workforce Management Remains a Challenge

A strategic approach to human capital management is critical to ensuring that an organization has the right people with the right skills at the right time to perform a given function. Based on our research of leading organizations, we issued a model for strategic human capital management in which strategic human capital planning was one cornerstone. Through such planning, organizations can remain aware of its current workforce capabilities and its future workforce needs, and can be prepared for meeting these needs. According to our guidance, key practices for effective strategic human capital planning are generic, applying to any organization or component, such as an agency’s acquisition or IT organization. They include: Involving top management, employees, and other stakeholders in developing, communicating, and implementing a strategic workforce plan; Determining the critical skills and competencies needed to achieve current and future programmatic results; Developing strategies tailored to address gaps between the current workforce and future needs; Building the capability to support workforce strategies; and Monitoring and evaluating an agency’s progress toward its human capital goals and the contribution that human capital results have made to achieving programmatic goals.

As is summarized below, DHS has yet to address either its acquisition or IT workforce needs in a manner that is fully consistent with these practices. Until DHS does so, it will continue to be at risk of not having sufficient people with the right knowledge, skills, and abilities to effectively and efficiently acquire key system investments.

In November 2008, we reported that DHS had not developed a comprehensive strategic acquisition workforce plan to direct the department’s future acquisition workforce efforts, and that the department lacked several elements that are key to developing such a plan. More specifically, we reported that DHS lacked an overall direction for acquisition workforce planning, and notwithstanding some recent actions, had not fully involved key stakeholders, such as the CHCO and component procurement and program offices, both of which have been shown to increase the likelihood of success for workforce planning; excluded some acquisition-related career fields from its definition of acquisition workforce, thus limiting the scope of its planning efforts, and while it intended to expand its definition, it had yet to identify which positions should be included; lacked sufficient data to fully assess its acquisition workforce needs, including the gaps in the number of employees needed or the skills of these employees; and lacked sufficient insight into the number of contractors supporting its acquisition function or the types of tasks that contractors were performing.

DHS has undertaken several initiatives to begin addressing its acquisition workforce challenges. For example, its recruiting, hiring, and training initiatives have allowed it to hire new contract specialists and expand workforce access to acquisition-related training. Specifically, in January 2008, the CPO implemented the Acquisition Professional Career Program, and as of September 2008, had hired 49 contract specialist interns. In addition, CPO established an Acquisition Training Program in 2008 that included DHS-specific training for program managers, and it formed a council to coordinate acquisition workforce training opportunities across components.

In November 2008, we reported on several challenges that DHS faced in managing these initiatives. For example, most initiatives aimed at defining and identifying the acquisition workforce and assessing acquisition workforce needs had yet to produce results, and in some cases were progressing more slowly than originally projected. DHS’s initiatives also primarily focused on contract specialists despite other identified acquisition workforce shortages, and DHS had not determined how it would expand the initiatives. Further, DHS generally lacked documented performance goals and implementation steps—such as actions to be taken, needed resources, and milestones—for these initiatives.

Since that time, DHS has taken steps to expand two of its recruiting and hiring initiatives to additional acquisition-related career fields. Specifically, DHS developed plans to include career fields such as program management and engineering in its fall 2009 Acquisition Professional Career Program cohort. According to a CPO representative, DHS also plans to add acquisition career fields to its centralized hiring program and has recently hired a recruitment coordinator to carry out this expansion.

In June 2004, we reported that DHS had begun strategic planning for IT human capital at the headquarters level, but it had not yet systematically gathered baseline data about its existing IT workforce across the department. Moreover, the DHS CIO had expressed concern at that time about staffing and acknowledged that progress in this area had been slow. In our report, we recommended that the department analyze whether it had appropriately allocated and deployed IT staff with the relevant skills to obtain its institutional and program-related goals. In response, the CIO established an IT human capital Center of Excellence to deliver, plans, processes, and procedures to execute an IT human capital strategy and to conduct an analysis of the skill sets of DHS IT professionals.

In September 2007, we reported that DHS had developed a IT human capital plan and related documents that were largely consistent with federal guidance and associated best practices. For example, they provided for developing a complete inventory of existing IT staff skills, identifying IT skills needed to achieve agency goals, determining skill gaps, and developing plans to address such gaps. They also provided for involving key stakeholders—such as the CIO, Chief Human Capital Officer (CHCO), and component agency CIOs and human capital directors—in carrying out the skill gap analyses and follow on workforce planning.

However, we also reported that the plan did not fully address twelve key practices. For example, although the plan and supporting documents described the department’s IT human capital goals and steps necessary to implement them, most steps did not include associated milestones. In addition, although the plan and supporting documents provided for involving key stakeholders, they did not assign those stakeholders specific responsibilities against which to hold them accountable for results. We also reported at that time that DHS had made limited progress in implementing its IT human capital plan. In particular, DHS CIO and CHCO officials, as well as officials from the three DHS agencies that we examined (CBP, FEMA, and the Coast Guard), all told us that they had yet to begin implementing the plan. Accordingly, we made recommendations aimed at strengthening and implementing the plan.

DHS has made limited progress in addressing our recommendations. For example it has not established implementation milestones, assigned stakeholder responsibilities and accountability, or begun to track, document, and report on human capital risks. Also, while DHS reported in 2007 that it intended to analyze its IT workforce makeup every 2 years, CIO and CHCO officials told us that this will not be done until after a planned 2010 Federal CIO Council-sponsored survey of the governmentwide IT workforce. Further, these officials stated that implementation of the 2007 IT human capital plan has been limited because the department’s focus has been on strengthening its executive leadership team and its acquisition workforce, and that it only recently became engaged on departmentwide IT workforce issues. However, they added that DHS component organizations have been working to strengthen staff core competencies in four IT disciplines—Project Management, Security/Information Assurance, Enterprise Architecture, and Solutions Architecture.

According to officials from CBP, FEMA, and the Coast Guard, none of these component organizations have taken specific actions to implement the 2005 DHS IT human capital plan because they have not received any departmental instruction or guidance for doing so. Moreover, the extent to which they are each proactively and strategically addressing their respective human capital needs varies. For example, CBP’s Office of Information Technology Workforce Management Group has a strategic IT human capital plan that defines goals (e.g., creating and enabling a team of leaders who have both the technical expertise and skills to manage and motivate employees, and providing education, training and development opportunities to allow employees to grow in their jobs and their careers), and the group has taken actions to achieve the goals (i.e., identifying employees with leadership potential, developing a leadership curriculum for them, establishing an internship program, and creating a skills inventory). In contrast, FEMA’s Office of Information Technology does not have a strategic IT human capital plan, although officials report that one is to be completed in fiscal year 2010, and in the interim, this office is assessing its workforce competency gaps, among other things. Further, while the Coast Guard has an IT strategic human capital plan, this plan is more than a decade old, as officials report that they have no immediate plans to update it.

Large-Scale IT Investments Exposed to Risk Because Key Acquisition and IT Management Controls Have Not Always Been Effectively Implemented

The success of a major IT program can be judged by the extent to which it delivers promised system capabilities and mission benefits on time and within schedule. As our research and evaluations show, a key determinant of program success is the extent to which the earlier discussed institutional acquisition and IT management controls are appropriately employed in managing each and every IT investment.

In this regard, our reviews of a number of large-scale DHS IT investments have disclosed a range of program management control weaknesses that have increased the risk of cost, schedule, and performance shortfalls. In many cases, DHS has since taken steps to address the weaknesses that we identified. However, some weaknesses have lingered, and we continue to identify issues on other programs. Moreover, these weaknesses are contributing to programs falling short of their capability, benefit, cost, and schedule expectations. To illustrate the prevalence and significance of these acquisition and IT management weaknesses, as well as DHS’s progress in addressing them, we discuss work related to five large-scale programs—ACE, US-VISIT, Rescue 21, Secure Flight, and SBInet.

ACE is a multi-billion dollar program to incrementally modernize trade processing and support border security. Since 1999, we have issued a series of reports that have disclosed a number of acquisition and investment management weaknesses that have contributed to ACE performance shortfalls, including program costs increasing from $1 billion to about $3.1 billion, and ACE schedule slipping from fiscal year 2007 to fiscal year 2010. To address the weaknesses, we have made a number of recommendations. CBP has largely agreed with our recommendations, and continues to work to implement many of them. Below we provide a brief summary of ACE-related efforts to implement effective acquisition and IT management controls.

Beginning in May 1999, we reported that ACE was not being defined in the context of an enterprise architecture, and that its life cycle cost estimates and cost/benefit analysis were inadequate. Further, ACE was not being acquired in accordance with disciplined investment management processes. As a result, CBP was not positioned to know that it was pursuing the right system solution for its needs and to deliver a defined a solution on time and schedule. Subsequently, CBP adopted an incremental approach to acquiring ACE, which we supported as a proven risk reduction measure for acquiring large-scale systems, but as we reported in June 2001, ACE was being pursued separate from another trade-related system (known as the International Trade Data System), which was duplicative of and not aligned with ACE. Subsequently, this related system was merged with ACE.

Between May 2002 and February 2003, we continued to report on ACE challenges and weaknesses. Specifically, we reported that ACE was risky for a variety of reasons, including cost overruns, implications for changing how trade processing was performed, and known key acquisition and IT management control weaknesses associated with, for example, program office human capital and software management processes. Subsequently, we reported that CBP was working to implement our previous recommendations aimed at addressing acquisition and IT management control weaknesses, but that problems continued. For example, ACE cost estimates were not reliable because they were not derived in accordance with estimating best practices. The next year we again reported that ACE was not following rigorous and disciplined acquisition and IT management controls, such as those related to managing the program office human capital, risks, and contract management. For example, while initial ACE test results were positive, CBP had not taken steps to independently oversee the contractor’s testing.

In May 2004, we reported that the first two ACE system increments were operating, but that CBP’s approach to incrementally acquiring and deploying ACE involved excessive overlap among increments. Moreover, the scheduling of increments had allowed for considerable overlap and concurrency among them, and this had produced a pattern of having to borrow resources from later increments to complete earlier increments. We concluded that this pattern had and would continue to result in ACE cost overruns and schedule delays. The next year, we reported that while CBP had revised its cost baselines in light of ACE overruns, this was not sufficient because the number of ACE increments had increased and system quality standards had been relaxed to allow increments to proceed through key milestones despite the presence of material system defects. We concluded that this practice, combined with the concurrency of increments, would exacerbate the program’s cost and schedule shortfalls.

We also reported that previously identified management control weaknesses remained, such as in system testing and in cost estimation, and that progress in addressing our recommendations had been slow.

In May 2006, we reported that CBP had begun to make progress in addressing our recommendations through the establishment and use of a program-wide performance and accountability framework, as we had also recommended. However, control weaknesses remained. For example, considerable concurrency still remained among increments, thus increasing the risk of continued cost and schedule overruns. Also, while earned value management was an OMB requirement, CBP discontinued its use on two ACE increments, thus limiting its ability to measure performance and progress.

In October 2007, we reported that CBP had continued to take steps to establish an accountability framework grounded in measuring and disclosing progress against program performance measures and targets. However, ACE costs were likely to increase further because prior limitations in how system requirements were defined had resulted in an increase requirements and the need to replace a key software product, even though the new product may reduce user productivity. In addition, the inventory of ACE-related risks was incomplete and that information needed to make informed decisions on these risks was not being maintained.

We plan to continue to monitor CBP’s progress in implementing our ACE- related recommendations.

US-VISIT is a multi-billion dollar program to collect and maintain biographic and biometric information on certain foreign nationals who enter and exit the United States through over 300 air, sea, and land ports of entry. Since 2003, we have continued to report on US-VISIT acquisition and IT management control weaknesses that increased the risk of delivering less system capabilities and mission benefits than envisioned, and taking longer and costing more than expected. To the department’s credit, it has addressed many of the recommendations that we have made for addressing these weaknesses, and as a result the program is better positioned today for success than it has been in the past. However, these weaknesses have contributed to instances of the program not living up to expectations, and some weaknesses still remain that pose future risks. Below we provide a brief summary of US-VISIT-related efforts to implement effective acquisition and IT management controls.

We first reported on US-VISIT in June 2003, finding that program plans did not sufficiently define what specific system capabilities and benefits would be delivered, by when, and at what cost, and how US-VISIT intended to manage the acquisition to provide reasonable assurance that it would meet their commitments. Without defining such commitments, it was not possible to measure program performance and promote accountability for results. Shortly thereafter, in September 2003, we concluded that the program was high risk because, among other things, its size, complexity, mission criticality, and enormous potential costs, coupled with a range of program management control weaknesses, including an immature governance structure, lack of clarity about its operational environment, facility implications, and mission value. In May 2004, we reported that US-VISIT did not have a current life-cycle cost estimate or a cost benefit analysis, and that testing of an initial increment of system capabilities was not well-managed, and was not completed until after the increment became operational. Moreover, the test plan used was not completed until after testing was concluded.

In February 2005, we reported that DHS had hired a prime integration contractor to augment its ability to deliver US-VISIT, but that acquisition management weaknesses continued. For example, we found that an effort to pilot alternative system solutions for delivering the capability to track persons exiting the U.S. was faced with a compressed time line, missed milestones, and a reduced scope that limited its value.

In February 2006, we reported that the DHS’s progress in implementing 18 GAO recommendations made in previous reports was mixed, but overall slow in critical areas, including completing cost-benefit analyses for increments, determining whether proposed increments would produce mission value consistent with costs and risks, developing well-defined and traceable test plans prior to testing, and assessing workforce and facility needs for new functionality.

In February 2007, we reported that DHS had not adequately defined and justified its proposed investment in planned and ongoing exit pilot and demonstration projects, and that it continued to invest in US-VISIT without a clearly defined operational context (enterprise architecture) that included explicit relationships with related border security and immigration enforcement initiatives. At the same time, program management costs had risen sharply, while costs for development had decreased, without any accompanying explanation of the reasons. We also reiterated our prior findings concerning a lack of program transparency and accountability due to inadequate definition and disclosure of planned expenditures, timelines, capabilities, and benefits, as well as limited measurement and reporting on progress against each.

In August 2007, we reported that while US-VISIT entry capabilities were operating at over 300 ports of entry, exit capabilities were not, and that DHS did not have a comprehensive plan or a complete schedule for delivering a biometric exit solution. In addition, DHS continued to invest heavily in program management activities without adequate justification for doing so, and it continued to propose spending tens of millions of dollars on US-VISIT exit projects that were not well-defined, planned, or justified on the basis of costs, benefits, and risks.

In February 2008, we reported that while DHS had partially defined a strategic solution for meeting US-VISIT goals, including defining and beginning development of a key capability known as “Unique Identity,” which was to establish a single identity for all individuals at their earliest possible interaction with any U.S. immigration and border management organization by capturing the individual’s biometrics, including 10 fingerprints and a digital image. However it had not defined and economically justified a comprehensive strategic solution for controlling and monitoring the exit of foreign visitors, which was critical to accomplishing the program’s goals. DHS was also taking a range of evolving actions, partially at the department level, to coordinate relationships among US-VISIT and other immigration and border control programs; however, this evolution had yet to progress to the point of reflecting the full scope of key practices that GAO previously identified as essential to enhancing and sustaining collaborative efforts that span multiple organizations. As a result, the department was at increased risk of introducing inefficiencies and reduced effectiveness resulting from suboptimizing these programs’ collective support of immigration and border management goals and objectives.

In December 2008, we reported on a lack of effective DHS executive oversight of the program, including involvement from the DHS CPO and the CHCO. In addition, we again reported that DHS lacked a detailed schedule for implementing an exit capability, and that, among other things, cost estimates for the then proposed exit solution were not reliable, risk management was not being effectively performed, and the program’s task orders were frequently rebaselined, thus minimizing the significance of earned value management-based schedule variances.

Currently, we have work underway for the Chairman of the House Homeland Security Committee on the US-VISIT Comprehensive Exit project, including the extent to which the project’s component efforts are being managed in an integrated fashion. In addition, we are required by statute to review the results of an ongoing pilot of exit solutions at airports.

Rescue 21

Rescue 21 is a billion dollar Coast Guard program to replace its existing search and rescue communications system—installed in the 1970’s. Among other things, Rescue 21 is to allow continuous, uninterrupted communications on the primary ship-to-shore channel, limit communications gaps to less than 10 percent in the United States, provide direction finding and digital selective calling to better locate boaters in distress, allow communication with other federal and state systems, and protect communication of sensitive information. We have issued reports citing a number of acquisition and investment management weaknesses that have contributed to Rescue 21 performance shortfalls, including program costs increasing from $250 million to about $1 billion, and the schedule slipping from fiscal year 2006 to fiscal year 2017. To address the weaknesses, we have made a number of recommendations. Coast Guard has largely agreed with our recommendations, and continues to work to implement many of them. Below we provide a brief summary of Rescue 21-related efforts to implement effective acquisition and IT management controls.

In September 2003, we reported that Rescue 21’s initial operating capability milestone of September 2003 had been postponed, and that a new schedule had yet to be finalized. Also, while the program had established processes for managing system requirements and managing risks, the processes were not being followed. For example, key deliverables for testing, such as test plans, were not yet defined and approved.

In May 2006, we reported that Rescue 21 continued to experience acquisition management weaknesses relative to requirements management, project monitoring and oversight, risk management, cost and schedule estimating, and executive oversight, and that these weaknesses had contributed to program cost overruns and schedule delays. Specifically, Rescue 21’s total acquisition cost had risen from $250 million to $710.5 million, an increase of 184 percent, and its timeline for achieving full operational capability had been delayed from 2006 to 2011. Moreover, the most recent cost and schedule estimates were not reliable, and the program faced a possible future cost overrun of $161.5 million, which would bring the total acquisition cost to $872 million. Finally, the schedule estimate was uncertain due to ongoing contract renegotiations for the remaining sites, and pending decisions regarding vessel tracking functionality. Since then, the Coast Guard estimates that the program’s total acquisition cost will exceed $1 billion; deployment of Rescue 21 to the 48 continuous states will be delayed to 2012; deployment of the vessel tracking capability will be delayed to 2015; and deployment to Alaska will not occur until 2017.

Secure Flight

Secure Flight is a multi-billion dollar TSA program to allow DHS to assume from airlines the responsibility of prescreening passengers for domestic flights by matching of passenger biographic information against terrorist watch lists. Among other things, Secure Flight is to prevent people suspected of posing a threat to aviation from boarding commercial aircraft in the United States, protect passengers’ privacy and civil liberties, and reduce the number of people unnecessarily selected for secondary screening. TSA is currently in the process of phasing in its use of Secure Flight for domestic flights. Since 2005, we have reported on a number of acquisition and investment management weaknesses, such as requirements, testing, cost and schedule estimation, and security management, and made recommendations to address them. To TSA’s credit, it has addressed most of the recommendations. Below we provide a brief summary of TSA efforts to implement effective acquisition and IT management controls.

We first reported on Secure Flight in March 2005, finding that TSA had not yet completed key development activities needed to successfully deliver an operational system, such as finalizing requirements documents or completing required test activities. In addition, TSA had not developed performance goals and measures to gauge the effectiveness of the Secure Flight program, nor had it developed life-cycle cost estimates, which limited oversight and accountability.

In February 2006, we reported that while TSA had made some progress in developing and testing Secure Flight, it had not followed a disciplined life cycle approach and, as a result, some project activities were conducted out of sequence, requirements were not well defined, and documentation contained contradictory information or omissions. Further, while TSA had taken steps to implement an information security management program for protecting information and assets, its efforts were incomplete, and that the program lacked schedule and cost estimates. Accordingly, we made recommendations to address these limitations. Later that year we reported that TSA had begun taking actions to address our recommendation, including suspending development and undertaking a rebaselining, of the program.

In February 2007, we reported that despite 4 years of effort, TSA had been unable to develop and implement Secure Flight, in large part, because it had not employed a range of acquisition and IT management control disciplines to effectively manage cost, schedule, performance, and privacy risks. At that time, TSA officials stated that they intended to put in place a new management team; rebaseline the program’s goals, capabilities, costs, and schedule; and establish more structured and controlled acquisition and IT management processes.

In February 2008, we reported that TSA had made substantial progress in instilling more discipline and rigor into Secure Flight’s development and implementation. For example, TSA had developed a detailed concept of operations, established a cost and schedule baseline, and drafted key management and systems development documents, among other systems development efforts. However, TSA had not followed established risk management processes and it had not followed key practices for developing reliable cost and schedule estimates. Further, TSA had yet to incorporate end-to-end testing into its testing strategy, and had not addressed all system security requirements and vulnerabilities.

On January 7, 2009, we reported that TSA had not demonstrated Secure Flight’s operational readiness and had generally not achieved several conditions set forth in the Department of Homeland Security (DHS) Appropriations Act, 2005. These conditions related to, among other things, performance of stress testing and estimation of cost and schedule. For example, we found that despite provisions for stress testing in Secure Flight test plans, stress testing had not been performed. Further, while TSA had made improvements to its life-cycle cost estimate and schedule, neither were developed in accordance with key best practices. As a result, the life-cycle cost estimate did not provide a meaningful baseline from which to track progress, hold TSA accountable, and provide a basis for sound investment decision making.

To TSA’s credit, we recently reported that it had made notable progress in developing Secure Flight, including meeting nine out of ten key legislative conditions, including conducting performance and stress testing. As a result, TSA was poised at the time to begin incremental deployment of Secure Flight. Since then, Secure Flight has begun operating at selected airports and for selected airlines.

SBInet is a multi-billion dollar program that involves the acquisition, development, integration, and deployment of surveillance systems and command, control, communications, and intelligence (C3I) technologies to create a “virtual fence” along our nation’s borders. Since 2007, we have reported on a number of SBInet acquisition and IT management weaknesses that increased the risk that the SBInet system will not perform as intended and meet user needs and expectations. For example, our first report identified weaknesses in how CBP was defining system requirements and managing program risks, including risks associated with acquiring SBInet through a series of concurrent task orders. In October 2007 and again in February 2008, we reported that the SBInet pilot, known as Project 28, was almost 8 months behind schedule in part because requirements were not adequately defined, contractor oversight was limited, and testing was not sufficiently performed. Later in 2008, we again reported on limitations in how SBInet risks were being managed, as well as areas in which SBInet had yet to demonstrate alignment to DHS’s enterprise architecture.

In September 2008, we reported that after investing about 3 years in acquiring and developing SBInet, important aspects of the program remained ambiguous and were in a continued state of flux, making it unclear and uncertain what technology capabilities would be delivered, when and where they would be delivered, and how they would be delivered. Also, the program did not have an approved integrated master schedule to guide the execution of the program, and that assimilation of available information indicated that the schedule had continued to change.

Further, we reiterated that the program had not effectively performed key requirements development and management practices, such as ensuring alignment between different levels of requirements. Finally, we reported that SBInet testing had not been effectively managed; individual system components to be deployed to the initial deployment locations had not been fully tested, a test management strategy had not yet been finalized and approved, and the draft plan contained omissions in content.

We made a series of recommendations to address these weaknesses, including assessing SBInet development, testing, and deployment risks and disclosing them to DHS leadership and the Congress, and defining and implementing relevant system deployment, requirements management, and testing weaknesses guidance. DHS largely agreed with our recommendations. We currently have work underway for the Chairman, House Homeland Security Committee, relative to SBInet risks and recommendation implementation, SBInet test management, planning, execution, and results, and SBInet contract management and oversight.

In closing, the department has made progress in establishing key institutional acquisition and IT investment management-related controls and implementing them on large-scale programs, including its recent efforts to increase corporate oversight of major investments and its recent deployment and operation of Secure Flight. However, considerable work remains to be accomplished before the department can be considered a mature IT system acquirer and investor. For example, the department has yet to address longstanding challenges in, among other things, sufficiently defining its enterprise architecture and strategically managing its acquisition and IT workforce. Moreover, while program-specific weaknesses that we have identified have in many cases eventually been addressed, our concern is that these types of weaknesses were allowed to exist and in some cases took years to address, and that we continue to find them on other programs that we later review. Such a pattern of inconsistency across major programs is indicative of institutional acquisition and IT management immaturity. Unless this changes, ongoing and future DHS major acquisitions will likely fall short in delivering promised capabilities and benefits on time and on budget.

Our existing recommendations continue to provide the department with a framework for maturation, and thus we encourage the department to move swiftly in implementing both our institutional and program-specific recommendations. To this end, we look forward to working constructively with the department in doing so and thereby maximizing the role that IT can play in DHS’s mission performance and transformation.

Madame Chairwoman, this concludes my statement. I would be happy to answer any questions you have at this time.

Contacts and Acknowledgements

For future information regarding this testimony, please contact Randolph C. Hite, Director, Information Technology Architecture and Systems Issues, at (202) 512-3439, or hiter@gao.gov. Other individuals who made key contributions to this testimony were Kathleen Agatone, Mathew Bader, Justin Booth, James Crimmer, Deborah Davis, Elena Epps, Ash Huda, John P. Hutton, Tonia Johnson, Neela Lakhmani, Anh Le, Anne McDonough-Hughes, Gary Mountjoy, Sabine Paul, Tomas Ramirez, Jr., Amelia Shachoy, and Teresa Smith.

This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.