Information Technology: DHS Needs to Enhance Management of Cost and Schedule for Major Investments
Released: 2012-09-26T00:00:00

Letter

Background

DHS’s mission is to lead the unified national effort to secure America by preventing and deterring terrorist attacks and protecting against and responding to threats and hazards to the nation. DHS is also responsible for ensuring that the nation’s borders are safe and secure, that they welcome lawful immigrants and visitors, and that they promote the free flow of commerce.

Created in 2002, DHS assumed control of about 209,000 civilian and military positions from 22 agencies and offices that specialize in one or more aspects of homeland security. The purpose behind the merger was to improve coordination, communication, and information sharing among these multiple federal agencies. Figure 1 shows DHS’s organizational structure and table 1 identifies DHS’s principal organizations and describes their missions.

Within the department’s Management Directorate, headed by the Under Secretary for Management, is the Office of the Chief Information Officer (CIO). The CIO’s responsibilities include setting departmental IT policies, processes, and standards, and ensuring that IT acquisitions comply with DHS IT management processes, technical requirements, and approved enterprise architecture, among other things. Additionally, the CIO chairs DHS’s Chief Information Officer Council (CIO Council), which is responsible for ensuring the development of IT resource management policies, processes, best practices, performance measures, and decision criteria for managing the delivery of IT services and investments, while controlling costs and mitigating risks.

DHS IT Investments Support Its Mission

DHS spends billions of dollars each year on IT investments to perform both mission-critical and support functions that frequently must be coordinated among components, as well as among external entities. Of the $5.6 billion that DHS plans to spend on 363 IT-related investments in fiscal year 2012, $4.4 billion is planned for the 83 the agency considers to be a major investment; namely, costly, complex, and/or mission critical.

Of these 83 major IT investments, 68 are under development and have planned fiscal year 2012 costs of approximately $4 billion. Examples of major investments under development that are being undertaken by DHS and its components include:

U. S. Customs and Border Protection—Automated Commercial Environment/International Trade Data System will incrementally replace existing cargo processing technology systems with a single system for land, air, rail, and sea cargo and serve as the central data collection system for federal agencies needing access to international trade data in a secure, paper-free, web-enabled environment.

Immigration and Customs Enforcement —TECS Modernization is to replace the legacy mainframe system developed by the U.S. Customs Service in the 1980s to support its inspections and investigations. Following the creation of DHS, those activities were assigned to CBP and ICE, respectively. CBP and ICE are now working to modernize their respective portions of the system in a coordinated effort with separate funding and schedules. ICE’s portion of the investment will include modernizing the investigative case management and related support modules of the legacy system.

National Protection and Programs Directorate—National Cybersecurity Protection System, also referred to as EINSTEIN, is an integrated system that includes intrusion detection, analytics, intrusion prevention, and information sharing capabilities that are used to defend the federal executive branch civilian agencies’ IT infrastructure from cyber threats. It consists of the hardware, software, supporting processes, training, and services that are being developed and acquired to support DHS’s mission requirements.

Best Practices and OMB Guidance Stress Importance of Keeping IT Investments within Cost and on Schedule

The success of major IT investments are judged by, among other things, the extent to which they deliver promised system capabilities and mission benefits on time and within cost. Consequently, our best practices research and extensive experience at federal agencies, as well as OMB guidance, stress the importance of federal IT investments meeting cost and schedule milestones.

GAO’s Information Technology Investment Management guidancehighlights the need to regularly determine each IT project’s progress toward cost and schedule milestones using established criteria and calls for corrective efforts when milestones are not being met. The guidance also calls for such corrective efforts to be defined and documented. OMB plays a key role in helping federal agencies manage their investments by working with them to better plan, justify, and determine how much they need to spend on projects and how to manage their approved projects. In December 2010, OMB issued its 25 Point Implementation Plan to Reform Federal Information Technology Management, a plan to change IT management throughout the federal government by strengthening the role of investment review boards to enable them to more adequately manage agency IT portfolios, redefining the role of agency CIOs and the Federal CIO Council to focus on portfolio management, and implementing face-to-face reviews to identify IT investments that are experiencing performance problems and to select them for a TechStat session—a review of selected IT investments between OMB and agency leadership that is led by the Federal CIO. In addition, OMB provides agencies with tools to measure how effectively investments are meeting established cost and schedule parameters. Specifically, OMB requires federal agencies to provide information on their IT investments as a part of their yearly budget submissions, and to do so using an exhibit 53, in which they list all of their IT investments and their associated costs, and an exhibit 300, also called the Capital Asset Plan and Business Case, which includes an investment’s cost and schedule commitments.

Further, in June 2009, OMB deployed the IT Dashboard, a website that displays near real-time information on, among other things, the cost and schedule performance of all of an agency’s major IT investments. The IT Dashboard provides, among other things, a cost and schedule performance rating for each major IT investment’s subsidiary project. These ratings are based on the extent to which the project is meeting its cost and schedule commitments. For example, projects experiencing a 10 percent or greater cost and/or schedule variance are considered to be at an elevated risk of not delivering promised capabilities on time and within budget, and, as such, require management attention.

Prior GAO Reports Highlighted Cost and Schedule Shortfalls of Major DHS Investments

We have previously reported on the cost and schedule challenges associated with major DHS IT investments, such as those with CBP’s Secure Border Network (SBInet) and NPPD’s United States Visitor and Immigrant Status Indicator Technology (US-VISIT). For example, in 2007 we reported that the Secure Border Network had experienced significant cost and schedule shortfalls due, in part, to the project not having fully defined activities. In addition, in May 2010, we reported that continued delays to the investment were likely because, among other things, it had not developed a reliable integrated master schedule and the schedule did not adequately capture all necessary activities. In these reports, we made recommendations to strengthen the program weaknesses to keep the investment on schedule and within cost.

With regard to the US-VISIT investment, we noted in a November 2009 report that officials had not adopted an integrated approach to scheduling, executing, and tracking the work that needed to be accomplished to deliver the Comprehensive Exit project to more than 300 ports of entry on schedule and within cost. Accordingly, we recommended that DHS strengthen management of the project by ensuring that it develop and maintain integrated scheduling plans in accordance with applicable key practices; DHS concurred with the recommendations.

Further, in 2011, as a part of our High Risk series, because of acquisition weaknesses, major investments, such as the recently canceled SBInet, continued to be challenged in meeting capability, benefit, cost, and schedule expectations. Based on our prior work, we identified and provided to DHS key actions and outcomes critical to addressing this and other challenges.

Most recently, we reported in July 2012 that DHS was making progress in developing and implementing a new IT governance process. We found that DHS had developed a new governance framework and that the associated policies and procedures were generally consistent with recent OMB guidance and with best practices for managing projects and portfolios identified in GAO’s Information Technology Investment Management framework; however, the agency had not yet finalized most policies and procedures and was not fully using best practices for the implementation. Accordingly, we made recommendations to DHS to, among other things, strengthen its new governance process and related IT management capabilities; the agency agreed to implement the recommendations.

GAO, High Risk Series: An Update, GAO-11-278 (Washington, D.C.: February 2011).

GAO, Information Technology: DHS Needs to Further Define and Implement Its New Governance Process, GAO-12-818 (Washington, D.C.: July 2012).

DHS Meeting Cost and Schedule Commitments for Most Major IT Investments

As discussed previously, our best practices research and experience at federal agencies as well as OMB guidance stress the importance of investments meeting their cost and schedule commitments. OMB requires agencies to report to the IT Dashboard information on the cost and schedule performance of all their major IT investments.

Our analysis of the cost and schedule performance for DHS’s 68 major IT investments shows that approximately two-thirds of these investments and their subsidiary projects were meeting cost and schedule commitments; the remaining one-third had at least one subsidiary project that was not meeting its commitments. Specifically, out of the 68 major investments under development, 47 were meeting their cost and schedule commitments. (See app. II for a listing of the 47 investments and subsidiary projects that are meeting their commitments.) The remaining 21 investments had one or more subsidiary projects that were not meeting cost and/or schedule commitments; the total planned cost for all projects in development for the 21 investments is approximately $1 billion. Table 2 lists the investments experiencing cost and/or schedule shortfalls, and the total planned project cost for each investment. A list of the investments and their subsidiary projects experiencing cost and/or schedule shortfalls is included in appendix III.

Of the 21 investments with a shortfall, 5 had one or more subsidiary project with a cost shortfall, 18 had one or more project with a schedule shortfall, and 2 had a project with both a cost and schedule shortfall. These shortfalls potentially impact the total cost of investments and can delay the implementation of key systems. For example:

TSA’s Federal Air Marshal Service Mission Scheduling and Notification System: project to modernize the core scheduling software component of the system, which, among other things, determines the allocation of federal air marshals to flights and coordinates and communicates mission assignments, was delayed.

NPPD’s Critical Infrastructure Technology and Architecture investment: project to develop an information-sharing application to be used by federal, state, and local stakeholders to increase their capability to combat terrorist use of improvised explosive devices had cost overruns of approximately 16 percent ($296,000).

CBP’s Northern Border, Remote Video Surveillance System investment: project to incorporate IT Security improvements to the remote video surveillance systems in Buffalo, New York, and Detroit, Michigan, was delayed by approximately 2 months.

FEMA’s Disaster Assistance Improvement Plan: a subsidiary project—site usability enhancements—that included enhancements to the DisasterAssistance.gov website to improve usability by making it easier and more intuitive for users to apply for and find information about disaster assistance from federal, state, local, tribal, and private nonprofit organizations was delayed.

Causes of Investment Cost and Schedule Shortfalls Vary

The primary causes of the shortfalls in cost and schedule associated with DHS’s 21 major IT investments were (in descending order of frequency): inaccurate preliminary cost and schedule estimates, technical issues in the development phase, changes in agency priorities, lack of understanding of user requirements, and dependencies on other investments that had schedule shortfalls. A summary of these causes by investment and the associated component are shown in table 3 and are followed by (1) our analysis of these causes by category and (2) discussion of our past work on the department’s major investments and related IT management processes where we identified some of these same causes and made recommendations to strengthen management in these areas.

Specifically, our analysis of these causes by category showed: Inaccurate preliminary cost and schedule estimates: Inaccurate cost and schedule estimates in eight investments resulted in significant cost and schedule increases. For example:

Preliminary schedule estimates for a project under CBP’s Non- Intrusive Inspection Systems Program investment—which supports the detection and prevention of contraband from entering the country—were inaccurate due to underestimating the time needed to complete a key task. Specifically, project officials did not accurately estimate how long it would take to complete an environmental assessment because they did not consider all requirements in their initial planning, thus resulting in a schedule delay of approximately 2 months.

The NPPD investment called Critical Infrastructure Technology and Architecture had a project—integral to developing an information sharing application to be used by federal, state, and local stakeholders to increase their capability to combat terrorist use of improvised explosive devices—where actual costs for completing critical tasks were about 16 percent over the cost estimated at project initiation. According to investment officials, this was due in part to project staff developing the cost estimates very quickly and not fully validating them before proceeding with the project.

TSA’s Hazmat Threat Assessment Program (which performs a threat assessment on commercial truck drivers who transport hazardous materials to determine the threat status to transportation security) had a schedule shortfall with a project, because, in part, the time needed to modify a contract was not accurately estimated, which led to a schedule delay of nearly 3 months.

Technology issues in the development phase: Technical issues in the development phase caused cost or schedule slippages in six investments. Examples include:

Changes made to one part of ICE’s Detention and Removal Operations Modernization investment, which is designed to upgrade IT capabilities to support efficient detention and removal of non-U.S. citizens, created a cascading effect, leading to changes to other parts of the system and contributed to delays of more than a month.

Issues in establishing a testing and development environment that matched the production environment delayed project testing in several projects under FEMA’s Disaster Assistance Improvement Plan investment (which is to ease the burden on disaster survivors by providing them with a mechanism to access and apply for disaster assistance).

Technical complications during deployment caused the schedule to slip by 79 days on a project under CBP’s Land Border Integration investment, which assists with the processing of inbound and outbound travel at border patrol checkpoints nationwide. Specifically, the handheld devices used for scanning license plates used a wireless spectrum that had interference problems at certain sites, and resolving this issue took more time than had been planned for.

Changes in agency priorities: Four investments experienced cost and schedule slippages due to changing priorities at the agency level. In particular,

The schedules were delayed for two NPPD US-VISIT investments: the Arrival and Departure Information System, which collects arrival and departure information on non-U.S. citizens traveling to the United States as well as current immigration status updates for each traveler, and the Automated Biometric Identification System, a fingerprint repository and biometric-matching system. Delays were due to a management decision to focus on accelerating the development of other investments or projects, which took resources (i.e., personnel) away from the investment. Consequently, the Arrival and Departure Information System’s fiscal year 2011 maintenance release project was delayed approximately 3 months, and the Automated Biometric Identification System’s fiscal year 2011 product support project was delayed by approximately 7 months.

A critical subsidiary project to deliver predictive analytical capabilities under USCG’s Business Intelligence investment, which is designed to reduce organizational uncertainty and risk in decision making, had a schedule delay of approximately 3 months due to changing priorities. Project officials said that Coast Guard management directed resources to other projects with a higher priority, thus limiting the ability to work on the predictive analytics capability project.

Lack of understanding user requirements: Three investments had slippages resulting from misunderstanding or inadequately developed user requirements and expectations.

A project under USCIS’s Claims 4 investment, which is a processing system for the adjudication of naturalization applications, was delayed by 2 weeks because inadequate user requirements led to a design flaw that required additional time to address.

Customer priorities and expectations for ICE’s Detention and Removal Operations Modernization investment changed over time, which contributed to schedule delays of more than a month.

The schedule for a CBP TECS Modernization investment project, which supports the screening of travelers entering the United States, was delayed due to users requesting that the application in development interface with a separate system. The project was delayed by 3 months while program officials developed new requirements.

Dependencies on other component’s investments that had schedule shortfalls: Investments also encountered schedule slippages when interdependent investments encountered delays. For example:

USSS’s Information Integration and Technology Transformation investment to provide advanced security measures to electronically send, receive, and track access to USSS’s unclassified and classified information was delayed approximately 6 months due to a component’s project being delayed.

Costs for a project under FEMA’s Disaster Assistance Improvement Plan investment rose approximately 27 percent ($210,000) due, in part, to the delayed deployment of another investment.

Other causes of cost and schedule slippages that were cited by department officials included delays in receiving funding and gaps in leadership due to key management turnover. Specifically,

The schedule for three projects under TSA’s Federal Air Marshal Service Mission Scheduling and Notification investment was delayed due to delays in receiving full funding. DHS had provided the investment with partial funding, and thus investment officials produced an investment plan based on that funding level; when full funding was subsequently restored, the plan had to be updated, which resulted in delays.

The costs for a key subsidiary project of NPPD’s Infrastructure Security Compliance, Chemical Security Assessment Tool, which is to provide for the electronic submission of chemical facility data and controlled use of such data, rose approximately 20 percent ($719,000) due, in part, to multiple director-level program changes, which led to corresponding changes in the investment’s vision and direction.

In our past work on DHS’s investments and related IT management processes, we have identified some of these same causes and made recommendations to strengthen management in these areas. For example, with regard to cost estimating, we reported that forming a reliable estimate of costs provides a sound basis for measuring against actual cost performance and that the lack of such a basis contributes to variances. To help agencies establish such a capability, we issued a guide in March 2009 that was based on the practices of leading organizations. In a July 2012 report examining how well DHS is implementing these practices, we reported that the department had weaknesses in cost estimating. Accordingly, we made recommendations to DHS to strengthen its cost estimating capabilities, and the department has plans and efforts under way to implement our recommendations.

GAO, Department of Homeland Security: Assessments of Selected Complex Acquisitions, GAO-10-588SP (Washington, D.C.: June 2010).

About Half of DHS’s Investments with Shortfalls Do Not Have Well-Developed Corrective Efforts

A variety of best practices exist to guide the successful acquisition of IT investments, including how to develop and document corrective actions for projects experiencing cost and schedule shortfalls. In particular, GAO’s Information Technology Investment Management framework calls for agencies to develop and document corrective efforts for underperforming projects. It also states that agencies are to ensure that, as projects develop and costs rise, the project continues to meet mission needs at the expected levels of cost and risk; if projects are not meeting expectations or if problems have arisen, agencies are to quickly take steps to address the deficiencies. In addition, DHS policy requires corrective actions when cost or schedule variances exceed 8 percent.

DHS developed and documented corrective efforts for 12 of the 21 major investments with a shortfall, but the remaining 9 did not have documented corrective efforts. Table 4 depicts the investments with shortfalls and whether corrective efforts had been developed and documented.

DHS took corrective actions to address investment shortfalls in 12 investments. Actions for the 12 included:

CBP: Automated Commercial Environment /International Trade Data System had schedule shortfalls due to an inadequate testing and development environment; they were resolved by leveraging CBP’s disaster recovery site to perform the testing.

CBP: Land Border Integration investment schedule delays due to technical complications were resolved through the use of risk management processes (e.g., identification, assessment, tracking, and mitigation of risks) identified in the investment’s July 2011 risk management plan, which addressed the cause of the investment shortfalls.

CBP: Non-Intrusive Inspection Systems Program, which had schedule shortfalls due to inaccurate estimates, tracked the project status and risks via project health status reports and other mitigation strategies.

CBP: The Remote Video Surveillance System investment officials developed and documented an investment rebaseline to address the investment’s schedule shortfall, which was due to an inaccurate initial project schedule estimate.

CBP: The TECS Modernization investment—which had schedule shortfalls due to system development being delayed due to questions about whether planned enhancements duplicated functions performed by another agency system—program officials briefed key management on the differences between the system functions and development was allowed to continue.

FEMA: Disaster Assistance Improvement Plan cost and schedule shortfalls were due to, among other things, dependencies on other investments. DHS developed a remediation plan for each shortfall to limit the negative impact.

ICE: Detention and Removal Operations Modernization investment schedule shortfalls were due in part to a lack of understanding of user requirements; to address these issues, investment officials worked with key stakeholders to engage users to more thoroughly identify user requirements.

NPPD: Critical Infrastructure Technology and Architecture investment’s cost shortfalls, which were due to inaccurate initial cost estimates, were resolved by investment officials through several corrective efforts, including completing the project’s life cycle cost estimate.

NPPD: Infrastructure Security Compliance, Chemical Security Assessment Tool investment schedule was delayed due to multiple changes in leadership and in the investment’s direction. Project officials developed and documented an investment rebaseline, which was approved in February 2012. It was intended to, among other things, develop a more accurate schedule.

TSA: The Hazmat Threat Assessment Program investment schedule was delayed because the time needed to adjust a contract had not been accurately estimated. In response, investment officials documented an investment rebaseline, which was approved in March 2012.

TSA: The Security Technology Integrated Program investment had schedule shortfalls from inaccurate estimates of the time needed to revise a contract. To resolve these issues, officials developed and documented an initiative to improve methods used to identify and track risks and resolve the schedule shortfalls. This effort is intended to help the investment avoid additional schedule changes.

USSS: The Information Integration and Technology Transformation investment had one project with schedule shortfalls due to dependencies on another component’s investment that had schedule slippages; issues were addressed by following the mitigation actions detailed in the investment’s risk management plan.

With regard to the remaining nine investments, three were unable to provide us with documentation, even though project officials stated that they had developed some corrective efforts, and six did not engage in corrective efforts to address shortfalls. Of the three investments, officials from TSA’s Federal Air Marshal Service Mission Scheduling and Notification System investment, for example, reported that they had addressed the project’s schedule shortfall—which was due, in part, to a support contractor not having adequate staffing—by performing the work within the agency instead of relying on the contractor. Further, according to TSA officials, the cost and schedule shortfalls on the Air Cargo Security investment, which were due to technical complications and dependencies on other investments, were addressed by establishing a new cost and schedule baseline. Nonetheless, this lack of documentation is inconsistent with the direction of DHS’s guidance and related best practices, and it shows a lack of process discipline and attention to key details, which raises concern about the thoroughness of corrective efforts.

Of the six investments without any corrective efforts, officials from these investments (namely, the Office of the Chief Information Officer’s Human Resources IT investment, NPPD’s US-VISIT Automated Biometric Identification System and Arrival and Departure Information System investments, USCG’s Business Intelligence investment, NPPD’s National Cybersecurity Protection System, and USCIS’s Claims 4 investment), stated that they did not develop and document corrective efforts because they believed DHS’s guidance does not call for it in their circumstances. Specifically, the officials said that although DHS’s guidance calls for corrective actions to be developed and documented when an investment or its projects experiences a life cycle cost or schedule variance of 8 percent or greater, the variances on their project activities thus far were not large enough to constitute such a life cycle variance.

The impact of this is that multiple projects can continue to experience shortfalls—which increases the risk that investments will experience serious lifecycle cost and schedule variances—without having to develop and document corrective actions and thus alert top management about potential problems and associated risks. This is inconsistent with the direction of OMB, which requires agencies to report (via the IT Dashboard) on the cost and schedule performance of their projects and considers those projects with a 10 percent or greater variance to be at an increased level of risk of not being able to deliver promised capabilities on time and within budget, and thus they require special attention from management. It is also inconsistent with our best practices research and experience at federal agencies, which stresses that agencies report to management when projects are not meeting expectations or when problems arise and quickly develop and document corrective efforts to address the problems. Further, our research and work at agencies has shown that waiting to act until significant life cycle variances occur can sometimes be risky and costly, as life cycle schedules are typically for multiyear periods, allowing the potential for underperforming projects to continue to vary from their cost and schedule goals for an extended amount of time without any requirement for corrective efforts. Consequently, until these guidance shortcomings are addressed and each underperforming project has defined and documented corrective actions, the department’s major investments these projects support will be at an increased risk of cost and schedule shortfalls.

Conclusions

Most of the projects comprising DHS’s 68 major IT investments are meeting their cost and schedule commitments, but 21 major investments—integral to DHS’s mission and costing approximately $1 billion—have projects that are experiencing significant cost and schedule shortfalls. These shortfalls place these investments at increased risk of not delivering promised capabilities on time and within budget, which, in turn, pose a risk to DHS’s ability to fully meet its mission of securing the homeland. DHS guidance does not require projects experiencing significant cost and schedule shortfalls to develop and document corrective efforts until they cause a life cycle cost and schedule variance. This increases risk and is contrary to effective IT investment practices. Given that DHS is currently establishing and implementing new IT governance processes, the department is positioned to address the guidance shortfalls.

Recommendations for Executive Action

We recommend that the Secretary of Homeland Security direct the appropriate officials to:

Establish guidance that provides for developing corrective efforts for major IT investment projects that are experiencing cost and schedule shortfalls of 10 percent or greater, similar to those identified in this report.

Ensure that major IT investment projects with shortfalls of 10 percent or greater have defined and documented corrective efforts.

Agency Comments and Our Evaluation

In its written comments signed by the Director for the Departmental GAO- OIG Liaison Office and reprinted in appendix IV, DHS concurred with our recommendations and estimated that it would implement the first recommendation by September 30, 2013, and the second one immediately. It also commented that the department was pleased that the report positively acknowledged that DHS (1) is meeting cost and schedule commitments for most of its major IT investments and (2) has plans and efforts under way to improve cost estimating capabilities and implement a center of excellence for requirements engineering. The department also provided technical comments, which we have incorporated where appropriate.

We are sending copies of this report to interested congressional committees and the Secretary of Homeland Security. In addition, the report is available at no charge on the GAO website at http://www.gao.gov.

If you or your staffs have any questions on the matters discussed in this report, please contact me at (202) 512-9286 or pownerd@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix V.

Appendix I: Objectives, Scope, and Methodology

The objectives of our review were to determine the (1) extent to which Department of Homeland Security (DHS) IT investments are meeting their cost and schedule commitments, (2) primary causes of any commitment shortfalls, and (3) adequacy of DHS’s efforts to address these shortfalls and causes.

To address our first objective, we analyzed how each of DHS’s 68 major investments under development was performing against its cost and schedules commitments, as reported by the agency to the Office of Management and Budget (OMB) for inclusion on OMB’s federal IT Dashboard. More specifically, we analyzed the extent to which each of these investments had met or exceeded, as of March 2012, cost and schedule commitments established when the investment was initiated. In doing this, we identified investments that had a project exceeding 10 percent of its cost and schedule commitments. We focused on these investments and their subsidiary projects because OMB considers them to be at an increased level of risk of not being able to deliver promised capabilities on time and within budget, and thus requiring special attention from management.

To assess the reliability of the IT Dashboard data we analyzed, we corroborated the data by interviewing investment and other DHS officials to determine whether the information on the dashboard was consistent with that reported by DHS. In addition, we followed up on the status of implementation of previous GAO recommendations to improve the quality Specifically, we analyzed of information on OMB’s federal IT Dashboard.plans and related documentation describing efforts by DHS to increase the scrutiny and quality of data submitted to the IT Dashboard. As part of this, we also interviewed department officials including those from the Office of the Chief Information Officer who are responsible for reviewing and submitting DHS’s investment cost and schedule data to the federal IT Dashboard. The documentation and interviews provided us a level of assurance that the data we used for this engagement were, in fact, reliable.

For our second objective, we used a structured interview instrument to survey the DHS and component officials responsible for the investments experiencing cost and schedule shortfalls in order to identify the causes of the shortfalls. As part of surveying these officials, we analyzed project and related documentation to corroborate the causes reported to us via the survey. We then analyzed these causes for commonalities, grouped them accordingly, and tallied the frequency of each cause by investment. In addition, we compared the causes to our prior reports on major DHS investments and related IT management processes to identify the extent to which we had made recommendations to address the causes associated with the department’s investment cost and schedule shortfalls.

To address our third objective, we initially identified and reviewed relevant criteria on developing and documenting corrective actions to address investment shortfalls. Specifically, these criteria included

DHS’s Acquisition Directive 102 (AD-102),

DHS’s Capital Planning and Investment Control Guide, and

GAO’s Information Technology Investment Management guide.

We then used a structured interview instrument to survey DHS and component officials responsible for those investments experiencing shortfalls; we used the survey to identify whether any corrective actions had been developed and documented to address investment shortfalls. We also reviewed investment planning and execution documentation (e.g., project plans, project status reports, program meeting minutes, and acquisition program baselines) to corroborate information provided by the officials during the survey process. We then compared these corrective efforts to the criteria to identify any gaps and in those cases where there were, we reviewed documentation and interviewed agency officials to assess the reason for the gaps and any negative impacts.

GAO, Information Technology Investment Management: A Framework for Assessing and Improving Process Maturity, GAO-04-394G (Washington, D.C.: Mar. 1, 2004). obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: List of DHS Major IT Investments that Are Meeting Cost and Schedule Commitments

Table 5 lists the DHS major IT investments that were meeting their cost and schedule commitments.

Appendix III: List of DHS Major IT Investments and Subsidiary Projects with Shortfalls

Table 6 lists the DHS major IT investments with cost and/or schedule shortfalls, their costs and subsidiary projects, whether or not they had a cost and/or schedule shortfall, and their planned costs.

Appendix IV: Comments from the Department of Homeland Security

Appendix V: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, the following staff also made key contributions to this report: Gary Mountjoy (assistant director), Scott Borre, Camille Chaires, and Nancy Glover.