Information Technology: DHS Needs to Improve Its Independent Acquisition Reviews
Released: 2011-07-28T00:00:00

Letter

Background

DHS’s mission is to lead the unified national effort to secure America by preventing and deterring terrorist attacks and protecting against and responding to threats and hazards to the nation. DHS is also responsible for ensuring that the nation’s borders are safe and secure, welcoming lawful immigrants and visitors, and promoting the free flow of commerce.

Created in 2003, DHS assumed control of about 209,000 civilian and military positions from 22 agencies and offices that specialize in one or more aspects of homeland security. The intent behind the merger that created DHS was to improve coordination, communication, and information sharing among these multiple federal agencies. Figure 1 shows DHS’s organizational structure; table 1 identifies DHS’s principal organizations and describes their missions.

Not since the creation of the Department of Defense in 1947 has the federal government undertaken a transformation of this magnitude. As we reported before the department was created, such a transformation is critically important and poses significant management and leadership challenges. For these reasons, we designated the implementation of the department and its transformation as high risk in 2003, and we continue to do so today. In this regard, we have stated that failure to effectively address DHS’s management challenges and program risks could have serious consequences for our national security.

DHS IT Acquisitions and Their Management

In support of its organizational transformation and in response to the nation’s evolving security needs, DHS has been spending billions of dollars each year to develop and acquire IT systems to perform both mission-critical and support functions, which frequently must be coordinated among components, as well as among external entities. For fiscal year 2010, DHS expected to spend approximately $6.3 billion on 348 IT-related programs, which included 53 major IT acquisition programs that were designated for oversight by the DHS Under Secretary for Management. We refer to these 53 programs as “large acquisitions” throughout this report. Table 2 describes the programs relevant to this review.

In order to manage these acquisitions, the department finalized an acquisition life cycle and review process in 2010, and established the Management Directorate, headed by the Under Secretary for Management, which houses both the Chief Information Officer (CIO) and Chief Procurement Officer (CPO). The CIO’s responsibilities include setting departmental IT policies, processes, and standards, and ensuring that IT acquisitions comply with DHS IT management processes, technical requirements, and approved enterprise architecture, among other things. Additionally, the CIO chairs DHS’s Chief Information Officer Council, which is responsible for ensuring the development of IT resource management policies, processes, best practices, performance measures, and decision criteria for managing the delivery of IT services and investments, while controlling costs and mitigating risks. The CPO is the department’s senior procurement executive, who has leadership and authority over DHS acquisition and contracting, including major investments. The CPO office’s responsibilities include issuing acquisition policies and implementation instructions, overseeing acquisition and contracting functions, and ensuring that a given acquisition’s contracting strategy and plans align with the intent of the department’s ARB, the department’s highest investment review board.

DHS’s acquisition management directive defines four acquisition life cycle phases that are to ensure consistent and efficient management, support, review, and approval for programs across the department. Each phase culminates in an ARB review on whether a program is ready to proceed to the next life cycle phase. The ARB’s chairperson is responsible for determining the readiness of a program and for approving the key acquisition documents critical to establishing a program’s business case, operational requirements, acquisition baseline, and testing and support plans. Also, the ARB’s chairperson is responsible for assessing breaches of the acquisition plan’s cost and schedule baselines and directing corrective actions.

Other DHS entities share responsibility for IT management and procurement activities. For example, control of IT management functions is shared by the DHS CIO and CIOs at the major organizational components (e.g., directorates, offices, and agencies). Similarly, DHS relies on a structure of dual accountability and collaboration between the CPO and the heads of DHS components to carry out the acquisition function. DHS components have also designated component acquisition executives to serve as the senior acquisition officials within the components and to be responsible for implementation of management and oversight of all component acquisition processes.

GAO Has Previously Reported That DHS Is Not Effectively Managing and Overseeing Its Major IT Acquisitions

Since its creation, DHS has faced challenges in acquiring large IT systems, leading to cost and schedule overruns on multiple programs. Our November 2008 report on DHS’s oversight of major acquisition programs described several of these challenges. Specifically,  DHS had not effectively implemented or adhered to its investment  DHS had not consistently enforced decisions that were reached by the investment review board because the department did not track whether components and offices had taken the actions required by the board; and two of nine components did not have the required component-level review processes to adequately manage their major investments.

Accordingly, we made a series of recommendations to DHS to address weaknesses in departmentwide acquisition policies and practices and with individual programs, such as reinstating the department’s oversight board to review and approve acquisition requirements and assess potential duplication of effort, and directing that component heads establish a mechanism to ensure that major investments comply with established component and departmental investment review policy standards. The department generally concurred with our findings and recommendations, citing actions that had been taken and efforts under way to improve the investment review process.

In September 2009 and again in June of 2010, we reported on the status of DHS’s acquisition improvement efforts and on selected major acquisition programs. Despite some progress, we found that many of DHS’s major system acquisitions were still not receiving effective oversight and that DHS continued to face challenges in fully defining and implementing key system investment and acquisition management policies and procedures. Among other things, we noted that the ARB had begun to meet more frequently than in the past and had reviewed dozens of major acquisition programs, but more than 40 programs had not been reviewed, and programs did not consistently implement review action items by established deadlines;  nearly 80 percent of major programs lacked basic acquisition documents, and a database established to track key program information—such as cost and schedule performance and program risks—relied on self-reported program data rather than independently verified data; and component acquisition review processes were not fully in place, and components’ efforts to implement department oversight directives and sufficiently staff the associated processes were not yet complete.

We concluded that, while the department had made progress in establishing key institutional acquisition and IT investment management- related controls and implementing them on large-scale programs, considerable effort remained before the department could be considered a mature IT system acquirer and investor and that our prior recommendations continued to provide the department with a framework to guide its efforts.

IV&V Can Reduce Risk in Developing and Acquiring Large-Scale IT Systems

IV&V is a process whereby organizations can reduce the risks inherent in system development and acquisition efforts by having a knowledgeable party who is independent of the developer determine that the system or product meets the users’ needs and fulfills its intended purpose. IV&V involves proactively determining early in a program’s life cycle what its risks are likely to be, and then identifying those that could be mitigated or lessened by performing additional reviews and quality assessments. IV&V activities can help ensure that quality is built into program deliverables from the beginning—starting with business and requirements analysis, continuing through software development and unit-testing activities, and ending with system and integration testing and acceptance.

We have previously identified IV&V as a leading practice for large and complex system development and acquisition programs. In addition, a review published in 1999 by the Institute of Electrical and Electronics Engineers (IEEE) found that IV&V had a measurable beneficial effect on a program’s development. For example, it  promoted the earlier detection of system faults, identified a greater number of faults,  helped to reduce the average time it takes to fix faults, and  enhanced operational correctness.

The study concluded that any process that systematically applies well- designed IV&V activities to a structured software development process would result in similar benefits.

Typically, IV&V is performed by an agent that is independent of the development organization to obtain unbiased reviews of a system’s processes, products, and results, with the goal of verifying and validating that these meet stated requirements, standards, and user needs. As such, we have reported that IV&V is work above and beyond the normal quality assurance and performance review activities performed during system development and acquisition. This work must not substitute for the developer’s responsibility, but should complement and reinforce the developer’s system engineering processes, configuration management, and qualification test functions. According to recognized industry standards, IV&V can provide management with an objective assessment of a program’s processes, products, and risks throughout its life cycle and help ensure conformance to program performance, schedule, and budget targets. Furthermore, it can help facilitate the early detection and correction of system anomalies and support the system’s conformance to performance, schedule, and budget goals, among other benefits.

We have previously identified the independence of the responsible agent as a key aspect of IV&V’s value to the IT acquisitions process. Independence is defined by the following three components:  Technical independence–requires the effort to be performed by personnel who are not involved in the development of the system. This ensures that the IV&V team brings a fresh viewpoint to the analysis of the system development process and its products.  Managerial independence–requires that the agent be managed separately from the development and program management organizations. The effort must be allowed to freely select the system components or segments it will analyze and test, and the test and analysis techniques it will use. The agent must also be allowed to freely report its findings to program management, without prior approval from the development group.  Financial independence–requires that the funding for IV&V be controlled by an organization separate from the development organization. This ensures that the effort will not be curtailed by having its funding diverted to other program needs, and that financial pressures cannot be used to influence the effort.

An IV&V effort that exhibits all three of these characteristics is fully independent (see fig. 2). Rigorous independence from the development or acquisition effort ensures that IV&V’s insights into a program’s processes and associated work products are objective.

Verification and validation (V&V) are related processes intended to provide evidence that developed or acquired products meet specified requirements and that they fulfill their intended use when placed in their intended environment, respectively. V&V practitioners gather this information through the assessment, analysis, evaluation, review, inspection, and testing of system engineering products and processes. In other words, verification ensures that “you built the product right,” while validation ensures that “you built the right product.”

Table 3 illustrates IV&V activities and work products for a typical development/acquisition life cycle.

IV&V activities may also focus on program management activities and work products across the development/acquisition life cycle. For example, the agent may be involved in the program’s risk management efforts by identifying new risks, or by providing recommendations to eliminate, reduce, or mitigate risks. The agent may also provide an independent view of the program’s progress in terms of its ability to meet cost, schedule, or performance commitments.

Congress has recognized the value of IV&V, in that it has previously required its implementation as one of several conditions for the obligation of funds for certain acquisitions at DHS. For example, Congress directed the department to certify that an IV&V agent was under contract as a condition for obligating funds in fiscal year 2007 for ACE, SBInet, and the U.S. Visitor and Immigrant Status Indicator Technology program. In addition, the Deputy Administrator of E-Government and Information Technology at the Office of Management and Budget told us that he has seen the value of the practice during their reviews of major investments, although its use is not required at federal agencies.

Our review of leading practices from industry and across the federal government identified several key elements of effective IV&V, which are described here along with examples we obtained from examining the policies of several federal departments and agencies contacted during this review.  Decision criteria. When deciding to perform IV&V, risk-based criteria should be used to determine which programs, or aspects of programs, should be subject to review. In other words, the determination to conduct IV&V and its extent should be made on the basis of the relative mission criticality of the program and its components, as well as on the potential impacts to the program from undetected system errors, immaturity of the technology to be used, and unreliability of program schedule and cost estimates, among other program risks. For example, NASA policy states that the IV&V Board of Advisors provides recommendations to the Chief, Safety and Mission Assurance, for implementing IV&V on specific programs, based on specific criteria such as technical complexity, human safety, consequences of failure, program costs, and required time frames. The chief then authorizes IV&V for the programs with the highest risk profiles.  Standards for independence. Organizations should also include standards that describe the degree of technical, managerial, and financial independence required of the personnel or agents performing IV&V. Having standards for independence helps to ensure that the results of activities are reported to program oversight officials, as well as to program management. In this regard, NASA has established an agencywide program for managing all of the system software IV&V efforts. The program includes an internal organization that functions as their IV&V agent and that has no technical, managerial, or financial ties to the development organization.  Defined scope of the effort. The effort should document which program development or acquisition activities will be subject to IV&V. Examples of such activities may include: requirements evaluation, concept/design evaluation, risk evaluation, risk management procedures evaluation, configuration management procedures evaluation, test evaluation, operational readiness evaluation, and cost estimate evaluation. Further, compliance criteria should be established for each activity. For example, NASA’s IV&V technical framework has defined assessment procedures for various system development activities, along with related pass/fail criteria.  Required program resources. Plans should identify the required personnel, funding, facilities, tools, and methods that will be required to perform the activities necessary for the defined scope of the IV&V effort. For example, the Federal Bureau of Investigation requires identification of staff, tools, and training necessary to perform IV&V activities and to develop and maintain work products.  Management and oversight. As with any investment, organizations should conduct proper management and oversight of their IV&V efforts. For example, in order to effectively manage the effort, the roles and responsibilities of all parties involved should be specified and a process for responding to issues raised by the effort should be defined. Several agencies we spoke with had established policies that defined the roles and responsibilities of parties involved in their IV&V process. For example, the Federal Bureau of Investigation’s policy defines the relationship between the program manager, the developer/integrator, and the contractor, including distribution channels for program artifacts, assessments, and deliverables.

Further, organizations should also provide the means for senior management to obtain timely information regarding the progress of their IV&V investments in terms of cost, capability, timeliness, and quality.

Concerning IV&V oversight, organizations should evaluate the effectiveness of their efforts. A variety of guidance recommends that organizations should actively monitor service providers to ensure that they are effective in delivering services and meeting requirements. Moreover, organizations should ensure that sufficient information about their IV&V investments is maintained to support current and future investment decisions and to highlight lessons learned. For example, NASA has found over the years that the application of rigorous IV&V has provided a positive return on investment, and has taken steps to assess the quality and consistency of their efforts through its Technical Quality and Excellence group, which examines IV&V results across all projects and ensures that efforts were conducted in accordance with approved guidelines and standards.

DHS’s IT Acquisition Policy Does Not Incorporate Key Elements of Effective IV&V

Adoption of IV&V can provide agencies with information to better manage their IT investments. To be effective, leading industry practices and government guidance recommend, among other things, that organizations adopt certain key elements of effective IV&V.

DHS’s Acquisition Guidebook recognizes IV&V as a leading practice, recommends (though generally does not require) its use, and cites the IEEE standard for V&V as the basis for IV&V. However, DHS’s policy contains key gaps or ambiguities relative to each of the key elements of effective IV&V.  Decision criteria. DHS policy does not specify a risk-based approach, does not define related criteria for making decisions regarding IV&V, and does not require component agencies to do so. Specifically, the department does not establish risk-based decision making criteria in its Acquisition Guidebook for determining whether, or the extent to which, IT programs should use it and does not require that programs conduct assessments against such criteria.  Standards for independence. DHS acquisition policy does not address the independence of agents. DHS’s policy does not define the required degree of independence that agents must demonstrate and does not require that its component agencies define such standards for themselves. Consequently, the policy does not specify mechanisms to ensure that efforts on major IT acquisitions are adequately objective. Moreover, the policy does not establish reporting mechanisms to ensure that the results of activities are reported to program oversight officials for use in DHS’s investment management process.  Defined scope of the effort. DHS does not require that the specific scope of efforts be defined. While department policy suggests performing IV&V on life cycle activities such as requirements definition, requirements management, and operational readiness activities, department policy does not require that its component agencies or acquisition programs critically assess their IT acquisition programs to determine and document the appropriate scope of IV&V efforts for each program. In addition, the policy does not require that such efforts identify and document compliance criteria for the validation and verification activities.  Required program resources. DHS acquisition policy does not require (or require that its component agencies ensure) that IT acquisitions identify and document the resources needed to execute their efforts— including facilities and tools. It is also silent on other essential aspects of planning the effort, including funding and human resources.  Management and oversight. DHS policy does not address IV&V management or the need to effectively oversee the department’s investment in this practice. While DHS policy assigns certain responsibilities for agents and government officials, it does not require a process for responding to issues raised by the effort and does not require that its component agencies or their acquisition programs do so. In addition, officials at both the Office of the CIO and Office of the CPO stated that DHS does not track which programs across the department employ IV&V unless a program is under a congressional mandate to do so. Further, DHS officials stated that they do not measure the effectiveness of IV&V efforts across the department. Thus, department officials were unaware whether or the extent to which IV&V was being used by the largest IT acquisition programs. They were also unaware of the department’s total expenditures for IV&V, or if those expenditures (which total approximately $91 million across the eight programs we reviewed in detail) are producing satisfactory results.

Officials from the Office of the Chief Information Officer said that they attempted to address IV&V in their 2010 acquisition policy, but they agreed that the policy still contains gaps relative to how it is currently planned, executed, and overseen across the department. They stated that this was due to limited resources and other priorities. Further, they acknowledged that these gaps should be addressed and that the current policy was being revised.

Until DHS provides a clear departmentwide policy requiring programs to employ the key elements of effective IV&V, it is less likely to achieve the full potential of such efforts on its large acquisitions. Consequently, IV&V may not provide the intended benefits of ensuring that DHS’s IT systems and their components meet quality standards, satisfy user needs, and operate as intended. Furthermore, in the absence of a clearly articulated risk-based decision framework for undertaking IV&V, applying its results, and evaluating its effectiveness, DHS’s investments in IV&V efforts are unlikely to provide optimal value for the department and, in some cases, may even fail to deliver any significant benefits.

DHS Reports Widespread Use of IV&V, but Implementation of Key Elements Is Limited

Many large IT acquisitions from across DHS report using IV&V as part of their acquisition and/or development process. However, despite reports of this widespread use, we found that the department did not consistently implement key elements of IV&V on eight major IT acquisition programs. For example, none of the eight used a structured, risk-based decision making process when deciding if, when, and how to use IV&V. In part, these weaknesses can be attributed to the lack of clear departmentwide policy requiring the application of such elements. As a result, DHS’s inconsistent use of IV&V may not reliably and significantly contribute toward meeting the schedule and mission goals of the department’s major IT programs.

DHS Reports Widespread Use of IV&V on Large IT Programs

DHS’s large IT programs reported widespread use of IV&V as part of their acquisition and/or development processes. Specifically, 35 of 41 major IT acquisition programs from DHS’s oversight list reported that IV&V efforts were planned, under way, or completed. The specific IV&V activities reported for each program are listed in appendix II, along with other descriptive program information.

The 35 programs reported using IV&V across a range of program activities. For example, 26 of the programs reported performing IV&V on at least half of the life cycle activities listed in our questionnaire. Requirements validation and verification and operational readiness were the most commonly reported activities (reported by 27 of the 35 programs responding); risk management was the least reported activity (reported by 19 programs). (See fig. 3 for total responses on the activities we specifically identified in our questionnaire.) A few programs reported other IV&V activities, such as assessment of standards compliance; readiness of integrated logistics support; assessment of equipment usability; contract auditing; and compliance with the system engineering life cycle.

To accomplish IV&V activities, program officials reported obtaining expertise from several different sources: commercial firms, federally funded research and development centers, internal resources, and other federal agencies. Some programs reported obtaining IV&V services from multiple sources. (For further information reported by DHS about the programs and their IV&V efforts, see app. II.)

DHS Has Not Consistently Applied Key Elements of Effective IV&V on Selected Programs

Despite DHS’s reported widespread use of IV&V, the eight programs selected for our review did not consistently implement the elements of effective IV&V. These programs—ACE, TASC, ITP, TECS-Mod, NCPS, ITIP, C4ISR, and Transformation—all reported that they planned and performed IV&V on system development and/or acquisition activities throughout their respective program life cycles, at a total estimated cost of about $91 million. (See app. III for a description of the IV&V efforts and costs of these programs, as reported by DHS.) However, our review of documents and artifacts for these programs determined that, in most cases, there were gaps in their implementation of IV&V. Notably, one program—NCPS—demonstrated almost none of the elements of IV&V leading practices. Table 4 summarizes the extent to which each program implemented key elements of effective IV&V. A high-level discussion of implementation across the programs, with selected examples, follows the table. Appendix IV provides the detailed results of our analysis.  Decision criteria. None of the eight programs had fully established decision criteria to guide their IV&V efforts. The five programs that partially met our criteria determined how or when to apply IV&V results to improve the program’s management, but they did not establish and use a risk-based approach for deciding whether or to what extent to use IV&V. For example, TASC officials told us that they meet weekly to review key findings and determine how they can improve the management of the program, but that they did not follow a structured, risk-based process in deciding to use IV&V. The remaining three programs did not incorporate either of these aspects into their program decision processes.  Standards for independence. Each of the programs at least partially addressed the independence of their IV&V agents, but none of them ensured full technical, managerial, and financial independence. For example, TECS-Mod requires that the IV&V contractor provide written certification of its technical, managerial, and financial independence, but the effort is not managerially independent because, according to officials, the contractor is overseen by the TECS-Mod program manager. In another example, the statement of work for Transformation’s IV&V effort requires the agent to be technically independent, but it does not address financial or managerial independence.  Defined scope of the effort. Almost all of the programs defined the scope of their IV&V effort to at least some degree. However, only one fully defined its scope. ITP’s IV&V plan describes activities subject to IV&V in the concept, requirements, design, and testing phases of the program and includes V&V compliance criteria for all its IV&V activities. Six programs partially defined their scope. Although they defined and documented their IV&V activities, they did not define the related compliance criteria for all activities. For example, the Transformation program identified 15 tasks the IV&V agent is to perform, such as reviewing requirements management and test and evaluation activities, but it did not define all of the required compliance criteria for these tasks. The eighth program—NCPS—did not address either aspect of scope. It documented a high-level description of desired IV&V services, but it did not define the specific activities to be performed or the related evaluation compliance criteria.  Required program resources. Just over half of the programs defined the resources required for their IV&V effort to at least some degree. However, only one fully defined them. Transformation identified the personnel needs, facilities, and tools that were needed to support its IV&V activities, for example, by listing certification requirements for personnel. On the other hand, the C4ISR’s IV&V statement of work defined the program’s needs for security and test-related activities, but did not define its resource needs for other planned IV&V activities, such as cost estimation and performance verification. Three programs did not specify the resources required for their efforts.  Management and oversight. Seven of the eight programs established some degree of management and oversight for their efforts, although each contained gaps. For example, ACE’s IV&V plan and the statement of work note that the agent is to report its results to the program, but do not identify a process for how ACE will respond to such issues. In addition, the responsibilities of ACE’s agent are defined; however, the roles and responsibilities for government officials and evaluating the effectiveness of the IV&V effort were not specified.

These weaknesses in IV&V efforts can be partly attributed to the fact that DHS’s acquisition policy does not require that programs apply recognized practices to their IV&V efforts. Performing IV&V without an established framework for planning and managing the effort could result in duplicative, unnecessary, or potentially ineffective IV&V efforts. As a result, DHS risks not maximizing the value of its investment in IV&V, in turn making it less likely that IV&V will contribute significantly toward meeting the schedule and mission goals of its major IT acquisition programs.

Conclusions

DHS spends billions of dollars on large IT acquisitions in support of its national security mission, including millions on independent reviews of these programs. Investment in IV&V by a large number of these programs reflects a view across DHS that it is a worthwhile acquisition practice, a view also represented in DHS’s acquisition policy. However, because DHS has not provided guidance for planning, executing, and overseeing the elements of this practice across the department nor required its components to do so, it lacks consistent approaches and criteria for determining whether and how to proceed with IV&V on programs, specifying the needed independence of agents, defining the scope of efforts, planning and procuring the needed resources, and managing and utilizing results.

Not surprisingly, none of the high-budget acquisition programs that we reviewed had fully implemented the key elements of effective IV&V. Executing such efforts without a disciplined framework for planning and management may make such efforts duplicative, unnecessary, or unusable. Moreover, without well-defined mechanisms for tracking IV&V efforts, results, and effectiveness, and incorporating this information into the department’s investment management processes, DHS’s investment decisions may not adequately take into account the concerns raised by these efforts. To realize IV&V’s promise as a tool for reducing the risks inherent in developing IT systems, DHS needs to promote a common understanding of effective IV&V across the department, and through its oversight activities, ensure that component agencies and their large IT programs conduct efforts that consistently contribute toward meeting IT acquisition cost, schedule, and mission goals.

Recommendations for Executive Action

To help guide consistent and effective execution of IV&V at DHS, we recommend that the Secretary of Homeland Security direct the department CIO and CPO to take the following three actions:  Revise DHS acquisition policy such that it establishes risk-based criteria for (1) determining which major and other high- risk IT acquisition programs should conduct IV&V and (2) selecting appropriate activities for independent review of these programs; requirements for technical, financial, and managerial independence of agents; standards and guidance for defining and documenting plans and products; controls for planning, managing, and overseeing efforts;  mechanisms to ensure that plans and significant findings inform DHS acquisition program reviews and decisions, including those of the ARB; and  mechanisms to monitor and ensure implementation of this policy on applicable new IT acquisition programs.  Reevaluate the approach to IV&V for ongoing programs (including the eight programs featured in this report) and ensure that appropriate actions are taken to bring each of them into alignment with the elements of leading practice.  Collect and analyze data on IV&V efforts for major IT acquisition programs to facilitate the development of lessons learned and evaluation of the effectiveness of DHS’s investments, and establish a process that uses the results to inform the department’s IT investment decisions.

Agency Comments and Our Evaluation

In written comments on a draft of this report, signed by the Director, Departmental GAO/Office of Inspector General Liaison and reprinted in appendix V, DHS stated that it concurred with our recommendations and described actions planned or under way to address them. Regarding our first recommendation, the department stated that it is drafting an interim IV&V policy implementation plan that will outline best practices, templates, tools, and processes for IV&V. The interim plan will also require programs to develop IV&V plans early in their life cycle, and to assess programs at their conclusion to ensure that all IV&V artifacts, processes, and systems had been developed properly. Further, the response stated that DHS components will be expected to use the department’s implementation plan to tailor IV&V activities based on program size, complexity, risk, and other program management factors. In addition, DHS stated that it is considering modifications to its existing guidance to reflect IV&V industry standards and best practices, and to demonstrate requirements for the independence of IV&V agents as called for in this report. Regarding our second recommendation, the department responded that it is creating an independent team of subject matter experts to provide oversight of IV&V efforts across DHS. This team is to determine whether appropriate resources, tools, and facilities have been allocated, and will report results as necessary. Given the limited extent to which the programs we reviewed are currently employing the key elements of effective IV&V, expeditiously establishing this team and conducting the reviews would help identify the full extent of the need for improvements departmentwide. Concerning our third recommendation, the department’s response stated that DHS will create a repository to store data about its IV&V efforts in order to generate lessons learned and gauge the effectiveness of these efforts, among other things. The department also provided technical comments, which we have incorporated in the report, as appropriate.

We are sending copies of this report to the appropriate congressional committees; the Secretary of the Department of Homeland Security; and other interested parties. In addition, this report is available at no charge on our Web site at http://www.gao.gov.

If you or your staff members have any questions on the matters discussed in this report, please contact me at (202) 512-9286 or pownerd@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix VI.

Appendix I: Objectives, Scope, and Methodology

Our objectives were to determine (1) how the Department of Homeland Security’s (DHS) independent verification and validation (IV&V) policies and procedures for information technology (IT) acquisitions compare with leading practices and (2) the extent to which DHS has implemented IV&V on its large IT system acquisitions.

To determine how DHS’s IV&V policies and procedures for IT acquisitions compare with leading practices, we first identified key elements of leading practices for IV&V. Specifically, we reviewed (1) the Software Engineering Institute’s Capability Maturity Model® Integration, focusing on the validation and verification process areas, (2) the Institute of Electrical and Electronics Engineers (IEEE) standard for software verification and validation, (3) the IEEE/ International Organization for Standardization/ International Electrotechnical Commission standard for system life cycle processes, (4) the International Organization for Standardization standard for software life cycle processes, and (5) our prior work. Within these documents, we identified validation, verification, and independence concepts and practices that these sources have in common. We then categorized the concepts and practices into the following key elements of leading practice for IV&V: (1) decision criteria, (2) effort independence, (3) project scope, (4) project resources, and (5) management and oversight.

We also examined how IV&V is used by other federal agencies to provide context for our review of DHS. We selected the additional agencies by identifying those that had the highest average IT spending per investment for fiscal years 2008 to 2010. They are: the Department of Commerce, the Department of Defense, the Department of Energy, the Department of Homeland Security, the Department of Justice, the National Aeronautics and Space Administration (NASA), the Social Security Administration, the Department of State, the Department of Transportation, and the Department of Veterans Affairs. We reviewed their policies regarding IV&V and selected examples that were used to illustrate some of the leading IV&V practices that they follow and perform. Specifically, we identified relevant examples from the Department of Justice and NASA.

Next, we held interviews with DHS officials, and gathered and reviewed the department’s policy documents related to IV&V. We then compared the policy and procedures with the five key elements of IV&V leading practice. We used the following rules to characterize the extent to which DHS’s policies addressed the elements:  Met. DHS provided evidence that fully satisfied all aspects of the element.  Partially met. DHS provided evidence that satisfied some, but not all aspects of the element.  Not met. DHS provided evidence that did not satisfy any aspects of the element or provided no evidence.

To determine the extent to which DHS had implemented IV&V on its large IT system acquisitions, we first collected information on the status and program characteristics of the 53 level 1 and 2 IT acquisitions listed in DHS’s Major Acquisitions Oversight List of May 26, 2010. To do so, we requested from program officials their respective program’s life cycle approach, estimated acquisition costs, and planned IV&V activities. During our review, 12 programs were defunded, recategorized to level 3 or non-IT, or taken off the oversight list by DHS and therefore were not analyzed for this report. We used the self-reported program data to populate table 5 in appendix II and to select a subset of programs for more detailed analysis of IV&V implementation.

We used three criteria to identify programs for further study. First, the selected programs were to represent IV&V implementation across a variety of DHS components. Second, the selected programs would be among DHS’s largest (level 1). Third, the selected programs would have the highest estimated acquisition cost within each of DHS’s components. Thus, we selected the largest program (based on estimated acquisition cost) from each DHS component that reported having at least one level 1 IT acquisition that reported using IV&V—with one exception. Since we have previously issued detailed reports on the Coast Guard’s largest program, Rescue 21, we instead selected U.S. Coast Guard’s Command, Control, Communications, Computers, Intelligence, Surveillance, and Reconnaissance (C4ISR) program. Using this approach, we selected the eight IT acquisitions featured in appendixes III and IV.

Next, we collected and analyzed IV&V related documents and information from each of these programs and conducted follow-up interviews with cognizant officials to clarify documentation and elaborate their responses. We then compared these data with key elements of effective IV&V and scored the programs using the previously described scoring methodology.

To assess the reliability of the data that was used to support the findings in the report, we reviewed relevant program and agency documentation to substantiate evidence obtained through interviews with knowledgeable agency officials. We validated that the documents we used in this review were current and officially issued by conferring with DHS and component agency officials in meetings and in the formal exit conference. On this basis, we determined that the data used in this report are sufficiently reliable. We appropriately attributed the sources of data we used throughout this report. This includes sections in which data is self- reported, such as figure 3, table 4, appendix II, and appendix III.

We conducted this performance audit at GAO headquarters in Washington, D.C., from March 2010 to July 2011, in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: DHS’s Large IT Acquisition Programs and Their Reported Use of IV&V

To characterize the extent to which DHS has implemented IV&V on its large IT acquisitions, DHS department, component, and program officials provided the data in table 5 for the level 1 and level 2 IT acquisition programs listed in the major acquisition oversight list issued by the Under Secretary for Management on May 26, 2010. Of the 41 programs, 35 reported planning, conducting, or completing some type of IV&V activity.

The decision makers for conducting IV&V on these programs— congressional mandate, departmental decision, or others—are summarized in the discussion that follows and in table 6.

DHS officials also provided information about the origins of their IV&V decisions for these programs. For the 18 level 1 acquisitions that reported IV&V activities, Congress mandated that 2 of the acquisitions perform IV&V; DHS required that 7 of the programs use IV&V; and the component, program, or other entity decided to perform IV&V on 9 of the acquisitions. For the 17 level 2 acquisitions that reported IV&V activities, none were congressionally mandated; DHS required IV&V for 2 of the acquisitions; and component, programs, or other entities decided to perform IV&V on 15 acquisitions. Table 6 summarizes the decision makers for conducting IV&V on level 1 and level 2 programs.

Appendix III: Overview of DHS’s Reported Use of IV&V on Selected Large IT Acquisitions

Table 7 summarizes key characteristics of eight large DHS IT acquisitions and their associated IV&V efforts and costs, as reported by DHS program officials for this review. The systems engineering life cycle stages identified in the table are further discussed in the context of DHS’s acquisition life cycle after the table.

DHS’s Acquisition Instruction/Guidebook establishes a four-phase acquisition life cycle. These life cycle phases are: (1) identify a capability need; (2) analyze and select the means to provide that capability; (3) obtain the capability; and (4) produce, deploy, and support the capability. These phases generally align with one or more systems engineering life cycle phases.

The directive requires the Acquisition Review Board (ARB) to review each major acquisition program at least three times at key Acquisition Decision Events during a program’s acquisition life cycle. Selected documents considered during Acquisition Decision Events, such as the mission need statement, the acquisition plan, and the integrated logistics support plan, are depicted in the following figure, along with the associated systems engineering life cycle phases.

Appendix IV: Assessments of Selected Programs’ Implementation of IV&V

This appendix presents brief program overviews and our assessment of DHS’s implementation of IV&V on eight select large IT acquisitions compared with key elements of effective IV&V.

U.S. Customs and Border Protection (CBP)— Automated Commercial Environment (ACE)/ International Trade Data System

ACE is a commercial trade processing system intended to facilitate the movement of legitimate trade, strengthen border security, and replace existing systems with a single system for collecting and providing trade data to federal agencies. It is a level 1 acquisition with a life cycle cost of approximately $4.5 billion. Of the total life cycle cost, approximately $2.8 million has been budgeted for IV&V. ACE has been divided into 11 segments, consisting of one in the planning stage, two in the development stage, and eight in the operations and maintenance stage.

ACE officials reported that the following IV&V activities are under way or planned: independent cost estimation or cost estimate validation;  program progress or performance verification or evaluation; requirements validation or verification; concept or design validation, verification, or alternatives analysis; security vulnerability or security risk assessment; component verification testing and evaluation; system or integration verification testing and evaluation; and  operational readiness testing and evaluation.

Table 8 describes the extent to which ACE has implemented the key elements of effective IV&V.

TASC was announced in 2007 and is intended to modernize, transform, and integrate the various financial acquisition and asset management systems in use at the department’s components. It is a level 1 acquisition and has a life cycle cost estimate of approximately $991 million, with $3.4 million budgeted for IV&V.

TASC reported that it was in the solutions engineering phase of its life cycle and that IV&V is planned or under way in the following activities:  program progress or performance verification or evaluation; requirements validation or verification; concept or design validation verification, or alternatives analysis; security vulnerability or security risk assessments; component verification testing and evaluation; system or integration verification testing and evaluation; and  operational readiness testing and evaluation.

Table 9 describes the extent to which the program has implemented the key elements of effective IV&V.

ITP is intended to contribute to DHS’s consolidated infrastructure investment, supporting areas such as data center, network, and e-mail consolidation. With a life cycle cost of approximately $1.2 billion, ITP is a level 1 acquisition. The program has four segments; one segment is in the planning stage, and three are in the operations and maintenance phase.

Officials from the Office of the Chief Information Officer report that the IV&V agent for ITP is currently performing the following program activities: independent cost estimation or cost estimate validation;  program progress or performance verification or evaluation; requirements validation or verification; concept or design validation verification, or alternatives analysis; security vulnerability or security risk assessments; component verification testing and evaluation; system or integration verification testing and evaluation; and  operational readiness testing and evaluation.

Table 10 describes the extent to which the program has implemented the key elements of effective IV&V.

TECS-Mod is intended to modernize the system ICE uses to perform investigative activities. Specifically, TECS-Mod involves modernizing the investigative case management system and related support modules of the legacy TECS system. ICE’s total combined life cycle cost is estimated at approximately $1.1 billion and an estimated $1.75 million for IV&V efforts.

The program is a level 1 acquisition and is currently in the requirements definition phase. According to ICE officials, the IV&V agents for TECS-Mod are to perform the following IV&V activities: independent cost estimation or cost estimate validation;  program progress or performance verification or evaluation; requirements validation or verification; concept or design validation, verification, or alternatives analysis; security vulnerability or security risk assessments; component verification testing and evaluation; system or integration verification testing and evaluation;  operational readiness testing and evaluation; and software design and development request for proposals.

Table 11 describes the extent to which the program has implemented the key elements of effective IV&V.

The NCPS program is intended to reduce the federal government’s vulnerabilities to cyber threats by decreasing the frequency of cyberspace disruptions and minimizing the duration and damage of those disruptions. It is classified as a level 1 acquisition with a total estimated life cycle cost of approximately $1.2 billion. The program is structured in five segments, three of which are under development—one in the planning stage, one in design, and one in integration. Two segments have been completed and are in the operations and maintenance phase.

NPPD officials reported that the following IV&V activities are either planned, under way, or have been completed: independent cost estimation or cost estimate validation; requirements validation or verification; concept or design validation, verification, or alternatives analysis; risk evaluation; security vulnerability or security risk assessments; component verification testing and evaluation; and system or integration verification testing and evaluation.

Table 12 describes the extent to which the program has implemented the key elements of effective IV&V.

ITIP is intended to provide comprehensive technical infrastructure support for TSA in four main program areas: (1) office automation, (2) infrastructure, (3) program management, and (4) contract support. It is a level 1 acquisition, with a life cycle cost of approximately $3.5 billion. ITIP is currently in operations and maintenance.

ITIP officials reported that the following IV&V activities are under way: independent cost estimation or cost estimate validation;  program progress or performance verification or evaluation; requirements validation or verification; security vulnerability or security risk assessments; configuration management verification; and  operational readiness testing and evaluation.

Table 13 describes the extent to which the program has implemented the key elements of effective IV&V.

USCG’s C4ISR program is intended to be an interoperable network that combines information from USCG assets and sensors, allowing the USCG to see, comprehend, and communicate rapidly. It is a level 1 acquisition with a life cycle cost of approximately $1.3 billion (according to its 2011 budget submission) and is currently in the development phase. Of this cost, about $20.6 million is budgeted for IV&V.

USCG officials report that the IV&V agent is performing the following IV&V activities: independent cost estimation or cost estimate validation;  program progress or performance verification or evaluation; security vulnerability or security risk assessments; component verification testing and evaluation; system or integration verification testing and evaluation; and  operational readiness testing and evaluation.

Table 14 describes the extent to which the program has implemented the key elements of effective IV&V.

Transformation is a 5-year effort to modernize business processes and information technology throughout USCIS. The goal of the program is to move USCIS from a paper-based filing system to a centralized and electronic filing system. It is a level 1 acquisition, and its current life cycle cost estimate is $1.7 billion; however, the LCCE is under review. The program has budgeted approximately $62 million for IV&V services. There are five segments of the program; four segments are in the planning phase, and one is in the requirements definition phase.

USCIS reports that the IV&V agent is currently reviewing or plans to review the following activities: independent cost estimation or cost estimate validation;  program progress or performance verification or evaluation; requirements validation or verification; concept or design validation, verification, or alternatives analysis; security vulnerability or security risk assessments; component verification testing and evaluation; system or integration verification testing and evaluation; and  operational readiness testing and evaluation.

Table 15 describes the extent to which the program has implemented the key elements of effective IV&V.

Appendix V: Comments from the Department of Homeland Security

Appendix VI: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the individual named above, the following staff also made key contributions to this report: Paula Moore (Assistant Director), Neil Doherty, Lynn Espedido, Rebecca Eyler, Nancy Glover, Daniel Gordon, Jim Houtz, Justin Palk, and Shawn Ward.