Information Technology: DHS Needs to Improve Its Independent Acquisition Reviews
Published: 2011-07-28T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Since its creation in 2003, the Department of Homeland Security (DHS) has been developing new information technology (IT) systems to perform both mission-critical and support functions; however, it has faced challenges in developing these systems. One way to manage the inherent risks of developing and acquiring systems is through independent verification and validation (IV&V)--a process conducted by a party independent of the development effort that provides an objective assessment of a project's processes, products, and risks throughout its life cycle and helps ensure that program performance, schedule, and budget targets are met. GAO was asked to determine (1) how DHS's IV&V policies and procedures for IT acquisitions compare with leading practices and (2) the extent to which DHS has implemented IV&V on its large IT system acquisitions. To do so, GAO assessed DHS's policy against industry standards and leading practice guides, as well as analyzed how eight selected IT programs had implemented IV&V.
section_title: What GAO Found
paragraphs: DHS recognizes the importance of IV&V and recommends its use on major IT programs. Nevertheless, its acquisition policy does not address the elements of leading practices for IV&V. Specifically, the department has not established risk-based decision making criteria for determining whether, or the extent to which, programs should utilize IV&V. In addition, department policy does not define the degree of independence required of agents and does not require that programs determine and document the planned scope of their efforts, including the program activities subject to review; the resources required; roles and responsibilities; and how the results will be reported and acted upon. Moreover, the policy does not address overseeing DHS's investment in IV&V. Thus, officials were unaware of the extent to which it was being used on major IT acquisition programs, associated expenditures, or if those expenditures are producing satisfactory results. Absent such policy elements and more effective oversight, the department's investments in IV&V efforts are unlikely to provide optimal value for the department and, in some cases, may even fail to deliver any significant benefits. Many large IT acquisition programs across DHS reported using IV&V as part of their acquisition and/or development processes. Nevertheless, the eight major IT acquisition programs that GAO analyzed did not consistently implement the elements of leading practice. For example, the eight did not fully apply a structured, risk-based decision making process when deciding if, when, and how to utilize IV&V. In part, these weaknesses are due to the lack of clear departmentwide guidance regarding the use of such practices. As a result, the department's IV&V efforts may not consistently contribute toward meeting IT acquisition cost, schedule, and mission goals.
section_title: What GAO Recommends
paragraphs: GAO recommends that DHS (1) update its acquisition policy to reflect elements of effective IV&V, (2) monitor and ensure implementation of this policy on applicable new and ongoing IT programs, and (3) collect data on IV&V usage and use it to evaluate the effectiveness of these investments. DHS concurred with GAO's recommendations and described actions planned or under way to address them.

section_title: Letter
paragraphs: The Department of Homeland Security (DHS) is charged with leading  national efforts to secure America by deterring terrorist attacks, ensuring  the nation’s borders are safe and secure, and welcoming lawful  immigrants and visitors, among other tasks. After it began operations in  March 2003, DHS began developing information technology (IT) systems  to perform both mission-critical and support functions. These systems  included the acquisition of an integrated financial management system,  the IT infrastructure to support the Secure Border Initiative Network  (SBInet) “virtual fence” along the nation’s southwest border, and the  Coast Guard’s (USCG) Rescue 21 system that supports its search and  rescue operations off our nation’s shores. DHS has faced challenges in  developing these and other systems, which have resulted in schedule  delays, cost increases, and not delivering the sought-after capabilities.
Independent verification and validation (IV&V) is a process whereby  organizations can reduce the risks inherent in system development and  acquisition efforts by having a knowledgeable party who is independent of  the developer determine whether the system or product meets the users’  needs and fulfills its intended purpose. We have previously recognized  the use of IV&V as a leading practice for federal agencies in the  acquisition of programs that are variously complex, large-scale, or high  risk. Congress has also previously required its implementation as one of  several conditions for the obligation of funds for several high risk DHS IT  acquisitions, including U.S. Customs and Border Protection’s (CBP)  SBInet and Automated Commercial Environment (ACE).
As agreed, our objectives were to determine (1) how DHS’s IV&V policies  and procedures for IT acquisitions compare with leading practices and (2)  the extent to which DHS has implemented IV&V on its large IT system  acquisitions. To accomplish this, we researched the IV&V policies of  recognized leading practices guides, industry standards, and other  federal departments and agencies; analyzed relevant DHS department  and component-level policies and guidance; and conducted interviews  with relevant department and component-level officials. We then identified  eight large IT system acquisitions from DHS components for further study  using specific criteria. For these eight, we analyzed relevant program  documentation, including IV&V plans, statements of work and reports  from IV&V service providers, and DHS’s Acquisition Review Board (ARB)  decision memoranda.
We conducted this performance audit from March 2010 to July 2011, in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives. See appendix I for a complete  description of our objectives, scope, and methodology.
subsections: section_title: Background
paragraphs: DHS’s mission is to lead the unified national effort to secure America by  preventing and deterring terrorist attacks and protecting against and  responding to threats and hazards to the nation. DHS is also responsible  for ensuring that the nation’s borders are safe and secure, welcoming  lawful immigrants and visitors, and promoting the free flow of commerce.
Created in 2003, DHS assumed control of about 209,000 civilian and  military positions from 22 agencies and offices that specialize in one or  more aspects of homeland security. The intent behind the merger that  created DHS was to improve coordination, communication, and  information sharing among these multiple federal agencies. Figure 1  shows DHS’s organizational structure; table 1 identifies DHS’s principal  organizations and describes their missions.
Not since the creation of the Department of Defense in 1947 has the  federal government undertaken a transformation of this magnitude. As we  reported before the department was created, such a transformation is  critically important and poses significant management and leadership  challenges. For these reasons, we designated the implementation of the  department and its transformation as high risk in 2003, and we continue  to do so today. In this regard, we have stated that failure to effectively  address DHS’s management challenges and program risks could have  serious consequences for our national security.
subsections: section_title: DHS IT Acquisitions and Their Management
paragraphs: In support of its organizational transformation and in response to the  nation’s evolving security needs, DHS has been spending billions of  dollars each year to develop and acquire IT systems to perform both  mission-critical and support functions, which frequently must be  coordinated among components, as well as among external entities. For  fiscal year 2010, DHS expected to spend approximately $6.3 billion on  348 IT-related programs, which included 53 major IT acquisition programs  that were designated for oversight by the DHS Under Secretary for  Management. We refer to these 53 programs as “large acquisitions”  throughout this report. Table 2 describes the programs relevant to this  review.
In order to manage these acquisitions, the department finalized an  acquisition life cycle and review process in 2010, and established the  Management Directorate, headed by the Under Secretary for  Management, which houses both the Chief Information Officer (CIO) and  Chief Procurement Officer (CPO). The CIO’s responsibilities include  setting departmental IT policies, processes, and standards, and ensuring  that IT acquisitions comply with DHS IT management processes,  technical requirements, and approved enterprise architecture, among  other things. Additionally, the CIO chairs DHS’s Chief Information Officer  Council, which is responsible for ensuring the development of IT resource  management policies, processes, best practices, performance measures,  and decision criteria for managing the delivery of IT services and  investments, while controlling costs and mitigating risks. The CPO is the  department’s senior procurement executive, who has leadership and  authority over DHS acquisition and contracting, including major  investments. The CPO office’s responsibilities include issuing acquisition  policies and implementation instructions, overseeing acquisition and  contracting functions, and ensuring that a given acquisition’s contracting  strategy and plans align with the intent of the department’s ARB, the  department’s highest investment review board.
DHS’s acquisition management directive defines four acquisition life cycle  phases that are to ensure consistent and efficient management, support,  review, and approval for programs across the department. Each phase  culminates in an ARB review on whether a program is ready to proceed to  the next life cycle phase. The ARB’s chairperson is responsible for  determining the readiness of a program and for approving the key  acquisition documents critical to establishing a program’s business case,  operational requirements, acquisition baseline, and testing and support  plans. Also, the ARB’s chairperson is responsible for assessing breaches  of the acquisition plan’s cost and schedule baselines and directing  corrective actions.
Other DHS entities share responsibility for IT management and  procurement activities. For example, control of IT management functions  is shared by the DHS CIO and CIOs at the major organizational  components (e.g., directorates, offices, and agencies). Similarly, DHS  relies on a structure of dual accountability and collaboration between the  CPO and the heads of DHS components to carry out the acquisition  function. DHS components have also designated component acquisition  executives to serve as the senior acquisition officials within the  components and to be responsible for implementation of management  and oversight of all component acquisition processes.
section_title: GAO Has Previously Reported That DHS Is Not Effectively Managing and Overseeing Its Major IT Acquisitions
paragraphs: Since its creation, DHS has faced challenges in acquiring large IT  systems, leading to cost and schedule overruns on multiple programs.  Our November 2008 report on DHS’s oversight of major acquisition  programs described several of these challenges. Specifically,    DHS had not effectively implemented or adhered to its investment    DHS had not consistently enforced decisions that were reached by  the investment review board because the department did not track  whether components and offices had taken the actions required by  the board; and  two of nine components did not have the required component-level  review processes to adequately manage their major investments.
Accordingly, we made a series of recommendations to DHS to address  weaknesses in departmentwide acquisition policies and practices and  with individual programs, such as reinstating the department’s oversight  board to review and approve acquisition requirements and assess  potential duplication of effort, and directing that component heads  establish a mechanism to ensure that major investments comply with  established component and departmental investment review policy  standards. The department generally concurred with our findings and  recommendations, citing actions that had been taken and efforts under  way to improve the investment review process.
In September 2009 and again in June of 2010, we reported on the  status of DHS’s acquisition improvement efforts and on selected major  acquisition programs. Despite some progress, we found that many of  DHS’s major system acquisitions were still not receiving effective  oversight and that DHS continued to face challenges in fully defining and  implementing key system investment and acquisition management  policies and procedures. Among other things, we noted that  the ARB had begun to meet more frequently than in the past and had  reviewed dozens of major acquisition programs, but more than 40  programs had not been reviewed, and programs did not consistently  implement review action items by established deadlines;    nearly 80 percent of major programs lacked basic acquisition  documents, and a database established to track key program  information—such as cost and schedule performance and program  risks—relied on self-reported program data rather than independently  verified data; and  component acquisition review processes were not fully in place, and  components’ efforts to implement department oversight directives and  sufficiently staff the associated processes were not yet complete.
We concluded that, while the department had made progress in  establishing key institutional acquisition and IT investment management- related controls and implementing them on large-scale programs,  considerable effort remained before the department could be considered  a mature IT system acquirer and investor and that our prior  recommendations continued to provide the department with a framework  to guide its efforts.
section_title: IV&V Can Reduce Risk in Developing and Acquiring Large-Scale IT Systems
paragraphs: IV&V is a process whereby organizations can reduce the risks inherent in  system development and acquisition efforts by having a knowledgeable  party who is independent of the developer determine that the system or  product meets the users’ needs and fulfills its intended purpose. IV&V  involves proactively determining early in a program’s life cycle what its  risks are likely to be, and then identifying those that could be mitigated or  lessened by performing additional reviews and quality assessments. IV&V  activities can help ensure that quality is built into program deliverables  from the beginning—starting with business and requirements analysis,  continuing through software development and unit-testing activities, and  ending with system and integration testing and acceptance.
We have previously identified IV&V as a leading practice for large and  complex system development and acquisition programs. In addition, a  review published in 1999 by the Institute of Electrical and Electronics  Engineers (IEEE) found that IV&V had a measurable beneficial effect on  a program’s development. For example, it    promoted the earlier detection of system faults,  identified a greater number of faults,    helped to reduce the average time it takes to fix faults, and    enhanced operational correctness.
The study concluded that any process that systematically applies well- designed IV&V activities to a structured software development process  would result in similar benefits.
Typically, IV&V is performed by an agent that is independent of the  development organization to obtain unbiased reviews of a system’s  processes, products, and results, with the goal of verifying and validating  that these meet stated requirements, standards, and user needs. As  such, we have reported that IV&V is work above and beyond the normal  quality assurance and performance review activities performed during  system development and acquisition. This work must not substitute for  the developer’s responsibility, but should complement and reinforce the  developer’s system engineering processes, configuration management,  and qualification test functions. According to recognized industry  standards, IV&V can provide management with an objective  assessment of a program’s processes, products, and risks throughout its  life cycle and help ensure conformance to program performance,  schedule, and budget targets. Furthermore, it can help facilitate the early  detection and correction of system anomalies and support the system’s  conformance to performance, schedule, and budget goals, among other  benefits.
We have previously identified the independence of the responsible agent  as a key aspect of IV&V’s value to the IT acquisitions process.  Independence is defined by the following three components:    Technical independence–requires the effort to be performed by  personnel who are not involved in the development of the system.  This ensures that the IV&V team brings a fresh viewpoint to the  analysis of the system development process and its products.    Managerial independence–requires that the agent be managed  separately from the development and program management  organizations. The effort must be allowed to freely select the system  components or segments it will analyze and test, and the test and  analysis techniques it will use. The agent must also be allowed to  freely report its findings to program management, without prior  approval from the development group.    Financial independence–requires that the funding for IV&V be  controlled by an organization separate from the development  organization. This ensures that the effort will not be curtailed by  having its funding diverted to other program needs, and that financial  pressures cannot be used to influence the effort.
An IV&V effort that exhibits all three of these characteristics is fully  independent (see fig. 2). Rigorous independence from the development  or acquisition effort ensures that IV&V’s insights into a program’s  processes and associated work products are objective.
Verification and validation (V&V) are related processes intended to  provide evidence that developed or acquired products meet specified  requirements and that they fulfill their intended use when placed in their  intended environment, respectively. V&V practitioners gather this  information through the assessment, analysis, evaluation, review,  inspection, and testing of system engineering products and processes. In  other words, verification ensures that “you built the product right,” while  validation ensures that “you built the right product.”
Table 3 illustrates IV&V activities and work products for a typical  development/acquisition life cycle.
IV&V activities may also focus on program management activities and  work products across the development/acquisition life cycle. For example,  the agent may be involved in the program’s risk management efforts by  identifying new risks, or by providing recommendations to eliminate,  reduce, or mitigate risks. The agent may also provide an independent  view of the program’s progress in terms of its ability to meet cost,  schedule, or performance commitments.
Congress has recognized the value of IV&V, in that it has previously  required its implementation as one of several conditions for the obligation  of funds for certain acquisitions at DHS. For example, Congress directed  the department to certify that an IV&V agent was under contract as a  condition for obligating funds in fiscal year 2007 for ACE, SBInet, and the  U.S. Visitor and Immigrant Status Indicator Technology program. In  addition, the Deputy Administrator of E-Government and Information  Technology at the Office of Management and Budget told us that he has  seen the value of the practice during their reviews of major investments,  although its use is not required at federal agencies.
Our review of leading practices from industry and across the federal  government identified several key elements of effective IV&V, which are  described here along with examples we obtained from examining the  policies of several federal departments and agencies contacted during  this review.    Decision criteria. When deciding to perform IV&V, risk-based criteria  should be used to determine which programs, or aspects of programs,  should be subject to review. In other words, the determination to  conduct IV&V and its extent should be made on the basis of the relative  mission criticality of the program and its components, as well as on the  potential impacts to the program from undetected system errors,  immaturity of the technology to be used, and unreliability of program  schedule and cost estimates, among other program risks. For example,  NASA policy states that the IV&V Board of Advisors provides  recommendations to the Chief, Safety and Mission Assurance, for  implementing IV&V on specific programs, based on specific criteria  such as technical complexity, human safety, consequences of failure,  program costs, and required time frames. The chief then authorizes  IV&V for the programs with the highest risk profiles.    Standards for independence. Organizations should also include  standards that describe the degree of technical, managerial, and  financial independence required of the personnel or agents  performing IV&V. Having standards for independence helps to ensure  that the results of activities are reported to program oversight officials,  as well as to program management. In this regard, NASA has  established an agencywide program for managing all of the system  software IV&V efforts. The program includes an internal organization  that functions as their IV&V agent and that has no technical,  managerial, or financial ties to the development organization.    Defined scope of the effort. The effort should document which  program development or acquisition activities will be subject to IV&V.  Examples of such activities may include: requirements evaluation,  concept/design evaluation, risk evaluation, risk management  procedures evaluation, configuration management procedures  evaluation, test evaluation, operational readiness evaluation, and cost  estimate evaluation. Further, compliance criteria should be  established for each activity. For example, NASA’s IV&V technical  framework has defined assessment procedures for various system  development activities, along with related pass/fail criteria.    Required program resources. Plans should identify the required  personnel, funding, facilities, tools, and methods that will be required  to perform the activities necessary for the defined scope of the IV&V  effort. For example, the Federal Bureau of Investigation requires  identification of staff, tools, and training necessary to perform IV&V  activities and to develop and maintain work products.    Management and oversight. As with any investment, organizations  should conduct proper management and oversight of their IV&V  efforts. For example, in order to effectively manage the effort, the  roles and responsibilities of all parties involved should be specified  and a process for responding to issues raised by the effort should be  defined. Several agencies we spoke with had established policies that  defined the roles and responsibilities of parties involved in their IV&V  process. For example, the Federal Bureau of Investigation’s policy  defines the relationship between the program manager, the  developer/integrator, and the contractor, including distribution  channels for program artifacts, assessments, and deliverables.
Further, organizations should also provide the means for senior  management to obtain timely information regarding the progress of  their IV&V investments in terms of cost, capability, timeliness, and  quality.
Concerning IV&V oversight, organizations should evaluate the  effectiveness of their efforts. A variety of guidance recommends that  organizations should actively monitor service providers to ensure that  they are effective in delivering services and meeting requirements.  Moreover, organizations should ensure that sufficient information about  their IV&V investments is maintained to support current and future  investment decisions and to highlight lessons learned. For example,  NASA has found over the years that the application of rigorous IV&V has  provided a positive return on investment, and has taken steps to assess  the quality and consistency of their efforts through its Technical Quality  and Excellence group, which examines IV&V results across all projects  and ensures that efforts were conducted in accordance with approved  guidelines and standards.
section_title: DHS’s IT Acquisition Policy Does Not Incorporate Key Elements of Effective IV&V
paragraphs: Adoption of IV&V can provide agencies with information to better manage  their IT investments. To be effective, leading industry practices and  government guidance recommend, among other things, that  organizations adopt certain key elements of effective IV&V.
DHS’s Acquisition Guidebook recognizes IV&V as a leading practice,  recommends (though generally does not require) its use, and cites the  IEEE standard for V&V as the basis for IV&V. However, DHS’s policy  contains key gaps or ambiguities relative to each of the key elements of  effective IV&V.    Decision criteria. DHS policy does not specify a risk-based approach,  does not define related criteria for making decisions regarding IV&V,  and does not require component agencies to do so. Specifically, the  department does not establish risk-based decision making criteria in  its Acquisition Guidebook for determining whether, or the extent to  which, IT programs should use it and does not require that programs  conduct assessments against such criteria.    Standards for independence. DHS acquisition policy does not address  the independence of agents. DHS’s policy does not define the  required degree of independence that agents must demonstrate and  does not require that its component agencies define such standards  for themselves. Consequently, the policy does not specify  mechanisms to ensure that efforts on major IT acquisitions are  adequately objective. Moreover, the policy does not establish  reporting mechanisms to ensure that the results of activities are  reported to program oversight officials for use in DHS’s investment  management process.    Defined scope of the effort. DHS does not require that the specific  scope of efforts be defined. While department policy suggests  performing IV&V on life cycle activities such as requirements  definition, requirements management, and operational readiness  activities, department policy does not require that its component  agencies or acquisition programs critically assess their IT acquisition  programs to determine and document the appropriate scope of IV&V  efforts for each program. In addition, the policy does not require that  such efforts identify and document compliance criteria for the  validation and verification activities.    Required program resources. DHS acquisition policy does not require  (or require that its component agencies ensure) that IT acquisitions  identify and document the resources needed to execute their efforts— including facilities and tools. It is also silent on other essential aspects  of planning the effort, including funding and human resources.    Management and oversight. DHS policy does not address IV&V  management or the need to effectively oversee the department’s  investment in this practice. While DHS policy assigns certain  responsibilities for agents and government officials, it does not require  a process for responding to issues raised by the effort and does not  require that its component agencies or their acquisition programs do  so. In addition, officials at both the Office of the CIO and Office of the  CPO stated that DHS does not track which programs across the  department employ IV&V unless a program is under a congressional  mandate to do so. Further, DHS officials stated that they do not  measure the effectiveness of IV&V efforts across the department.  Thus, department officials were unaware whether or the extent to  which IV&V was being used by the largest IT acquisition programs.  They were also unaware of the department’s total expenditures for  IV&V, or if those expenditures (which total approximately $91 million  across the eight programs we reviewed in detail) are producing  satisfactory results.
Officials from the Office of the Chief Information Officer said that they  attempted to address IV&V in their 2010 acquisition policy, but they  agreed that the policy still contains gaps relative to how it is currently  planned, executed, and overseen across the department. They stated  that this was due to limited resources and other priorities. Further, they  acknowledged that these gaps should be addressed and that the current  policy was being revised.
Until DHS provides a clear departmentwide policy requiring programs to  employ the key elements of effective IV&V, it is less likely to achieve the  full potential of such efforts on its large acquisitions. Consequently, IV&V  may not provide the intended benefits of ensuring that DHS’s IT systems  and their components meet quality standards, satisfy user needs, and  operate as intended. Furthermore, in the absence of a clearly articulated  risk-based decision framework for undertaking IV&V, applying its results,  and evaluating its effectiveness, DHS’s investments in IV&V efforts are  unlikely to provide optimal value for the department and, in some cases,  may even fail to deliver any significant benefits.
section_title: DHS Reports Widespread Use of IV&V, but Implementation of Key Elements Is Limited
paragraphs: Many large IT acquisitions from across DHS report using IV&V as part of  their acquisition and/or development process. However, despite reports of  this widespread use, we found that the department did not consistently  implement key elements of IV&V on eight major IT acquisition programs.  For example, none of the eight used a structured, risk-based decision  making process when deciding if, when, and how to use IV&V. In part,  these weaknesses can be attributed to the lack of clear departmentwide  policy requiring the application of such elements. As a result, DHS’s  inconsistent use of IV&V may not reliably and significantly contribute  toward meeting the schedule and mission goals of the department’s major  IT programs.
subsections: section_title: DHS Reports Widespread Use of IV&V on Large IT Programs
paragraphs: DHS’s large IT programs reported widespread use of IV&V as part of their  acquisition and/or development processes. Specifically, 35 of 41 major IT  acquisition programs from DHS’s oversight list reported that IV&V efforts  were planned, under way, or completed. The specific IV&V activities  reported for each program are listed in appendix II, along with other  descriptive program information.
The 35 programs reported using IV&V across a range of program  activities. For example, 26 of the programs reported performing IV&V on  at least half of the life cycle activities listed in our questionnaire.  Requirements validation and verification and operational readiness were  the most commonly reported activities (reported by 27 of the 35 programs  responding); risk management was the least reported activity (reported by  19 programs). (See fig. 3 for total responses on the activities we  specifically identified in our questionnaire.) A few programs reported other  IV&V activities, such as assessment of standards compliance; readiness  of integrated logistics support; assessment of equipment usability;  contract auditing; and compliance with the system engineering life cycle.
To accomplish IV&V activities, program officials reported obtaining  expertise from several different sources: commercial firms, federally  funded research and development centers, internal resources, and other  federal agencies. Some programs reported obtaining IV&V services from  multiple sources. (For further information reported by DHS about the  programs and their IV&V efforts, see app. II.)
section_title: DHS Has Not Consistently Applied Key Elements of Effective IV&V on Selected Programs
paragraphs: Despite DHS’s reported widespread use of IV&V, the eight programs  selected for our review did not consistently implement the elements of  effective IV&V. These programs—ACE, TASC, ITP, TECS-Mod,  NCPS, ITIP, C4ISR, and Transformation—all reported that they planned  and performed IV&V on system development and/or acquisition activities  throughout their respective program life cycles, at a total estimated cost of  about $91 million. (See app. III for a description of the IV&V efforts and  costs of these programs, as reported by DHS.) However, our review of  documents and artifacts for these programs determined that, in most  cases, there were gaps in their implementation of IV&V. Notably, one  program—NCPS—demonstrated almost none of the elements of IV&V  leading practices. Table 4 summarizes the extent to which each program  implemented key elements of effective IV&V. A high-level discussion of  implementation across the programs, with selected examples, follows the  table. Appendix IV provides the detailed results of our analysis.    Decision criteria. None of the eight programs had fully established  decision criteria to guide their IV&V efforts. The five programs that  partially met our criteria determined how or when to apply IV&V  results to improve the program’s management, but they did not  establish and use a risk-based approach for deciding whether or to  what extent to use IV&V. For example, TASC officials told us that they  meet weekly to review key findings and determine how they can  improve the management of the program, but that they did not follow  a structured, risk-based process in deciding to use IV&V. The  remaining three programs did not incorporate either of these aspects  into their program decision processes.    Standards for independence. Each of the programs at least partially  addressed the independence of their IV&V agents, but none of them  ensured full technical, managerial, and financial independence. For  example, TECS-Mod requires that the IV&V contractor provide written  certification of its technical, managerial, and financial independence,  but the effort is not managerially independent because, according to  officials, the contractor is overseen by the TECS-Mod program  manager. In another example, the statement of work for  Transformation’s IV&V effort requires the agent to be technically  independent, but it does not address financial or managerial  independence.    Defined scope of the effort. Almost all of the programs defined the  scope of their IV&V effort to at least some degree. However, only one  fully defined its scope. ITP’s IV&V plan describes activities subject to  IV&V in the concept, requirements, design, and testing phases of the  program and includes V&V compliance criteria for all its IV&V  activities. Six programs partially defined their scope. Although they  defined and documented their IV&V activities, they did not define the  related compliance criteria for all activities. For example, the  Transformation program identified 15 tasks the IV&V agent is to  perform, such as reviewing requirements management and test and  evaluation activities, but it did not define all of the required compliance  criteria for these tasks. The eighth program—NCPS—did not address  either aspect of scope. It documented a high-level description of  desired IV&V services, but it did not define the specific activities to be  performed or the related evaluation compliance criteria.    Required program resources. Just over half of the programs defined  the resources required for their IV&V effort to at least some degree.  However, only one fully defined them. Transformation identified the  personnel needs, facilities, and tools that were needed to support its  IV&V activities, for example, by listing certification requirements for  personnel. On the other hand, the C4ISR’s IV&V statement of work  defined the program’s needs for security and test-related activities,  but did not define its resource needs for other planned IV&V activities,  such as cost estimation and performance verification. Three programs  did not specify the resources required for their efforts.    Management and oversight. Seven of the eight programs established  some degree of management and oversight for their efforts, although  each contained gaps. For example, ACE’s IV&V plan and the  statement of work note that the agent is to report its results to the  program, but do not identify a process for how ACE will respond to  such issues. In addition, the responsibilities of ACE’s agent are  defined; however, the roles and responsibilities for government  officials and evaluating the effectiveness of the IV&V effort were not  specified.
These weaknesses in IV&V efforts can be partly attributed to the fact that  DHS’s acquisition policy does not require that programs apply recognized  practices to their IV&V efforts. Performing IV&V without an established  framework for planning and managing the effort could result in  duplicative, unnecessary, or potentially ineffective IV&V efforts. As a  result, DHS risks not maximizing the value of its investment in IV&V, in  turn making it less likely that IV&V will contribute significantly toward  meeting the schedule and mission goals of its major IT acquisition  programs.
section_title: Conclusions
paragraphs: DHS spends billions of dollars on large IT acquisitions in support of its  national security mission, including millions on independent reviews of  these programs. Investment in IV&V by a large number of these programs  reflects a view across DHS that it is a worthwhile acquisition practice, a  view also represented in DHS’s acquisition policy. However, because  DHS has not provided guidance for planning, executing, and overseeing  the elements of this practice across the department nor required its  components to do so, it lacks consistent approaches and criteria for  determining whether and how to proceed with IV&V on programs,  specifying the needed independence of agents, defining the scope of  efforts, planning and procuring the needed resources, and managing and  utilizing results.
Not surprisingly, none of the high-budget acquisition programs that we  reviewed had fully implemented the key elements of effective IV&V.  Executing such efforts without a disciplined framework for planning and  management may make such efforts duplicative, unnecessary, or  unusable. Moreover, without well-defined mechanisms for tracking IV&V  efforts, results, and effectiveness, and incorporating this information into  the department’s investment management processes, DHS’s investment  decisions may not adequately take into account the concerns raised by  these efforts. To realize IV&V’s promise as a tool for reducing the risks  inherent in developing IT systems, DHS needs to promote a common  understanding of effective IV&V across the department, and through its  oversight activities, ensure that component agencies and their large IT  programs conduct efforts that consistently contribute toward meeting IT  acquisition cost, schedule, and mission goals.
section_title: Recommendations for Executive Action
paragraphs: To help guide consistent and effective execution of IV&V at DHS, we  recommend that the Secretary of Homeland Security direct the  department CIO and CPO to take the following three actions:    Revise DHS acquisition policy such that it establishes  risk-based criteria for (1) determining which major and other high- risk IT acquisition programs should conduct IV&V and (2)  selecting appropriate activities for independent review of these  programs;  requirements for technical, financial, and managerial  independence of agents;  standards and guidance for defining and documenting plans and  products;  controls for planning, managing, and overseeing efforts;    mechanisms to ensure that plans and significant findings inform  DHS acquisition program reviews and decisions, including those  of the ARB; and    mechanisms to monitor and ensure implementation of this policy  on applicable new IT acquisition programs.    Reevaluate the approach to IV&V for ongoing programs (including the  eight programs featured in this report) and ensure that appropriate  actions are taken to bring each of them into alignment with the  elements of leading practice.    Collect and analyze data on IV&V efforts for major IT acquisition  programs to facilitate the development of lessons learned and  evaluation of the effectiveness of DHS’s investments, and establish a  process that uses the results to inform the department’s IT investment  decisions.
section_title: Agency Comments and Our Evaluation
paragraphs: In written comments on a draft of this report, signed by the Director,  Departmental GAO/Office of Inspector General Liaison and reprinted in  appendix V, DHS stated that it concurred with our recommendations and  described actions planned or under way to address them. Regarding our first  recommendation, the department stated that it is drafting an interim IV&V  policy implementation plan that will outline best practices, templates, tools,  and processes for IV&V. The interim plan will also require programs to  develop IV&V plans early in their life cycle, and to assess programs at their  conclusion to ensure that all IV&V artifacts, processes, and systems had  been developed properly. Further, the response stated that DHS  components will be expected to use the department’s implementation plan to  tailor IV&V activities based on program size, complexity, risk, and other  program management factors. In addition, DHS stated that it is considering  modifications to its existing guidance to reflect IV&V industry standards and  best practices, and to demonstrate requirements for the independence of  IV&V agents as called for in this report. Regarding our second  recommendation, the department responded that it is creating an  independent team of subject matter experts to provide oversight of IV&V  efforts across DHS. This team is to determine whether appropriate  resources, tools, and facilities have been allocated, and will report results as  necessary. Given the limited extent to which the programs we reviewed are  currently employing the key elements of effective IV&V, expeditiously  establishing this team and conducting the reviews would help identify the full  extent of the need for improvements departmentwide. Concerning our third  recommendation, the department’s response stated that DHS will create a  repository to store data about its IV&V efforts in order to generate lessons  learned and gauge the effectiveness of these efforts, among other things.  The department also provided technical comments, which we have  incorporated in the report, as appropriate.
We are sending copies of this report to the appropriate congressional  committees; the Secretary of the Department of Homeland Security; and  other interested parties. In addition, this report is available at no charge  on our Web site at http://www.gao.gov.
If you or your staff members have any questions on the matters  discussed in this report, please contact me at (202) 512-9286 or  pownerd@gao.gov. Contact points for our Offices of Congressional  Relations and Public Affairs may be found on the last page of this report.  Key contributors to this report are listed in appendix VI.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: Our objectives were to determine (1) how the Department of Homeland  Security’s (DHS) independent verification and validation (IV&V) policies  and procedures for information technology (IT) acquisitions compare with  leading practices and (2) the extent to which DHS has implemented IV&V  on its large IT system acquisitions.
To determine how DHS’s IV&V policies and procedures for IT acquisitions  compare with leading practices, we first identified key elements of leading  practices for IV&V. Specifically, we reviewed (1) the Software Engineering  Institute’s Capability Maturity Model® Integration, focusing on the  validation and verification process areas, (2) the Institute of Electrical and  Electronics Engineers (IEEE) standard for software verification and  validation, (3) the IEEE/ International Organization for Standardization/  International Electrotechnical Commission standard for system life cycle  processes, (4) the International Organization for Standardization   standard for software life cycle processes, and (5) our prior work. Within  these documents, we identified validation, verification, and independence  concepts and practices that these sources have in common. We then  categorized the concepts and practices into the following key elements of  leading practice for IV&V: (1) decision criteria, (2) effort independence, (3)  project scope, (4) project resources, and (5) management and oversight.
We also examined how IV&V is used by other federal agencies to provide  context for our review of DHS. We selected the additional agencies by  identifying those that had the highest average IT spending per investment  for fiscal years 2008 to 2010. They are: the Department of Commerce,  the Department of Defense, the Department of Energy, the Department of  Homeland Security, the Department of Justice, the National Aeronautics  and Space Administration (NASA), the Social Security Administration, the  Department of State, the Department of Transportation, and the  Department of Veterans Affairs. We reviewed their policies regarding  IV&V and selected examples that were used to illustrate some of the  leading IV&V practices that they follow and perform. Specifically, we  identified relevant examples from the Department of Justice and NASA.
Next, we held interviews with DHS officials, and gathered and reviewed  the department’s policy documents related to IV&V. We then compared  the policy and procedures with the five key elements of IV&V leading  practice. We used the following rules to characterize the extent to which  DHS’s policies addressed the elements:    Met. DHS provided evidence that fully satisfied all aspects of the  element.    Partially met. DHS provided evidence that satisfied some, but not all  aspects of the element.    Not met. DHS provided evidence that did not satisfy any aspects of  the element or provided no evidence.
To determine the extent to which DHS had implemented IV&V on its large  IT system acquisitions, we first collected information on the status and  program characteristics of the 53 level 1 and 2 IT acquisitions listed in  DHS’s Major Acquisitions Oversight List of May 26, 2010. To do so, we  requested from program officials their respective program’s life cycle  approach, estimated acquisition costs, and planned IV&V activities.  During our review, 12 programs were defunded, recategorized to level 3  or non-IT, or taken off the oversight list by DHS and therefore were not  analyzed for this report. We used the self-reported program data to  populate table 5 in appendix II and to select a subset of programs for  more detailed analysis of IV&V implementation.
We used three criteria to identify programs for further study. First, the  selected programs were to represent IV&V implementation across a  variety of DHS components. Second, the selected programs would be  among DHS’s largest (level 1). Third, the selected programs would have  the highest estimated acquisition cost within each of DHS’s components.  Thus, we selected the largest program (based on estimated acquisition  cost) from each DHS component that reported having at least one level 1  IT acquisition that reported using IV&V—with one exception. Since we  have previously issued detailed reports on the Coast Guard’s largest  program, Rescue 21, we instead selected U.S. Coast Guard’s Command,  Control, Communications, Computers, Intelligence, Surveillance, and  Reconnaissance (C4ISR) program. Using this approach, we selected the  eight IT acquisitions featured in appendixes III and IV.
Next, we collected and analyzed IV&V related documents and information  from each of these programs and conducted follow-up interviews with  cognizant officials to clarify documentation and elaborate their responses.  We then compared these data with key elements of effective IV&V and  scored the programs using the previously described scoring methodology.
To assess the reliability of the data that was used to support the findings  in the report, we reviewed relevant program and agency documentation to  substantiate evidence obtained through interviews with knowledgeable  agency officials. We validated that the documents we used in this review  were current and officially issued by conferring with DHS and component  agency officials in meetings and in the formal exit conference. On this  basis, we determined that the data used in this report are sufficiently  reliable. We appropriately attributed the sources of data we used  throughout this report. This includes sections in which data is self- reported, such as figure 3, table 4, appendix II, and appendix III.
We conducted this performance audit at GAO headquarters in  Washington, D.C., from March 2010 to July 2011, in accordance with  generally accepted government auditing standards. Those standards  require that we plan and perform the audit to obtain sufficient, appropriate  evidence to provide a reasonable basis for our findings and conclusions  based on our audit objectives. We believe that the evidence obtained  provides a reasonable basis for our findings and conclusions based on  our audit objectives.
section_title: Appendix II: DHS’s Large IT Acquisition Programs and Their Reported Use of IV&V
paragraphs: To characterize the extent to which DHS has implemented IV&V on its  large IT acquisitions, DHS department, component, and program officials  provided the data in table 5 for the level 1 and level 2 IT acquisition  programs listed in the major acquisition oversight list issued by the Under  Secretary for Management on May 26, 2010. Of the 41 programs, 35  reported planning, conducting, or completing some type of IV&V activity.
The decision makers for conducting IV&V on these programs— congressional mandate, departmental decision, or others—are  summarized in the discussion that follows and in table 6.
DHS officials also provided information about the origins of their IV&V  decisions for these programs. For the 18 level 1 acquisitions that reported  IV&V activities, Congress mandated that 2 of the acquisitions perform  IV&V; DHS required that 7 of the programs use IV&V; and the  component, program, or other entity decided to perform IV&V on 9 of the  acquisitions. For the 17 level 2 acquisitions that reported IV&V activities,  none were congressionally mandated; DHS required IV&V for 2 of the  acquisitions; and component, programs, or other entities decided to  perform IV&V on 15 acquisitions. Table 6 summarizes the decision  makers for conducting IV&V on level 1 and level 2 programs.
section_title: Appendix III: Overview of DHS’s Reported Use of IV&V on Selected Large IT Acquisitions
paragraphs: Table 7 summarizes key characteristics of eight large DHS IT acquisitions  and their associated IV&V efforts and costs, as reported by DHS program  officials for this review. The systems engineering life cycle stages  identified in the table are further discussed in the context of DHS’s  acquisition life cycle after the table.
DHS’s Acquisition Instruction/Guidebook establishes a four-phase  acquisition life cycle. These life cycle phases are: (1) identify a capability  need; (2) analyze and select the means to provide that capability; (3)  obtain the capability; and (4) produce, deploy, and support the capability.  These phases generally align with one or more systems engineering life  cycle phases.
The directive requires the Acquisition Review Board (ARB) to review each  major acquisition program at least three times at key Acquisition Decision  Events during a program’s acquisition life cycle. Selected documents  considered during Acquisition Decision Events, such as the mission need  statement, the acquisition plan, and the integrated logistics support plan,  are depicted in the following figure, along with the associated systems  engineering life cycle phases.
section_title: Appendix IV: Assessments of Selected Programs’ Implementation of IV&V
paragraphs: This appendix presents brief program overviews and our assessment of  DHS’s implementation of IV&V on eight select large IT acquisitions  compared with key elements of effective IV&V.
subsections: section_title: U.S. Customs and Border Protection (CBP)— Automated Commercial Environment (ACE)/ International Trade Data System
paragraphs: ACE is a commercial trade processing system intended to facilitate the  movement of legitimate trade, strengthen border security, and replace  existing systems with a single system for collecting and providing trade  data to federal agencies. It is a level 1 acquisition with a life cycle cost of  approximately $4.5 billion. Of the total life cycle cost, approximately $2.8  million has been budgeted for IV&V. ACE has been divided into 11  segments, consisting of one in the planning stage, two in the  development stage, and eight in the operations and maintenance stage.
ACE officials reported that the following IV&V activities are under way or  planned:  independent cost estimation or cost estimate validation;    program progress or performance verification or evaluation;  requirements validation or verification;  concept or design validation, verification, or alternatives analysis;  security vulnerability or security risk assessment;  component verification testing and evaluation;  system or integration verification testing and evaluation; and    operational readiness testing and evaluation.
Table 8 describes the extent to which ACE has implemented the key  elements of effective IV&V.
TASC was announced in 2007 and is intended to modernize, transform,  and integrate the various financial acquisition and asset management  systems in use at the department’s components. It is a level 1 acquisition  and has a life cycle cost estimate of approximately $991 million, with $3.4  million budgeted for IV&V.
TASC reported that it was in the solutions engineering phase of its life  cycle and that IV&V is planned or under way in the following activities:    program progress or performance verification or evaluation;  requirements validation or verification;  concept or design validation verification, or alternatives analysis;  security vulnerability or security risk assessments;  component verification testing and evaluation;  system or integration verification testing and evaluation; and    operational readiness testing and evaluation.
Table 9 describes the extent to which the program has implemented the  key elements of effective IV&V.
ITP is intended to contribute to DHS’s consolidated infrastructure  investment, supporting areas such as data center, network, and e-mail  consolidation. With a life cycle cost of approximately $1.2 billion, ITP is a  level 1 acquisition. The program has four segments; one segment is in  the planning stage, and three are in the operations and maintenance  phase.
Officials from the Office of the Chief Information Officer report that the  IV&V agent for ITP is currently performing the following program  activities:  independent cost estimation or cost estimate validation;    program progress or performance verification or evaluation;  requirements validation or verification;  concept or design validation verification, or alternatives analysis;  security vulnerability or security risk assessments;  component verification testing and evaluation;  system or integration verification testing and evaluation; and    operational readiness testing and evaluation.
Table 10 describes the extent to which the program has implemented the  key elements of effective IV&V.
TECS-Mod is intended to modernize the system ICE uses to perform  investigative activities. Specifically, TECS-Mod involves modernizing the  investigative case management system and related support modules of  the legacy TECS system. ICE’s total combined life cycle cost is estimated  at approximately $1.1 billion and an estimated $1.75 million for IV&V  efforts.
The program is a level 1 acquisition and is currently in the requirements  definition phase. According to ICE officials, the IV&V agents for   TECS-Mod are to perform the following IV&V activities:  independent cost estimation or cost estimate validation;    program progress or performance verification or evaluation;  requirements validation or verification;  concept or design validation, verification, or alternatives analysis;  security vulnerability or security risk assessments;  component verification testing and evaluation;  system or integration verification testing and evaluation;    operational readiness testing and evaluation; and  software design and development request for proposals.
Table 11 describes the extent to which the program has implemented the  key elements of effective IV&V.
The NCPS program is intended to reduce the federal government’s  vulnerabilities to cyber threats by decreasing the frequency of cyberspace  disruptions and minimizing the duration and damage of those disruptions.  It is classified as a level 1 acquisition with a total estimated life cycle cost  of approximately $1.2 billion. The program is structured in five segments,  three of which are under development—one in the planning stage, one in  design, and one in integration. Two segments have been completed and  are in the operations and maintenance phase.
NPPD officials reported that the following IV&V activities are either  planned, under way, or have been completed:  independent cost estimation or cost estimate validation;  requirements validation or verification;  concept or design validation, verification, or alternatives analysis;  risk evaluation;  security vulnerability or security risk assessments;  component verification testing and evaluation; and  system or integration verification testing and evaluation.
Table 12 describes the extent to which the program has implemented the  key elements of effective IV&V.
ITIP is intended to provide comprehensive technical infrastructure support  for TSA in four main program areas: (1) office automation, (2)  infrastructure, (3) program management, and (4) contract support. It is a  level 1 acquisition, with a life cycle cost of approximately $3.5 billion. ITIP  is currently in operations and maintenance.
ITIP officials reported that the following IV&V activities are under way:  independent cost estimation or cost estimate validation;    program progress or performance verification or evaluation;  requirements validation or verification;  security vulnerability or security risk assessments;  configuration management verification; and    operational readiness testing and evaluation.
Table 13 describes the extent to which the program has implemented the  key elements of effective IV&V.
USCG’s C4ISR program is intended to be an interoperable network that  combines information from USCG assets and sensors, allowing the  USCG to see, comprehend, and communicate rapidly. It is a level 1  acquisition with a life cycle cost of approximately $1.3 billion (according to  its 2011 budget submission) and is currently in the development phase.  Of this cost, about $20.6 million is budgeted for IV&V.
USCG officials report that the IV&V agent is performing the following  IV&V activities:  independent cost estimation or cost estimate validation;    program progress or performance verification or evaluation;  security vulnerability or security risk assessments;  component verification testing and evaluation;  system or integration verification testing and evaluation; and    operational readiness testing and evaluation.
Table 14 describes the extent to which the program has implemented the  key elements of effective IV&V.
Transformation is a 5-year effort to modernize business processes and  information technology throughout USCIS. The goal of the program is to  move USCIS from a paper-based filing system to a centralized and  electronic filing system. It is a level 1 acquisition, and its current life cycle  cost estimate is $1.7 billion; however, the LCCE is under review. The  program has budgeted approximately $62 million for IV&V services.  There are five segments of the program; four segments are in the  planning phase, and one is in the requirements definition phase.
USCIS reports that the IV&V agent is currently reviewing or plans to  review the following activities:  independent cost estimation or cost estimate validation;    program progress or performance verification or evaluation;  requirements validation or verification;  concept or design validation, verification, or alternatives analysis;  security vulnerability or security risk assessments;  component verification testing and evaluation;  system or integration verification testing and evaluation; and    operational readiness testing and evaluation.
Table 15 describes the extent to which the program has implemented the  key elements of effective IV&V.
section_title: Appendix V: Comments from the Department of Homeland Security
section_title: Appendix VI: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the individual named above, the following staff also made  key contributions to this report: Paula Moore (Assistant Director), Neil  Doherty, Lynn Espedido, Rebecca Eyler, Nancy Glover, Daniel Gordon,  Jim Houtz, Justin Palk, and Shawn Ward.