Immigration Benefits: Consistent Adherence to DHS's Acquisition Policy Could Help Improve Transformation Program Outcomes
Published: 2011-11-22T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Each year, the Department of Homeland Security's (DHS) U.S. Citizenship and Immigration Services (USCIS) processes millions of applications for immigration benefits using a paper-based process. In 2005, USCIS embarked on a major, multiyear program to transform its process to a system that is to incorporate electronic application filing, adjudication, and case management. In 2007, GAO reported that USCIS was in the early stages of the Transformation Program and that USCIS's plans partially or fully met key practices. In 2008, USCIS contracted with a solutions architect to help develop the new system. As requested, GAO evaluated the extent to which USCIS has followed DHS acquisition policy in developing and managing the Transformation Program. GAO reviewed DHS acquisition management policies and guidance; analyzed transformation program planning and implementation documents such as operational requirements; compared schedule and cost information with GAO best practice guidance; and interviewed USCIS officials..
section_title: What GAO Found
paragraphs: USCIS has not consistently followed the acquisition management approach that DHS outlined in its management directives in developing and managing the Transformation Program. USCIS awarded a solutions architect contract in November 2008, in effect selecting an acquisition approach before completing documents required by DHS management directives. Specifically, DHS's acquisition policy requires that prior to selecting an acquisition approach, programs establish operational requirements, develop a program baseline against which to measure progress, and complete a plan that outlines the program's acquisition strategy. However, USCIS did not complete an Operational Requirements Document until October 2009, which was to inform the Acquisition Program Baseline and the Acquisition Plan. Consequently, USCIS awarded a solutions architect contract to begin capability development activities prior to having a full understanding of the program's operational requirements and the resources needed to execute the program. GAO has previously reported that firm requirements must be established and sufficient resources must be allocated at the beginning of an acquisition program, or the program's execution will be subpar. The lack of defined requirements, acquisition strategy, and associated cost parameters contributed to program deployment delays of over 2 years. In addition, through fiscal year 2011, USCIS estimates it will have spent about $703 million, about $292 million more than the original program baseline estimate. USCIS expects to begin deployment of the first release of the Transformation Program in December 2011. However, USCIS is continuing to manage the program without specific acquisition management controls, such as reliable schedules, which detail work to be performed by both the government and its contractor over the expected life of the program. As a result, USCIS does not have reasonable assurance that it can meet its future milestones. USCIS has established schedules for the first release of the Transformation Program, but GAO's analysis shows that these schedules are not reliable as they do not meet best practices for schedule estimating. For example, program schedules did not identify all activities to be performed by the government and solutions architect. Moreover, as outlined by DHS acquisition management guidance, a life-cycle cost estimate is a required and critical element in the acquisition process. USCIS has developed and updated the $1.7 billion life-cycle cost estimate for the Transformation Program, but USCIS's individual schedules for the Transformation Program did not meet best practices for schedule estimating, raising questions about the credibility of the program's life-cycle cost estimates. Because some program costs such as labor, supervision, and facilities cost more if the program takes longer, reliable schedules can contribute to an understanding of the cost impact if the program does not finish on time. Collectively, and moving forward, not meeting best practices increases the risk of schedule slippages and related cost overruns, making meaningful measurement and oversight of program status and progress, and accountability for results, difficult to achieve.
section_title: What GAO Recommends
paragraphs: GAO recommends that USCIS ensure its program schedules and life-cycle cost estimates are developed in accordance with best practices guidance. DHS concurred with GAO's recommendations and outlined the actions that USCIS is taking or has taken to address each recommendation.

section_title: Letter
paragraphs: Each year, the U.S. Citizenship and Immigration Services (USCIS), within  the Department of Homeland Security (DHS), processes millions of  applications and petitions for more than 50 types of immigrant and  nonimmigrant-related benefits for persons seeking to study, work, visit, or  live in the United States, and for persons seeking to become U.S.  citizens. Having a system that allows USCIS to accurately grant  immigration and citizenship benefits in a timely manner to eligible  applicants and deny benefits to those who are ineligible, as well as one  that can quickly and accurately identify fraudulent and criminal activity, is  essential for ensuring the integrity of the immigration process. USCIS has  long recognized the need to improve its benefits application and  adjudication processes and underlying technology infrastructure.  Moreover, we have previously reported on benefit processing  inefficiencies that make it difficult to manage and process immigration  benefit applications in a timely manner. Further, the current systems and  paper records do not allow USCIS to easily share information with other  government agencies, limiting their ability to quickly identify criminals and  possible terrorists. We have previously recommended that USCIS  improve its quality assurance program to help ensure that immigration  benefits are provided only to eligible individuals. USCIS generally agreed  with this recommendation and has taken action to address.
To address program inefficiencies, USCIS embarked on a major initiative  in 2005 to transform its current paper-based system into an electronic  account-based system that is to use electronic adjudication and account- based case management tools, including tools that are to allow applicants  to apply online for benefits. In July 2007, we reported that USCIS was in  the early stages of its Transformation Program and that its plans partially  or fully addressed most key practices for organizational transformations.  However, we identified gaps in USCIS’s plans that created risks that  could undermine its success as it began to implement the program, such  as the lack of clear performance measures and targets for the  transformed agency to show progress towards goals. We recommended  that USCIS address gaps in its plans in the areas of performance  measurement, strategic human capital management, communications,  and information technology management practices. USCIS agreed with  these recommendations and has taken actions to address them. In  October 2007, USCIS prepared an acquisition plan that described its  planned phased approach for acquiring new electronic tools (i.e.,  capabilities), established key milestones, and included an estimated  program cost. In November 2008, USCIS selected a solutions architect to  help design, build, and implement the Transformation Program.
To help its component agencies manage large-scale acquisitions such as  the Transformation Program, DHS has established an acquisition review  process to provide departmental oversight at key points in an  investment’s life-cycle to assess the cost, schedule, and performance of  these acquisitions. At critical steps in the acquisition process, agency  components are required to prepare certain documents and to obtain  approval from DHS’s Acquisition Review Board (ARB) before moving into  the next phase of the acquisition process. In 2007, DHS established an  Acquisition Program Management Division to support the ARB in  managing DHS acquisitions.
Given the critical nature of the Transformation Program to USCIS, you asked  that we evaluate USCIS’s efforts to implement the program. Specifically, this  report addresses the extent to which USCIS has followed DHS acquisition  policy in developing and managing the Transformation Program.
To address this objective, we reviewed DHS departmental policies and  guidance, such as DHS acquisition management directives, and relevant  Transformation Program planning documents, including the 2007  Acquisition Plan and Expenditure Plan; 2008, 2010, and 2011 Acquisition  Program Baseline; and 2009, 2010, and 2011 Operational Requirements  Document. We also reviewed the program’s monthly status reports dated  April 2010, when we began our work, through July 2011; the solutions  architect contract for contractual program milestones; and program  schedules for tracking program milestones and activities. Further, we  reviewed DHS ARB decision memorandums for information on DHS-level  approval of the Transformation Program’s key acquisition events and  action items to be addressed by the program. We interviewed USCIS’s  Transformation Program officials concerning early program planning  efforts and challenges, schedule estimates and costs, and the  development and planned deployment of the first phase of the program,  among other things. We also interviewed DHS Acquisition Program  Management Division officials about the acquisition process and  decisions related to the Transformation Program. We analyzed two key  Transformation Program schedules as of November 2010, and one  updated schedule as of August 2011, against nine best practices  associated with developing and maintaining reliable schedules in our Cost  Estimating and Assessment Guide.  We used commercially available  software to determine whether these schedules, among other best  practices, included all major activities and had a logical sequence of  activities and reasonable activity durations.
We conducted this performance audit from April 2010 through November  2011, in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
subsections: section_title: Background
subsections: section_title: Overview of Transformation Program
paragraphs: Once the Transformation Program is completed, USCIS envisions that  the new electronic adjudication capabilities and improved information  technology will improve agency operations and enable greater data  sharing and management of information. USCIS expects the new system,  named the USCIS Electronic Immigration System (ELIS), to have features  that will allow USCIS to meet its transformation goals of enhanced  national security and system integrity, better customer service, and  operational efficiency. For example, once USCIS ELIS is implemented,  USCIS expects that:  Individuals will be able to establish an account with USCIS and file  applications over the internet, as well as obtain information on the  status of their application.    USCIS will automatically apply risk-based rules to incoming  applications to identify potentially fraudulent applications and national  security risks.    Adjudicators will have electronic access to applications, as well as  relevant USCIS policies and procedures and external databases, to  aid in decision making.    USCIS will have the necessary management information to help it  allocate workload and measure performance.    USCIS will have electronic linkages with other agencies, such as the  Departments of Justice and State, for data sharing and security  purposes.
Figure 1 depicts the key features of USCIS ELIS, as envisioned, from a  USCIS customer perspective.  Figure 2 depicts these key features, as  envisioned, from a USCIS adjudicator perspective.
The Transformation Program intends to design and develop five core  business processes to form the foundation of USCIS ELIS and process  and manage all applications. Table 1 identifies and briefly describes the  five core business processes.
USCIS plans to deploy USCIS ELIS in a series of five releases, labeled A  through E. Within the first two releases, USCIS ELIS is to be available to  USCIS customers applying for nonimmigrant benefit types, followed by  immigrant benefits, humanitarian benefits, and citizenship benefits. Much  of the functionality needed to operate the five core business processes  are to be established during Releases A and B, with an enhanced level of  functionality to be added during Releases C through E. Table 2 below  shows the order in which ELIS’s five releases are to be deployed and  available to USCIS customers.
In 2006, USCIS drafted a transformation strategic plan to guide its  modernization efforts and established the Transformation Program Office  (TPO) to lead and carry out the effort. By 2007, USCIS established a  governance structure for the overall management, leadership, decision  making, and oversight of the Transformation Program. The TPO  governance structure includes three key groups: (1) Transformation  Leadership Team (TLT) responsible for the overall program direction and  coordination of transformation initiatives within the agency; (2) Program  Integrated Product Teams (PIPT) responsible for advising on and  approving strategy and performance measures, and overseeing and  managing the program, including cost, schedule, and performance; and,  (3) Working Integrated Product Teams (WIPT), composed of agencywide  representatives with expertise to help define the transformed business  processes and its operational aspects.
In addition to these key groups, USCIS also holds Program Management  Review (PMR) meetings to help manage the transformation effort. Each  month, the Transformation Program conducts PMR meetings to assess  the status of the overall program and solutions architect activities in terms  of cost, schedule, performance, and risk. Major program groups  associated with USCIS’s transformation efforts, and the solutions  architect, report on the status of activities and deliverables for which they  have responsibility. The monthly PMR reports help provide an up-to- date snapshot of top program risks and concerns and how they are being  mitigated, as well as the overall status of the program in meeting its  milestones, among other information.
In November 2008, USCIS awarded a solutions architect contract for  approximately $500 million to be allocated over a 5-year period, to  design, develop, test, deploy, and sustain the Transformation Program by  November 2013. As such, the Transformation Program is USCIS’s  largest acquisition and according to USCIS’s current Director “no project  is more important to long-term operational improvement and efficiency  than Transformation.” USCIS has funded the Transformation Program  through both direct legislative appropriations and revenue from  applicants’ application fees paid. In fiscal years 2006 and 2007, Congress  appropriated a combined total of $71.7 million to fund Transformation  Program efforts. Since fiscal year 2007, USCIS’s premium processing fee  revenue has been the primary source of funding for the Transformation  Program. In addition, the program has used funds from its application  fee account to pay for the salaries and benefits of USCIS Transformation  Program staff. As shown in Table 3, USCIS spent about $455 million from  fiscal years 2006 through 2010, which includes costs incurred by both the  solutions architect and USCIS. USCIS estimates it will spend  approximately $248 million in fiscal year 2011 for an estimated total cost  of about $703 million through fiscal year 2011.
In 2003, DHS established an investment review process to help reduce  risk and increase the chances for successful acquisition outcomes by  providing departmental oversight of major investments throughout their  life-cycles. The process was intended to help ensure that funds allocated  for investments through the budget process were being spent wisely,  efficiently, and effectively. In March 2006, DHS issued Management  Directive No. 1400 that defined and updated DHS’s investment review  process. The directive required programs to prepare certain documents  before transitioning to the next acquisition phase to ensure the program is  ready to move to the next phase. To implement more rigor and discipline  in its acquisition processes, DHS created the Acquisition Program  Management Division in 2007 to develop and maintain acquisition  policies, procedures, and guidance as a part of the system acquisition  process. In November 2008, DHS issued an interim acquisition directive  and guidebook that superseded Management Directive No. 1400, which  provided programs guidance to use in preparing key documentation to  support component and departmental decision making. In January  2010, DHS finalized the acquisition directive, which established  acquisition life-cycle phases and senior-level approval of each major  acquisition program at key acquisition decision events during a program’s  acquisition life-cycle. This directive established the acquisition life-cycle  framework with four phases:  1.  identify a capability need (need phase);  2.  analyze and select the means to provide that capability  (analyze/select phase);  3.  obtain the capability (obtain phase); and  4.  produce, deploy, and support the capability (produce/deploy/support  phase).
Each acquisition phase culminates in a presentation to the DHS ARB,  which is to review each acquisition at least three times at key acquisition  decision events during a program’s life-cycle. Figure 3 presents the four  DHS acquisition phases, including the documents presented to ARB and  their review as defined in the acquisition directive.
The Acquisition Decision Authority—the Chief Acquisition Officer or other  designated senior-level official—is to chair ARB and decide whether the  proposed acquisition meets certain requirements necessary to move onto  the next phase and eventually to full production. The directive outlines the  extent and scope of required program, project, and service management;  level of reporting; and the acquisition decision authority based on whether  the acquisition is considered a major life-cycle cost (estimated at or  above $300 million) or nonmajor (life-cycle costs estimated to be below  $300 million). DHS considers the USCIS Transformation Program a major  acquisition, and as such, the decision authority is the DHS Under  Secretary for Management.
Following an ARB meeting, the Acquisition Program Management  Division is to prepare an acquisition decision memorandum as the official  record of the meeting. This memorandum is to be signed by the  acquisition decision authority and must describe the approval or other  decisions made at the ARB and any action items to be satisfied as  conditions of the decision. The ARB reviews provide the department an  opportunity to determine a program’s readiness to proceed to the  following life-cycle phase. However, we reported in March 2011 that the  ARB had not reviewed most of DHS’s major acquisition programs by the  end of fiscal year 2009, and the programs that were reviewed had not  consistently implemented action items identified in the review by  established deadlines. Our prior work has shown that when these types  of reviews are skipped or not fully implemented, programs move forward  with little, if any, early department-level assessment of the programs’  costs and feasibility, which contributes to poor cost, schedule, and  performance outcomes. In June 2011, DHS reported that it was taking  action to strengthen its acquisition management processes by reviewing  programs on an ongoing basis rather than only at key acquisition decision  events, and developing decision-making support tools to aid with  oversight. These are positive steps that if effectively implemented should  help strengthen its acquisition management processes.
section_title: USCIS Did Not Always Follow Acquisition Policy Contributing to Program Delays, Cost Increases, and Unreliable Schedules
paragraphs: USCIS has not consistently followed the acquisition management  approach that DHS outlined in its management directives in developing  and managing the Transformation Program. Consistent with DHS  acquisition policy, USCIS prepared a Mission Needs Statement to justify  the need and value of the Transformation Program in pursuing the  proposed acquisition. In addition, USCIS identified and analyzed various  alternatives for transforming its business processes. However, USCIS did  not complete several acquisition planning documents required by DHS  policy prior to moving forward with an acquisition approach and selecting  a solutions architect to develop USCIS ELIS’s capabilities. The lack of  this program documentation contributed to the Transformation Program  being more than 2 years behind schedule in its planned initial deployment  of USCIS ELIS and increased program costs. In addition, USCIS has not  developed reliable or integrated schedules, both of which, under DHS  acquisition guidance, are required and essential acquisition management  elements. As a result, USCIS cannot reliably estimate when all releases  of the Transformation Program will be delivered.
subsections: section_title: USCIS Awarded a Contract to Develop Program Capabilities without Fully Understanding Requirements and Resources Needed
paragraphs: USCIS awarded a solutions architect contract to begin capability  development activities prior to having a full understanding of requirements  and resources needed to execute the program. DHS’s acquisition policy  requires that programs conduct planning efforts to establish a program’s  operational requirements, to develop a program baseline against which to  measure progress, and a plan that outlines the program’s acquisition  strategy. These planning efforts are to be documented in three key  documents: the Operational Requirements Document, the Acquisition  Program Baseline, and the Acquisition Plan. According to DHS policy,  these key documents are to be completed before selecting and moving  forward with an acquisition approach. According to agency officials, the  goal is to help ensure that before committing funds to develop a  capability, the program’s operational requirements, cost, schedule, and  performance parameters have been fully defined. We have previously  reported that firm requirements must be established and sufficient  resources must be allocated at the beginning of an acquisition program,  or the program’s execution will be subpar. We have also reported that  well-defined requirements are critical to ensuring communication about  what the government needs from the contractor providing services.  However, when the solutions architect contract was awarded in  November 2008, one document had not been completed and the other  two did not fully address the program’s estimated cost, planned schedule,  or performance parameters. Below is a summary of the three planning  documents that USCIS did not develop according to DHS policy:  Operational Requirements Document (ORD)—According to DHS  acquisition policy, this document is to describe the operational mission,  objectives, capabilities, and operational user key performance parameters  (i.e., the minimum as well as the desired levels of performance that must  be met to provide a useful capability to the user) and should be  completed before an acquisition approach is selected. However, USCIS  did not develop the first version of the ORD until October 2009, almost a  year after the award of the solutions architect contract.
Program officials acknowledged that an ORD was not prepared prior to  selecting an acquisition approach but stated that the solutions architect  had sufficient information on the program’s operational requirements to  begin work. For example, they stated that the contractor received  USCIS’s Enterprise Segment Activity Roadmap (ESAR), which described  various activities related to ELIS’s core business process. However, in a  February 2009 memorandum, the USCIS Chief Information Officer stated  that the ESAR did not provide a realistic capability to guide, constrain, or  measure the solutions architect because the business process mappings  were incomplete and vague, among other reasons.
Acquisition Program Baseline (APB)—This document is to provide cost,  schedule, and performance parameters. DHS policy requires it to be  prepared prior to selecting an acquisition approach. USCIS completed a  draft acquisition program baseline in May 2008 prior to awarding the  solutions architect contract. However, the May 2008 APB did not fully  address cost, schedule, and performance parameters as required by DHS  policy. Regarding cost, the APB estimated the Transformation Program  would cost approximately $410.7 million for fiscal years 2009 through the  second quarter of fiscal year 2013. However, this estimate only included  the estimated contract cost for a solutions architect. According to program  officials, the estimate did not include USCIS costs for upgrading its  information technology infrastructure, such as upgrading networks and  servers or the costs of USCIS Transformation Program personnel and  other support contractors, because these costs had yet to be defined.  Moreover, USCIS had not yet developed a life-cycle cost estimate, which  per DHS acquisition policy, is a source document used to develop the  APB’s cost parameters.
Regarding the schedule included in the May 2008 APB, it was a high- level view of the program’s key milestones. The acquisition program  baseline shows that the program’s start was expected in fiscal year 2009  and the deployment of all benefit types in USCIS ELIS by fiscal year  2013. According to DHS acquisition policy, this high-level schedule is to  be based upon a program’s integrated master schedule, a larger and  more detailed delineation of program milestones and associated  deliverables. However, USCIS did not complete an integrated master  schedule prior to contract award. In the absence of an integrated master  schedule, program officials were unable to clarify for us how USCIS  determined the program’s key milestones, which had USCIS  implementing USCIS ELIS from fiscal years 2009 through 2013. Lastly,  DHS acquisition policy required that performance parameters be based  upon operational requirements. The May 2008 acquisition program  baseline captured performance parameters, but they were not based on  operational requirements since USCIS had not yet developed operational  requirements, as discussed above.
Acquisition Plan—This document is to address, among other things,  technical, business, management, and other significant considerations  affecting the acquisition strategy and contract selection. USCIS  developed an acquisition plan in October 2007; however, this document  did not address all capabilities for sustaining and maintaining the  acquisition, such as certain technical considerations that would affect the  acquisition strategy, as required by DHS acquisition guidance. For  example, USCIS was to upgrade its information technology infrastructure.  However, the October 2007 acquisition plan did not reflect these technical  considerations.
Moreover, cost information in the acquisition plan is not traceable to other  documents, such as a validated life-cycle cost estimate or an acquisition  program baseline, as required by DHS guidance. Specifically, the October  2007 acquisition plan presented a $3.4 billion estimated cost for the  Transformation Program. According to program officials, the $3.4 billion  included information technology costs and covered the life of the  program, which is similar to a life-cycle cost estimate. However, the     $3.4 billion cost had not been validated as a life-cycle cost estimate by  the DHS Cost Analysis Division. Moreover, the May 2008 acquisition  program baseline makes no reference to the $3.4 billion cost over the life  of the program. However, as required by DHS guidance, the acquisition  program baseline is to reflect all cost parameters.
According to program officials, the solutions architect contract was  performance-based, meaning that USCIS specified the outcomes it was  seeking to achieve and gave the solutions architect responsibility for  identifying and delivering the assets needed to achieve these outcomes.  As a result of this approach, many of the specifics that would affect the  program’s cost and schedule were to be determined after the contract  was signed. The contract called for the solutions architect to use the 90- day base period from November 2008 to February 2009 to develop a plan  to (1) identify work activities to be performed; (2) assign resources to  these activities; (3) project the start and completion dates for these  activities; (4) provide deliverables to the TPO; and (5) establish  performance measures that the contractor and USCIS could use to  measure progress. For example, the specific operational requirements  and USCIS information technology upgrades that would be needed would  depend upon the solutions architect plan. However, as early as 2004—    4 years prior to the solutions architect contract—we reported that this type  of acquisition strategy and contracting approach had led to poor  acquisition outcomes at DHS. Specifically, we reported that a contracting  approach that assigned a U.S. Coast Guard contractor significant  responsibilities, such as the identification of work activities and  deliverables had been a primary reason for performance, cost, and  schedule problems, as it had led to incomplete information about  performance and production risks. Program officials emphasized that  the work completed during this 90-day base period was done in  conjunction with USCIS, which helped to inform the production of these  deliverables.
section_title: Incomplete Planning Contributed to Missed Milestones, Increased Costs, and Anticipated Benefits That Were Not Achievable
paragraphs: Incomplete program planning documents at the start of the program  contributed to the delayed deployment of USCIS ELIS, increased costs,  and anticipated benefits not being achieved.
According to the November 2008 solutions architect contract, the  deployment of capabilities was to begin by September 2009 and be  completed by 2013. USCIS did not meet the September 2009 deployment  milestone. In an April 2009 memorandum to USCIS’s Acting Deputy  Director and Chief Financial Officer, the program manager stated that  based on the solutions architect’s proposal, the program did not have  sufficient staff to provide adequate government oversight of the solutions  architect or funding to support the proposed solution “rendering the  solution unachievable.” Consequently, the solutions architect contract  was modified by scaling back the scope to allow the contractor to focus  on work activities necessary to develop the five core business processes.  Accordingly, TPO was not authorized to start preliminary design work for  Release A until December 2009. By December 2009, TPO proposed  deployment of this first release by April 2011. However, as noted in its  July 2010 Acquisition Decision Memorandum, TPO experienced delays  while defining Release A requirements. These delays resulted in a  revised deployment milestone to occur between June and August 2011.  Difficulties defining requirements continued and in November 2010,  USCIS revised the deployment milestone to December 2011. By January  2011, the requirements had not yet been completed, and by April 2011  USCIS reduced the scope of the first release to meet the newly revised  December 2011 deployment time-frame. Operational requirements were  completed in April 2011 and approved by the ARB in July 2011, nearly     3 years after the solutions contract was awarded. Table 4 provides  information on Transformation Program milestones, status, and  acquisition planning postcontract award.
Because the acquisition strategy and associated cost parameters were  not fully outlined at the start of the program, costs associated with the  Transformation Program have increased above the original estimate. The  program’s May 2008 acquisition program baseline estimated that the total  cost of the program from fiscal years 2009 to 2013 would be             $410.7 million. However, the estimated cost through fiscal year 2011 is  about $703 million, about $292 million more than estimated in May 2008.  This increase in the cost estimate is due to the fact that USCIS’s original  planning efforts did not cover the entire program, as required by DHS  acquisition planning guidance. For example, the acquisition program  baseline did not include USCIS’s information technology enabling costs  which, based on data gathered from program officials, totals  approximately $618 million and includes activities such as upgrading its  technology infrastructure. In addition, the staffing levels have significantly  increased from original projections. At the start of the Transformation  Program, USCIS had allocated funding for 20 full-time equivalent staff  assigned to TPO. As of June 2011, the program had an authorized  staffing level of 98. Other costs not planned for have contributed to the  program’s overall cost increases. For example, the cost of an operational  testing agent, who would be responsible for planning, conducting and  reporting independent operational testing and evaluation for Release A,  was not included in the acquisition planning process. USCIS officials from  TPO and the Office of Information Technology (OIT) agreed that an  operational test agent appeared to be a duplicative effort because TPO  had already planned to conduct independent testing. However, DHS  denied TPO’s request for a waiver of the operational testing agent. As a  result, USCIS contracted with an independent operational test agent by  October 2010, and as of June 2011, TPO has awarded approximately  $1.8 million towards this contract.
USCIS’s Transformation Program planned to deploy USCIS ELIS first to  USCIS customers applying for citizenship benefits. However, once USCIS  defined the costs associated with digitizing (scanning paper documents  into an electronic format) existing records following the June 2009 ARB,  USCIS concluded that the original plans were not achievable within the  associated budget. As a result, in December 2009, USCIS requested— and was authorized by the ARB—to change the order of deployment and  begin with the nonimmigrant instead of citizenship line of business.  Moreover, according to program officials, from June 2010 to March 2011,  USCIS worked to fully define the operational requirements that had not  been developed prior to the start of the solutions architect contract. For  example, an operational requirement of USCIS ELIS is account set up  and intake, which is the ability of USCIS customers to set up accounts  and for adjudicators to process them through one, person-centric  account. TPO worked with subject-matter experts from USCIS’s field and  headquarters offices who were most familiar with the adjudication process  to map out steps that were needed to fully define USCIS ELIS’s  operational requirements. However, in an ARB meeting held in July 2010,  and in a program management review meeting for January 2011,  program officials explained that defining operational requirements was  taking longer than expected due to the complexity of the rules that  needed to be defined in USCIS ELIS, and the review of and agreement to  these rules by all stakeholders. For example, one requirement—the  account set-up and intake requirement—identified 35 operational  functions for USCIS ELIS to perform this action, including set up account  online, schedule an appointment, and evaluate any identity discrepancy.  To enable completion of the operational requirements needed to move  into subsequent phases of development for Release A, USCIS moved  approximately 10 percent of the capabilities into the second release. In  May 2011, program officials told us they changed the scope of the first  release and that full automation of Release A would not be in place in  December 2011. Further, only one nonimmigrant benefit would be  deployed at that time. Other nonimmigrant benefits were scheduled to be  deployed between January and October 2012.
section_title: DHS Increased Its Oversight to Help Ensure Transformation Program’s Compliance with DHS Acquisition Process
paragraphs: DHS has increased its oversight of the Transformation Program since it  authorized USCIS to award the solutions architect contract in October  2008. In 2008, we reported that DHS’s investment review process had not  provided the oversight needed to identify and address cost, schedule, and  performance problems in its major acquisitions, including ensuring that  programs prepared key documents prior to moving into subsequent  phases of program development. At the time, we made several  recommendations aimed at better ensuring DHS fully implemented and  adhered to its acquisition review process, including tracking major  investments. DHS generally agreed with our recommendations and has  since taken actions to improve its acquisition review process, including  developing a database to capture and track key program information,  such as cost and schedule performance, contract awards, and program  risks. The database became fully operational in September 2009. DHS  Acquisition Program Management Division officials acknowledged that  there was limited oversight of the Transformation Program at the time the  contract was signed primarily due to having limited staff to oversee DHS’s  programs. These officials further stated that DHS was continuing to  develop its acquisition oversight function and had begun to implement the  revised acquisition management directive that included more detailed  guidance for programs to use when informing component and  departmental decision making.
Since the contract award, the ARB has met six times to review the  Transformation Program’s status. At these meetings, the ARB has  directed the TPO to address a number of issues related to cost, schedule,  and performance. For example, in June 2009, the ARB held two meetings  to discuss risks that had been identified during the 90-day baseline  period, such as inadequate staffing levels and delays in delivering  required government-furnished items to the contractor. As a result of  these risks, the ARB authorized USCIS to move forward with awarding  contract options one and two, but restricted the amount that could be  expended. The Transformation Program Office was also required to  return to the ARB for authorization to award any additional options.  According to the ARB Acquisition Decision Memorandum from December  2009, the program had improved its staffing significantly, but issues  remained, including the need to fully define system requirements prior to  returning to the ARB for authorization to enter into design, development,  and testing phases, as noted in the August 2010 ARB Acquisition  Decision Memorandum. As a result of this and other outstanding action  items, the ARB did not grant the program permission to proceed with  development as requested by USCIS at the July 2010 and November  2010 ARBs. Subsequently, in April 2011, the program completed  development of operational requirements and the acquisition program  baseline. USCIS received departmental approval for both the  requirements and acquisition baseline in July 2011, along with approval to  proceed with development. DHS Acquisition Program officials stated that  USCIS had received approval because they had fully defined operational  requirements for Release A, but that USCIS was expected to return to the  ARB in December 2011 in order to obtain a decision on whether Release  A can be deployed as scheduled at the end of the year. In addition, DHS  officials stated that before the ARB approves releases beyond Release A,  TPO will need to demonstrate that:    USCIS ELIS’s core business processes work in accordance with its    USCIS can afford to pay for the rest of the program.
In several meetings, the ARB has requested that USCIS refine or  otherwise provide a complete and documented life-cycle cost estimate for  DHS review and validation. USCIS subsequently completed life-cycle cost  estimates in September 2009, November 2010, and an updated version  in March 2011. This most recent version estimated that the  Transformation Program’s life-cycle cost would be approximately         $1.7 billion from fiscal years 2006 through 2022. However, as referenced  in the life-cycle cost estimate—a planning document—USCIS cannot  estimate several work elements because the program does not have  required information to estimate complete cost, such as requirements  beyond Release A. According to the TPO Program Manager, DHS has  reviewed and provided guidance on the development of the life-cycle cost  estimate, but it has not yet validated the life-cycle cost estimate as being  sound and reasonable. Therefore, at this time, the total expected costs of  the program from initiation through completion remain uncertain. Prior to  validation of the life-cycle cost estimate, program officials stated that TPO  and the DHS Cost Analysis Division are to work closely to ensure the cost  estimate is sound and reasonable. According to best practices in cost  estimating, an updated life-cycle cost estimate is to, among others, show  the source of data. In the case of the Transformation Program, an  updated life-cycle cost estimate should show the source of data  underlying the software design and cost estimating model, and the  equations used to estimate the costs of this large effort. The most recent  Acquisition Decision Memorandum dated July 7, 2011, states that the  Transformation Program Office must work closely with the DHS Cost  Analysis Division to complete a life-cycle cost estimate by September 30,  2011.
section_title: Improved Acquisition Planning Could Help USCIS Avoid Further Delays and Potential Cost Overruns for the Transformation Program
subsections: section_title: USCIS Does Not Have Reasonable Assurance That Future Milestones Are Achievable
paragraphs: USCIS is continuing to manage the Transformation Program without  specific acquisition management controls such as reliable schedules and  as a result it will be difficult for USCIS to provide reasonable assurance that  it can meet its future milestones. USCIS has established schedules for the  first release of the Transformation Program, but our analysis shows that  these schedules are not reliable as they do not meet best practices for  schedule estimating. For example, the schedules did not identify all  activities to be performed by the government and solutions architect.  Additionally, USCIS has encountered a number of challenges in  implementing the schedules, such as assumptions that have not been met  regarding the time frames that either the solutions architect or USCIS  would complete certain tasks. For example, according to an April 2010  program management review, USCIS planned to provide the solutions  architect with two technical environments to conduct production and testing  activities by December 2010. However, USCIS has since revised the  schedule due to challenges in procuring hardware and software needed  before these two environments were ready for the solutions architect.
Based on the revised schedule, delivery of the technical environments  was delayed to April 2011, according to USCIS, so that OIT could take  actions to address the delay, such as borrowing equipment until a  contract protest was resolved, and providing the solutions architect with a  walk-through of the technical environments to ensure it met their needs.  Program officials stated that factors outside their control, such as contract  protests or review and approval of system requirements, have contributed  to challenges in implementing the schedules. According to program  officials, defining and developing requirements was expected to last about  2 ½ to 3 months. However, USCIS completed the requirements in             9 months, which included review and validation of these requirements by  agency leadership. Program officials stated that detailed reviews and  approval by agency leadership took longer than expected.
Best practices in schedule estimating state that a comprehensive  schedule should include a schedule risk analysis, so that the risk to the  estimate if items are delayed can be modeled and presented to  management including, among others, assumptions on equipment  deliveries or length of internal and external reviews. However, according  to a January 2011 program management review, as changes to the  program were happening rapidly, there was no analysis completed to  determine the impact on the schedule.
In addition to the challenges USCIS has encountered in carrying out the  schedules as originally planned, on the basis of our analysis we found  that the current schedules for the first release of the Transformation  Program are of questionable reliability. Best practices state that the  success of a large-scale system acquisition program, such as the  Transformation Program, depends in part on having reliable schedules  that identify:    when the program’s set of work activities and milestone events will    how long they will take; and    how they are related to one another.
Among other things, reliable schedules provide a road map for systematic  execution of a program and the means by which to gauge progress,  identify and address potential problems, and promote accountability. Our  research has identified nine best practices associated with developing  and maintaining a reliable schedule. To be considered reliable, a  schedule should meet all nine practices. In a July 2008 memorandum,  DHS’s Under Secretary for Management endorsed the use of these  scheduling practices and noted that DHS would be using them. The nine  scheduling best practices are summarized in table 5.
The Transformation Program has 18 individual schedules. Table 6  summarizes the findings of our assessments of two of these individual  schedules as of November 2010 representing the bulk of the  Transformation Program efforts and those most critical to the production  of USCIS ELIS. Specifically, these schedules track activities associated  with USCIS’s OIT and the solutions architect. TPO is responsible for  managing key acquisition functions associated with the Transformation  Program; thus, USCIS is responsible for tracking and oversight of the OIT  and solutions architect’s activities and associated schedules. Appendix I  includes a detailed discussion of our analysis.
As shown above, the two Transformation Program schedules, for the  most part, did not substantially or fully meet the nine best practices. For  example, neither the OIT nor the USCIS-approved solutions architect  schedule contained detailed information for Release A activities beyond  March 2011. In addition, both schedules were missing a significant  number of logic links between activities which indicate activities that must  finish before others and which activities may not begin until others have  been completed. While we cannot generalize these findings to all 18  schedules, our review raises questions about the reliability of the  program’s schedules.
Based on our discussions with the Transformation Program’s lead  program scheduler, this condition stems, in part, from the  “aggressiveness of the Transformation Program to implement most of the  capabilities within the first 3 years of the program,” and a lack of program  management resources for developing knowledge to create and maintain  schedules. Moreover, and regardless of the aggressiveness of the  solution, our best practices call for schedules to reflect all activities— government, contractors, and any other necessary external parties— essential for successful program completion. As such, neither of the  Transformation Program schedules we reviewed substantially met this  practice. Furthermore, as demonstrated in the challenges USCIS has  encountered in carrying out the schedule as originally planned, not  including all work for all deliverables, regardless of whether the  deliverables are the responsibility of the government or contractor, may  result in confusion among team members and lead to management  difficulties because of an incomplete understanding of the plan and of the  progress being made.
Collectively, and moving forward, not meeting the nine key practices  increases the risk of schedule slippages and related cost overruns and  makes meaningful measurement and oversight of program status and  progress, as well as accountability for results, difficult to achieve. For  example, in June 2011, a program management review noted a schedule  risk if the development, testing, and deployment process slip again. This  could result in USCIS being unable to deliver the first release in  December 2011. A schedule risk analysis could be used to determine the  level of uncertainty and to help mitigate this risk. Similarly, capturing and  sequencing all activities, as outlined in best practices, could help identify  the extent to which other activities linked to this schedule risk are  affecting its progress. Furthermore, without the development of a  schedule that meets scheduling best practices, it will be difficult for  USCIS to effectively monitor and oversee the progress of an estimated  $1.7 billion to be invested in the acquisition of USCIS ELIS.
In August 2011, TPO provided us the updated USCIS-approved solutions  architect schedule. Program officials indicated that this updated schedule  addressed some areas in which their previous schedule was deficient  according to our assessment of the nine scheduling best practices. For  example, they said that this schedule included activities through December  2011 rather than only through March 2011. In addition, officials indicated  that they have confidence in meeting the December 2011 milestone.  Specifically, they said that USCIS and the solutions architect have tested  over 70 percent of the Release A capabilities that are to be released in  December 2011, and demonstrated these capabilities to the USCIS  leadership team in August 2011. On the basis of our analysis of the  updated USCIS-approved solutions architect schedule, we determined that  the updated schedule did not address many of the deficiencies we  identified in the earlier version of the schedule. For example, the USCIS- approved solutions architect schedule did contain activities through  December 2011, but logical links were missing between activities indicating  which activities must finish before others and which activities may not begin  until others have been completed. The schedule’s authorized work has  therefore not been established in a way that describes the sequence of  work, which prevents the schedule from meeting other best practices, such  as establishing a critical path or developing a schedule risk analysis. Thus,  the updated USCIS-approved solutions architect schedule does not fully  meet all nine key practices, making meaningful measure and oversight of  program status and progress difficult to achieve. USCIS did not provide us  with an updated OIT schedule; therefore, we were unable to determine to  what extent many of the deficiencies we identified in the earlier versions  were addressed.
Further, USCIS established the Transformation Program as a long-term  program made up of five releases to procure, test, deploy, and maintain  USCIS ELIS, but USCIS officials confirmed in October 2010 that there  was no integrated master schedule for the entire Transformation  Program. In addition, the schedules we received in August 2011 were  also not integrated into a master schedule. According to best practices,  an integrated master schedule is to contain the detailed tasks necessary  to ensure program execution and is a required document to develop key  acquisition planning documents under DHS acquisition management  guidance. Among other things, best practices and related federal  guidance call for a program schedule to be programwide in scope,  meaning that it should include the integrated breakdown of the work to be  performed by both the government and its contractors over the expected  life of the program. According to program officials, when the  Transformation Program’s planning efforts began, USCIS was unable to  develop an integrated master schedule for the Transformation Program  due to the complexity of integrating the numerous individual schedules  and the lack of skilled staff necessary to develop and manage such an  integrated master schedule. In addition, program officials explained that  scheduling software to develop and maintain individual schedules was not  used by every organization performing transformation work, such as OIT,  even though the program issued guidance in August 2010 to all  organizations on scheduling best practices, including the use of  scheduling software.
As an alternative to an integrated master schedule and for ease of  reporting to the ARB and other senior officials, TPO developed a high- level tracking tool summarizing dates and activities for the first release of  the program and based on individual schedules such as the OIT and  solutions architect schedule, which are not directly managed by TPO.  According to program officials, in a September 2010 briefing to agency  leadership, this high-level tracking tool created capacity for USCIS to  analyze the schedule. In this briefing, program officials stated TPO used  the high-level tracking tool to ensure coordination and alignment of  activities by collaborating with staff responsible for the management of  individual schedules. However, this tracking tool is not an integrated  master schedule as it does not integrate all activities necessary to meet  the milestones for Release A; rather, it is a selection of key activities  drawn from the individual schedules maintained by USCIS components  and the solutions architect. Moreover, the Transformation Program  Manager expressed concern in a May 2011 program management review  that the information reported in the high-level tracking tool was not being  reported in the individual schedules. In addition, this tracking tool is not an  integrated master schedule because it does not show activities over the  life of the program. That is, there are no dates or activities for when the  other four releases’ set of work activities will occur, how long they will  take, and how they are related to one another. As a result, it will be  difficult for program officials to predict, with any degree of confidence,  how long it will take to complete all five releases of the Transformation  Program. It will also be difficult for program officials to manage and  measure progress in executing the work needed to deliver the program,  thus increasing the risk of cost, schedule, and performance shortfalls.  Lastly, USCIS’s ability to accurately communicate the status of  Transformation Program efforts to key stakeholders such as its  employees, Congress and the public will be hindered.
Because USCIS lacks reliable schedules, its ability to develop reliable  life-cycle cost estimates is hampered. As outlined by DHS acquisition  management guidance, a life-cycle cost estimate is a required and critical  element in the acquisition process. USCIS has developed and updated  the life-cycle cost estimate for the Transformation Program, but USCIS’s  individual schedules for the Transformation Program do not meet best  practices for schedule estimating, thus raising questions about the  credibility of the program’s life-cycle cost estimates. For example, neither  the OIT nor the solutions architect schedule fully captured all activities to  be performed by the government and contractor. Therefore, when USCIS  is developing the life-cycle cost estimate there is an incomplete  understanding of the work necessary to accomplish the five releases of  the Transformation Program. Further, in the case of both individual  schedules, the absence of a schedule risk analysis makes it difficult for  officials to account for the cost effects of schedule slippage when  developing the life-cycle cost estimate. Further, a reliable life-cycle cost  estimate is essential for helping the program determine how much  funding is needed and whether it will be available to achieve the  Transformation Program’s goals.
Best practices that we have previously identified for cost estimation state  that because some program costs such as labor, supervision, rented  equipment, and facilities cost more if the program takes longer, a reliable  schedule can contribute to an understanding of the cost impact if the  program does not finish on time. Meeting planned milestones and  controlling costs are both dependent on the quality of a program’s  schedule. An integrated schedule is key to managing program  performance and is necessary for determining what work remains and the  expected cost to complete the work.
section_title: Conclusions
paragraphs: USCIS’s effort to develop a modern, automated system for processing  benefit applications and addressing the many current program  inefficiencies has been in progress for nearly 6 years. The program is  now more than 2 years behind its planned deployment schedule for  implementing the agencywide transformed business process, and given  the enormity, significance, and complexity of this transformation, it is  essential that USCIS ensures it takes the proper steps for  implementation. Although only one benefit type is expected to be  available for online account management and adjudication in December  2011, the decision to channel resources and efforts to focus on ensuring  the core businesses are ready for a December 2011 launch prior to  making other application types available for online-processing appears to  be prudent.
Moving forward, it is essential that USCIS consistently follows DHS  acquisition management guidance to best position the department to  develop and share information within the department and with Congress  and the public that can be relied upon for purposes of informed decision  making. Moreover, ensuring that the program’s schedules are consistent  with schedule estimating best practices and integrated through an  integrated master schedule would better position USCIS to reliably  estimate the amount of time and effort needed to complete the program.  Reliable schedules could also assist USCIS in developing and  maintaining a complete and reliable life-cycle cost estimate for the  program which is essential for helping the program determine how much  funding is needed and whether it will be available to achieve the  Transformation Program’s goals.
section_title: Recommendations for Executive Action
paragraphs: To help ensure that USCIS takes a comprehensive and cost-effective  approach to the development and deployment of transformation efforts to  meet the agency’s goals of improved adjudications and customer services  processes, we recommend that the Director of USCIS take the following  three actions:  1.  Ensure program schedules are consistent with the nine estimating  best practices.  2.  Develop and maintain an Integrated Master Schedule consistent with  these same best practices for the Transformation Program.  3.  Ensure that the life-cycle cost estimate is informed by milestones and  associated tasks from reliable schedules that are developed in  accordance with the nine best practices we identified.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to DHS for comment. DHS provided  written comments, which are reprinted in Appendix II. In commenting on  this report, DHS, including USCIS, concurred with the recommendations.
DHS's letter outlined the actions that USCIS is taking action or has taken  to address each recommendation. Regarding the first recommendation to  ensure program schedules are consistent with best practices, DHS stated  that USCIS is incorporating the nine schedule estimating best practices  we identified into Transformation Program management reviews, as well  as the Acquisition Review Board review. Regarding the second  recommendation to develop and maintain an Integrated Master Schedule  consistent with these same best practices for the Transformation  Program, DHS stated that USCIS will develop an Integrated Master  Schedule to depict the multiple tasks, implementation activities, and  interrelationships needed to successfully develop and deploy the  Transformation Program. Regarding the third recommendation to ensure  that life-cycle cost estimates are developed in accordance with the nine  best practices, DHS stated that they will refine the Transformation  Program life-cycle cost estimate in accordance with GAO’s 12-Step  Process for Cost Estimation. In addition, DHS noted that their revised  master schedule will clearly identify work elements to ensure a  reasonable and cost-effective timeframe for accomplishing the five  releases associated with the program. If fully implemented, we believe  that the actions that DHS identified will address our recommendations.
DHS also provided technical comments, which we have incorporated, as  appropriate.
We are sending copies of this report to the Secretary of Homeland  Security and other interested parties. The report also will be available at  no charge on the GAO Web site at http://www.gao.gov.
If you or your staff members have any questions about this report, please  contact me at (202) 512-8777 or stanar@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on  the last page of this report. GAO staff who made major contributions to  this report are listed in appendix III.
section_title: Appendix I: Detailed Results of GAO Assessment of USCIS’s Detailed Transformation Program Schedules
paragraphs: In prior work, we have identified nine best practices associated with  effective schedule estimating. These are (1) capturing all activities; (2)  sequencing all activities; (3) assigning resources to all activities; (4)  establishing the duration of all activities; (5) integrating activities  horizontally and vertically; (6) establishing the critical path for all activities;  (7) identifying float time between activities; (8) conducting a schedule risk  analysis; and (9) updating the schedule using logic and durations. We  assessed the extent to which two detailed schedules, Office of  Information Technology (OIT) and the solutions architect, dated  November 2010, met each of the nine practices. We characterized  whether the schedules met each of the nine best practices as follows:    Not met—the program provided no evidence that satisfies any portion  of the criterion.    Minimally met—the program provided evidence that satisfies less than  one-half of the criterion.    Partially met—the program provided evidence that satisfies about  one-half of the criterion.    Substantially met—the program provided evidence that satisfies more  than one-half of the criterion.    Met—the program provided evidence that satisfies the entire criterion.
Tables 7 and 8 provide the detailed results of our analysis of these  schedules.
section_title: Appendix II: Comments from the Department of Homeland Security
section_title: Appendix III: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Mike Dino, Assistant Director;  Kathryn Bernet, Assistant Director; and Carla Brown, Analyst-in-Charge;  managed this assignment. Sylvia Bascopé, Jim Russell, and Ulyana  Panchishin made significant contributions to the work. Nate Tranquilly  and Bill Russell provided expertise on acquisition issues. Tisha Derricotte  and Jason Lee provided expertise on scheduling best practices. Frances  Cook provided legal support. Linda Miller and Labony Chakraborty  provided assistance in report preparation, and Robert Robinson  developed the report’s graphics.
section_title: Related GAO Products
paragraphs: Acquisition Planning: Opportunities to Build Strong Foundations for Better  Services Contracts. GAO-11-672. Washington, D.C.: August 9, 2011.
Opportunities to Reduce Potential Duplication in Government Programs,  Save Tax Dollars, and Enhance Revenue. GAO-11-318SP. Washington,  D.C.: March 1, 2011.
Secure Boarder Initiative: DHS Needs to Strengthen Management and  Oversight of its Prime Contractor. GAO-11-6. Washington, D.C.:   October 18, 2010.
Department of Homeland Security: Assessments of Selected Complex  Acquisitions. GAO-10-588SP. Washington, D.C.: June 30, 2010.
GAO Cost Estimating and Assessment Guide: Best Practices for  Developing and Managing Capital Program Costs. GAO-09-3SP.  Washington, D.C.: March 2009.
Department of Homeland Security: Billions Invested in Major Programs  Lack Appropriate Oversight. GAO-09-29. Washington, D.C.: November  18, 2008.
Defense Acquisitions: Realistic Business Cases Needed to Execute Navy  Shipbuilding Programs. GAO-07-943T. Washington, D.C.: July 24, 2007.
USCIS Transformation: Improvements to Performance, Human Capital,  and Information Technology Management Needed as Modernization  Proceeds. GAO-07-1013R. Washington, D.C.: July 17, 2007.
Immigration Benefits: Additional Efforts Needed to Help Ensure Alien  Files Are Located when Needed. GAO-07-85. Washington, D.C.:   October 27, 2006.
Information Technology: Near-Term Effort to Automate Paper-Based  Immigration Files Needs Planning Improvements. GAO-06-375.  Washington, D.C.: March 31, 2006.
Immigration Benefits: Improvements Needed to Address Backlogs and  Ensure Quality of Adjudications. GAO-06-20. Washington, D.C.:  November 21, 2005.
Contract Management: Coast Guard’s Deepwater Program Needs  Increased Attention to Management and Contractor Oversight.   GAO-04-380. Washington, D.C.: March 9, 2004.