Immigration Benefits: Consistent Adherence to DHS's Acquisition Policy Could Help Improve Transformation Program Outcomes
Released: 2011-11-22T00:00:00

Letter

Background

Overview of Transformation Program

Once the Transformation Program is completed, USCIS envisions that the new electronic adjudication capabilities and improved information technology will improve agency operations and enable greater data sharing and management of information. USCIS expects the new system, named the USCIS Electronic Immigration System (ELIS), to have features that will allow USCIS to meet its transformation goals of enhanced national security and system integrity, better customer service, and operational efficiency. For example, once USCIS ELIS is implemented, USCIS expects that: Individuals will be able to establish an account with USCIS and file applications over the internet, as well as obtain information on the status of their application.  USCIS will automatically apply risk-based rules to incoming applications to identify potentially fraudulent applications and national security risks.  Adjudicators will have electronic access to applications, as well as relevant USCIS policies and procedures and external databases, to aid in decision making.  USCIS will have the necessary management information to help it allocate workload and measure performance.  USCIS will have electronic linkages with other agencies, such as the Departments of Justice and State, for data sharing and security purposes.

Figure 1 depicts the key features of USCIS ELIS, as envisioned, from a USCIS customer perspective. Figure 2 depicts these key features, as envisioned, from a USCIS adjudicator perspective.

The Transformation Program intends to design and develop five core business processes to form the foundation of USCIS ELIS and process and manage all applications. Table 1 identifies and briefly describes the five core business processes.

USCIS plans to deploy USCIS ELIS in a series of five releases, labeled A through E. Within the first two releases, USCIS ELIS is to be available to USCIS customers applying for nonimmigrant benefit types, followed by immigrant benefits, humanitarian benefits, and citizenship benefits. Much of the functionality needed to operate the five core business processes are to be established during Releases A and B, with an enhanced level of functionality to be added during Releases C through E. Table 2 below shows the order in which ELIS’s five releases are to be deployed and available to USCIS customers.

In 2006, USCIS drafted a transformation strategic plan to guide its modernization efforts and established the Transformation Program Office (TPO) to lead and carry out the effort. By 2007, USCIS established a governance structure for the overall management, leadership, decision making, and oversight of the Transformation Program. The TPO governance structure includes three key groups: (1) Transformation Leadership Team (TLT) responsible for the overall program direction and coordination of transformation initiatives within the agency; (2) Program Integrated Product Teams (PIPT) responsible for advising on and approving strategy and performance measures, and overseeing and managing the program, including cost, schedule, and performance; and, (3) Working Integrated Product Teams (WIPT), composed of agencywide representatives with expertise to help define the transformed business processes and its operational aspects.

In addition to these key groups, USCIS also holds Program Management Review (PMR) meetings to help manage the transformation effort. Each month, the Transformation Program conducts PMR meetings to assess the status of the overall program and solutions architect activities in terms of cost, schedule, performance, and risk. Major program groups associated with USCIS’s transformation efforts, and the solutions architect, report on the status of activities and deliverables for which they have responsibility. The monthly PMR reports help provide an up-to- date snapshot of top program risks and concerns and how they are being mitigated, as well as the overall status of the program in meeting its milestones, among other information.

In November 2008, USCIS awarded a solutions architect contract for approximately $500 million to be allocated over a 5-year period, to design, develop, test, deploy, and sustain the Transformation Program by November 2013. As such, the Transformation Program is USCIS’s largest acquisition and according to USCIS’s current Director “no project is more important to long-term operational improvement and efficiency than Transformation.” USCIS has funded the Transformation Program through both direct legislative appropriations and revenue from applicants’ application fees paid. In fiscal years 2006 and 2007, Congress appropriated a combined total of $71.7 million to fund Transformation Program efforts. Since fiscal year 2007, USCIS’s premium processing fee revenue has been the primary source of funding for the Transformation Program. In addition, the program has used funds from its application fee account to pay for the salaries and benefits of USCIS Transformation Program staff. As shown in Table 3, USCIS spent about $455 million from fiscal years 2006 through 2010, which includes costs incurred by both the solutions architect and USCIS. USCIS estimates it will spend approximately $248 million in fiscal year 2011 for an estimated total cost of about $703 million through fiscal year 2011.

In 2003, DHS established an investment review process to help reduce risk and increase the chances for successful acquisition outcomes by providing departmental oversight of major investments throughout their life-cycles. The process was intended to help ensure that funds allocated for investments through the budget process were being spent wisely, efficiently, and effectively. In March 2006, DHS issued Management Directive No. 1400 that defined and updated DHS’s investment review process. The directive required programs to prepare certain documents before transitioning to the next acquisition phase to ensure the program is ready to move to the next phase. To implement more rigor and discipline in its acquisition processes, DHS created the Acquisition Program Management Division in 2007 to develop and maintain acquisition policies, procedures, and guidance as a part of the system acquisition process. In November 2008, DHS issued an interim acquisition directive and guidebook that superseded Management Directive No. 1400, which provided programs guidance to use in preparing key documentation to support component and departmental decision making. In January 2010, DHS finalized the acquisition directive, which established acquisition life-cycle phases and senior-level approval of each major acquisition program at key acquisition decision events during a program’s acquisition life-cycle. This directive established the acquisition life-cycle framework with four phases: 1. identify a capability need (need phase); 2. analyze and select the means to provide that capability (analyze/select phase); 3. obtain the capability (obtain phase); and 4. produce, deploy, and support the capability (produce/deploy/support phase).

Each acquisition phase culminates in a presentation to the DHS ARB, which is to review each acquisition at least three times at key acquisition decision events during a program’s life-cycle. Figure 3 presents the four DHS acquisition phases, including the documents presented to ARB and their review as defined in the acquisition directive.

The Acquisition Decision Authority—the Chief Acquisition Officer or other designated senior-level official—is to chair ARB and decide whether the proposed acquisition meets certain requirements necessary to move onto the next phase and eventually to full production. The directive outlines the extent and scope of required program, project, and service management; level of reporting; and the acquisition decision authority based on whether the acquisition is considered a major life-cycle cost (estimated at or above $300 million) or nonmajor (life-cycle costs estimated to be below $300 million). DHS considers the USCIS Transformation Program a major acquisition, and as such, the decision authority is the DHS Under Secretary for Management.

Following an ARB meeting, the Acquisition Program Management Division is to prepare an acquisition decision memorandum as the official record of the meeting. This memorandum is to be signed by the acquisition decision authority and must describe the approval or other decisions made at the ARB and any action items to be satisfied as conditions of the decision. The ARB reviews provide the department an opportunity to determine a program’s readiness to proceed to the following life-cycle phase. However, we reported in March 2011 that the ARB had not reviewed most of DHS’s major acquisition programs by the end of fiscal year 2009, and the programs that were reviewed had not consistently implemented action items identified in the review by established deadlines. Our prior work has shown that when these types of reviews are skipped or not fully implemented, programs move forward with little, if any, early department-level assessment of the programs’ costs and feasibility, which contributes to poor cost, schedule, and performance outcomes. In June 2011, DHS reported that it was taking action to strengthen its acquisition management processes by reviewing programs on an ongoing basis rather than only at key acquisition decision events, and developing decision-making support tools to aid with oversight. These are positive steps that if effectively implemented should help strengthen its acquisition management processes.

USCIS Did Not Always Follow Acquisition Policy Contributing to Program Delays, Cost Increases, and Unreliable Schedules

USCIS has not consistently followed the acquisition management approach that DHS outlined in its management directives in developing and managing the Transformation Program. Consistent with DHS acquisition policy, USCIS prepared a Mission Needs Statement to justify the need and value of the Transformation Program in pursuing the proposed acquisition. In addition, USCIS identified and analyzed various alternatives for transforming its business processes. However, USCIS did not complete several acquisition planning documents required by DHS policy prior to moving forward with an acquisition approach and selecting a solutions architect to develop USCIS ELIS’s capabilities. The lack of this program documentation contributed to the Transformation Program being more than 2 years behind schedule in its planned initial deployment of USCIS ELIS and increased program costs. In addition, USCIS has not developed reliable or integrated schedules, both of which, under DHS acquisition guidance, are required and essential acquisition management elements. As a result, USCIS cannot reliably estimate when all releases of the Transformation Program will be delivered.

USCIS Awarded a Contract to Develop Program Capabilities without Fully Understanding Requirements and Resources Needed

USCIS awarded a solutions architect contract to begin capability development activities prior to having a full understanding of requirements and resources needed to execute the program. DHS’s acquisition policy requires that programs conduct planning efforts to establish a program’s operational requirements, to develop a program baseline against which to measure progress, and a plan that outlines the program’s acquisition strategy. These planning efforts are to be documented in three key documents: the Operational Requirements Document, the Acquisition Program Baseline, and the Acquisition Plan. According to DHS policy, these key documents are to be completed before selecting and moving forward with an acquisition approach. According to agency officials, the goal is to help ensure that before committing funds to develop a capability, the program’s operational requirements, cost, schedule, and performance parameters have been fully defined. We have previously reported that firm requirements must be established and sufficient resources must be allocated at the beginning of an acquisition program, or the program’s execution will be subpar. We have also reported that well-defined requirements are critical to ensuring communication about what the government needs from the contractor providing services. However, when the solutions architect contract was awarded in November 2008, one document had not been completed and the other two did not fully address the program’s estimated cost, planned schedule, or performance parameters. Below is a summary of the three planning documents that USCIS did not develop according to DHS policy: Operational Requirements Document (ORD)—According to DHS acquisition policy, this document is to describe the operational mission, objectives, capabilities, and operational user key performance parameters (i.e., the minimum as well as the desired levels of performance that must be met to provide a useful capability to the user) and should be completed before an acquisition approach is selected. However, USCIS did not develop the first version of the ORD until October 2009, almost a year after the award of the solutions architect contract.

Program officials acknowledged that an ORD was not prepared prior to selecting an acquisition approach but stated that the solutions architect had sufficient information on the program’s operational requirements to begin work. For example, they stated that the contractor received USCIS’s Enterprise Segment Activity Roadmap (ESAR), which described various activities related to ELIS’s core business process. However, in a February 2009 memorandum, the USCIS Chief Information Officer stated that the ESAR did not provide a realistic capability to guide, constrain, or measure the solutions architect because the business process mappings were incomplete and vague, among other reasons.

Acquisition Program Baseline (APB)—This document is to provide cost, schedule, and performance parameters. DHS policy requires it to be prepared prior to selecting an acquisition approach. USCIS completed a draft acquisition program baseline in May 2008 prior to awarding the solutions architect contract. However, the May 2008 APB did not fully address cost, schedule, and performance parameters as required by DHS policy. Regarding cost, the APB estimated the Transformation Program would cost approximately $410.7 million for fiscal years 2009 through the second quarter of fiscal year 2013. However, this estimate only included the estimated contract cost for a solutions architect. According to program officials, the estimate did not include USCIS costs for upgrading its information technology infrastructure, such as upgrading networks and servers or the costs of USCIS Transformation Program personnel and other support contractors, because these costs had yet to be defined. Moreover, USCIS had not yet developed a life-cycle cost estimate, which per DHS acquisition policy, is a source document used to develop the APB’s cost parameters.

Regarding the schedule included in the May 2008 APB, it was a high- level view of the program’s key milestones. The acquisition program baseline shows that the program’s start was expected in fiscal year 2009 and the deployment of all benefit types in USCIS ELIS by fiscal year 2013. According to DHS acquisition policy, this high-level schedule is to be based upon a program’s integrated master schedule, a larger and more detailed delineation of program milestones and associated deliverables. However, USCIS did not complete an integrated master schedule prior to contract award. In the absence of an integrated master schedule, program officials were unable to clarify for us how USCIS determined the program’s key milestones, which had USCIS implementing USCIS ELIS from fiscal years 2009 through 2013. Lastly, DHS acquisition policy required that performance parameters be based upon operational requirements. The May 2008 acquisition program baseline captured performance parameters, but they were not based on operational requirements since USCIS had not yet developed operational requirements, as discussed above.

Acquisition Plan—This document is to address, among other things, technical, business, management, and other significant considerations affecting the acquisition strategy and contract selection. USCIS developed an acquisition plan in October 2007; however, this document did not address all capabilities for sustaining and maintaining the acquisition, such as certain technical considerations that would affect the acquisition strategy, as required by DHS acquisition guidance. For example, USCIS was to upgrade its information technology infrastructure. However, the October 2007 acquisition plan did not reflect these technical considerations.

Moreover, cost information in the acquisition plan is not traceable to other documents, such as a validated life-cycle cost estimate or an acquisition program baseline, as required by DHS guidance. Specifically, the October 2007 acquisition plan presented a $3.4 billion estimated cost for the Transformation Program. According to program officials, the $3.4 billion included information technology costs and covered the life of the program, which is similar to a life-cycle cost estimate. However, the $3.4 billion cost had not been validated as a life-cycle cost estimate by the DHS Cost Analysis Division. Moreover, the May 2008 acquisition program baseline makes no reference to the $3.4 billion cost over the life of the program. However, as required by DHS guidance, the acquisition program baseline is to reflect all cost parameters.

According to program officials, the solutions architect contract was performance-based, meaning that USCIS specified the outcomes it was seeking to achieve and gave the solutions architect responsibility for identifying and delivering the assets needed to achieve these outcomes. As a result of this approach, many of the specifics that would affect the program’s cost and schedule were to be determined after the contract was signed. The contract called for the solutions architect to use the 90- day base period from November 2008 to February 2009 to develop a plan to (1) identify work activities to be performed; (2) assign resources to these activities; (3) project the start and completion dates for these activities; (4) provide deliverables to the TPO; and (5) establish performance measures that the contractor and USCIS could use to measure progress. For example, the specific operational requirements and USCIS information technology upgrades that would be needed would depend upon the solutions architect plan. However, as early as 2004— 4 years prior to the solutions architect contract—we reported that this type of acquisition strategy and contracting approach had led to poor acquisition outcomes at DHS. Specifically, we reported that a contracting approach that assigned a U.S. Coast Guard contractor significant responsibilities, such as the identification of work activities and deliverables had been a primary reason for performance, cost, and schedule problems, as it had led to incomplete information about performance and production risks. Program officials emphasized that the work completed during this 90-day base period was done in conjunction with USCIS, which helped to inform the production of these deliverables.

Incomplete Planning Contributed to Missed Milestones, Increased Costs, and Anticipated Benefits That Were Not Achievable

Incomplete program planning documents at the start of the program contributed to the delayed deployment of USCIS ELIS, increased costs, and anticipated benefits not being achieved.

According to the November 2008 solutions architect contract, the deployment of capabilities was to begin by September 2009 and be completed by 2013. USCIS did not meet the September 2009 deployment milestone. In an April 2009 memorandum to USCIS’s Acting Deputy Director and Chief Financial Officer, the program manager stated that based on the solutions architect’s proposal, the program did not have sufficient staff to provide adequate government oversight of the solutions architect or funding to support the proposed solution “rendering the solution unachievable.” Consequently, the solutions architect contract was modified by scaling back the scope to allow the contractor to focus on work activities necessary to develop the five core business processes. Accordingly, TPO was not authorized to start preliminary design work for Release A until December 2009. By December 2009, TPO proposed deployment of this first release by April 2011. However, as noted in its July 2010 Acquisition Decision Memorandum, TPO experienced delays while defining Release A requirements. These delays resulted in a revised deployment milestone to occur between June and August 2011. Difficulties defining requirements continued and in November 2010, USCIS revised the deployment milestone to December 2011. By January 2011, the requirements had not yet been completed, and by April 2011 USCIS reduced the scope of the first release to meet the newly revised December 2011 deployment time-frame. Operational requirements were completed in April 2011 and approved by the ARB in July 2011, nearly 3 years after the solutions contract was awarded. Table 4 provides information on Transformation Program milestones, status, and acquisition planning postcontract award.

Because the acquisition strategy and associated cost parameters were not fully outlined at the start of the program, costs associated with the Transformation Program have increased above the original estimate. The program’s May 2008 acquisition program baseline estimated that the total cost of the program from fiscal years 2009 to 2013 would be $410.7 million. However, the estimated cost through fiscal year 2011 is about $703 million, about $292 million more than estimated in May 2008. This increase in the cost estimate is due to the fact that USCIS’s original planning efforts did not cover the entire program, as required by DHS acquisition planning guidance. For example, the acquisition program baseline did not include USCIS’s information technology enabling costs which, based on data gathered from program officials, totals approximately $618 million and includes activities such as upgrading its technology infrastructure. In addition, the staffing levels have significantly increased from original projections. At the start of the Transformation Program, USCIS had allocated funding for 20 full-time equivalent staff assigned to TPO. As of June 2011, the program had an authorized staffing level of 98. Other costs not planned for have contributed to the program’s overall cost increases. For example, the cost of an operational testing agent, who would be responsible for planning, conducting and reporting independent operational testing and evaluation for Release A, was not included in the acquisition planning process. USCIS officials from TPO and the Office of Information Technology (OIT) agreed that an operational test agent appeared to be a duplicative effort because TPO had already planned to conduct independent testing. However, DHS denied TPO’s request for a waiver of the operational testing agent. As a result, USCIS contracted with an independent operational test agent by October 2010, and as of June 2011, TPO has awarded approximately $1.8 million towards this contract.

USCIS’s Transformation Program planned to deploy USCIS ELIS first to USCIS customers applying for citizenship benefits. However, once USCIS defined the costs associated with digitizing (scanning paper documents into an electronic format) existing records following the June 2009 ARB, USCIS concluded that the original plans were not achievable within the associated budget. As a result, in December 2009, USCIS requested— and was authorized by the ARB—to change the order of deployment and begin with the nonimmigrant instead of citizenship line of business. Moreover, according to program officials, from June 2010 to March 2011, USCIS worked to fully define the operational requirements that had not been developed prior to the start of the solutions architect contract. For example, an operational requirement of USCIS ELIS is account set up and intake, which is the ability of USCIS customers to set up accounts and for adjudicators to process them through one, person-centric account. TPO worked with subject-matter experts from USCIS’s field and headquarters offices who were most familiar with the adjudication process to map out steps that were needed to fully define USCIS ELIS’s operational requirements. However, in an ARB meeting held in July 2010, and in a program management review meeting for January 2011, program officials explained that defining operational requirements was taking longer than expected due to the complexity of the rules that needed to be defined in USCIS ELIS, and the review of and agreement to these rules by all stakeholders. For example, one requirement—the account set-up and intake requirement—identified 35 operational functions for USCIS ELIS to perform this action, including set up account online, schedule an appointment, and evaluate any identity discrepancy. To enable completion of the operational requirements needed to move into subsequent phases of development for Release A, USCIS moved approximately 10 percent of the capabilities into the second release. In May 2011, program officials told us they changed the scope of the first release and that full automation of Release A would not be in place in December 2011. Further, only one nonimmigrant benefit would be deployed at that time. Other nonimmigrant benefits were scheduled to be deployed between January and October 2012.

DHS Increased Its Oversight to Help Ensure Transformation Program’s Compliance with DHS Acquisition Process

DHS has increased its oversight of the Transformation Program since it authorized USCIS to award the solutions architect contract in October 2008. In 2008, we reported that DHS’s investment review process had not provided the oversight needed to identify and address cost, schedule, and performance problems in its major acquisitions, including ensuring that programs prepared key documents prior to moving into subsequent phases of program development. At the time, we made several recommendations aimed at better ensuring DHS fully implemented and adhered to its acquisition review process, including tracking major investments. DHS generally agreed with our recommendations and has since taken actions to improve its acquisition review process, including developing a database to capture and track key program information, such as cost and schedule performance, contract awards, and program risks. The database became fully operational in September 2009. DHS Acquisition Program Management Division officials acknowledged that there was limited oversight of the Transformation Program at the time the contract was signed primarily due to having limited staff to oversee DHS’s programs. These officials further stated that DHS was continuing to develop its acquisition oversight function and had begun to implement the revised acquisition management directive that included more detailed guidance for programs to use when informing component and departmental decision making.

Since the contract award, the ARB has met six times to review the Transformation Program’s status. At these meetings, the ARB has directed the TPO to address a number of issues related to cost, schedule, and performance. For example, in June 2009, the ARB held two meetings to discuss risks that had been identified during the 90-day baseline period, such as inadequate staffing levels and delays in delivering required government-furnished items to the contractor. As a result of these risks, the ARB authorized USCIS to move forward with awarding contract options one and two, but restricted the amount that could be expended. The Transformation Program Office was also required to return to the ARB for authorization to award any additional options. According to the ARB Acquisition Decision Memorandum from December 2009, the program had improved its staffing significantly, but issues remained, including the need to fully define system requirements prior to returning to the ARB for authorization to enter into design, development, and testing phases, as noted in the August 2010 ARB Acquisition Decision Memorandum. As a result of this and other outstanding action items, the ARB did not grant the program permission to proceed with development as requested by USCIS at the July 2010 and November 2010 ARBs. Subsequently, in April 2011, the program completed development of operational requirements and the acquisition program baseline. USCIS received departmental approval for both the requirements and acquisition baseline in July 2011, along with approval to proceed with development. DHS Acquisition Program officials stated that USCIS had received approval because they had fully defined operational requirements for Release A, but that USCIS was expected to return to the ARB in December 2011 in order to obtain a decision on whether Release A can be deployed as scheduled at the end of the year. In addition, DHS officials stated that before the ARB approves releases beyond Release A, TPO will need to demonstrate that:  USCIS ELIS’s core business processes work in accordance with its  USCIS can afford to pay for the rest of the program.

In several meetings, the ARB has requested that USCIS refine or otherwise provide a complete and documented life-cycle cost estimate for DHS review and validation. USCIS subsequently completed life-cycle cost estimates in September 2009, November 2010, and an updated version in March 2011. This most recent version estimated that the Transformation Program’s life-cycle cost would be approximately $1.7 billion from fiscal years 2006 through 2022. However, as referenced in the life-cycle cost estimate—a planning document—USCIS cannot estimate several work elements because the program does not have required information to estimate complete cost, such as requirements beyond Release A. According to the TPO Program Manager, DHS has reviewed and provided guidance on the development of the life-cycle cost estimate, but it has not yet validated the life-cycle cost estimate as being sound and reasonable. Therefore, at this time, the total expected costs of the program from initiation through completion remain uncertain. Prior to validation of the life-cycle cost estimate, program officials stated that TPO and the DHS Cost Analysis Division are to work closely to ensure the cost estimate is sound and reasonable. According to best practices in cost estimating, an updated life-cycle cost estimate is to, among others, show the source of data. In the case of the Transformation Program, an updated life-cycle cost estimate should show the source of data underlying the software design and cost estimating model, and the equations used to estimate the costs of this large effort. The most recent Acquisition Decision Memorandum dated July 7, 2011, states that the Transformation Program Office must work closely with the DHS Cost Analysis Division to complete a life-cycle cost estimate by September 30, 2011.

Improved Acquisition Planning Could Help USCIS Avoid Further Delays and Potential Cost Overruns for the Transformation Program

USCIS Does Not Have Reasonable Assurance That Future Milestones Are Achievable

USCIS is continuing to manage the Transformation Program without specific acquisition management controls such as reliable schedules and as a result it will be difficult for USCIS to provide reasonable assurance that it can meet its future milestones. USCIS has established schedules for the first release of the Transformation Program, but our analysis shows that these schedules are not reliable as they do not meet best practices for schedule estimating. For example, the schedules did not identify all activities to be performed by the government and solutions architect. Additionally, USCIS has encountered a number of challenges in implementing the schedules, such as assumptions that have not been met regarding the time frames that either the solutions architect or USCIS would complete certain tasks. For example, according to an April 2010 program management review, USCIS planned to provide the solutions architect with two technical environments to conduct production and testing activities by December 2010. However, USCIS has since revised the schedule due to challenges in procuring hardware and software needed before these two environments were ready for the solutions architect.

Based on the revised schedule, delivery of the technical environments was delayed to April 2011, according to USCIS, so that OIT could take actions to address the delay, such as borrowing equipment until a contract protest was resolved, and providing the solutions architect with a walk-through of the technical environments to ensure it met their needs. Program officials stated that factors outside their control, such as contract protests or review and approval of system requirements, have contributed to challenges in implementing the schedules. According to program officials, defining and developing requirements was expected to last about 2 ½ to 3 months. However, USCIS completed the requirements in 9 months, which included review and validation of these requirements by agency leadership. Program officials stated that detailed reviews and approval by agency leadership took longer than expected.

Best practices in schedule estimating state that a comprehensive schedule should include a schedule risk analysis, so that the risk to the estimate if items are delayed can be modeled and presented to management including, among others, assumptions on equipment deliveries or length of internal and external reviews. However, according to a January 2011 program management review, as changes to the program were happening rapidly, there was no analysis completed to determine the impact on the schedule.

In addition to the challenges USCIS has encountered in carrying out the schedules as originally planned, on the basis of our analysis we found that the current schedules for the first release of the Transformation Program are of questionable reliability. Best practices state that the success of a large-scale system acquisition program, such as the Transformation Program, depends in part on having reliable schedules that identify:  when the program’s set of work activities and milestone events will  how long they will take; and  how they are related to one another.

Among other things, reliable schedules provide a road map for systematic execution of a program and the means by which to gauge progress, identify and address potential problems, and promote accountability. Our research has identified nine best practices associated with developing and maintaining a reliable schedule. To be considered reliable, a schedule should meet all nine practices. In a July 2008 memorandum, DHS’s Under Secretary for Management endorsed the use of these scheduling practices and noted that DHS would be using them. The nine scheduling best practices are summarized in table 5.

The Transformation Program has 18 individual schedules. Table 6 summarizes the findings of our assessments of two of these individual schedules as of November 2010 representing the bulk of the Transformation Program efforts and those most critical to the production of USCIS ELIS. Specifically, these schedules track activities associated with USCIS’s OIT and the solutions architect. TPO is responsible for managing key acquisition functions associated with the Transformation Program; thus, USCIS is responsible for tracking and oversight of the OIT and solutions architect’s activities and associated schedules. Appendix I includes a detailed discussion of our analysis.

As shown above, the two Transformation Program schedules, for the most part, did not substantially or fully meet the nine best practices. For example, neither the OIT nor the USCIS-approved solutions architect schedule contained detailed information for Release A activities beyond March 2011. In addition, both schedules were missing a significant number of logic links between activities which indicate activities that must finish before others and which activities may not begin until others have been completed. While we cannot generalize these findings to all 18 schedules, our review raises questions about the reliability of the program’s schedules.

Based on our discussions with the Transformation Program’s lead program scheduler, this condition stems, in part, from the “aggressiveness of the Transformation Program to implement most of the capabilities within the first 3 years of the program,” and a lack of program management resources for developing knowledge to create and maintain schedules. Moreover, and regardless of the aggressiveness of the solution, our best practices call for schedules to reflect all activities— government, contractors, and any other necessary external parties— essential for successful program completion. As such, neither of the Transformation Program schedules we reviewed substantially met this practice. Furthermore, as demonstrated in the challenges USCIS has encountered in carrying out the schedule as originally planned, not including all work for all deliverables, regardless of whether the deliverables are the responsibility of the government or contractor, may result in confusion among team members and lead to management difficulties because of an incomplete understanding of the plan and of the progress being made.

Collectively, and moving forward, not meeting the nine key practices increases the risk of schedule slippages and related cost overruns and makes meaningful measurement and oversight of program status and progress, as well as accountability for results, difficult to achieve. For example, in June 2011, a program management review noted a schedule risk if the development, testing, and deployment process slip again. This could result in USCIS being unable to deliver the first release in December 2011. A schedule risk analysis could be used to determine the level of uncertainty and to help mitigate this risk. Similarly, capturing and sequencing all activities, as outlined in best practices, could help identify the extent to which other activities linked to this schedule risk are affecting its progress. Furthermore, without the development of a schedule that meets scheduling best practices, it will be difficult for USCIS to effectively monitor and oversee the progress of an estimated $1.7 billion to be invested in the acquisition of USCIS ELIS.

In August 2011, TPO provided us the updated USCIS-approved solutions architect schedule. Program officials indicated that this updated schedule addressed some areas in which their previous schedule was deficient according to our assessment of the nine scheduling best practices. For example, they said that this schedule included activities through December 2011 rather than only through March 2011. In addition, officials indicated that they have confidence in meeting the December 2011 milestone. Specifically, they said that USCIS and the solutions architect have tested over 70 percent of the Release A capabilities that are to be released in December 2011, and demonstrated these capabilities to the USCIS leadership team in August 2011. On the basis of our analysis of the updated USCIS-approved solutions architect schedule, we determined that the updated schedule did not address many of the deficiencies we identified in the earlier version of the schedule. For example, the USCIS- approved solutions architect schedule did contain activities through December 2011, but logical links were missing between activities indicating which activities must finish before others and which activities may not begin until others have been completed. The schedule’s authorized work has therefore not been established in a way that describes the sequence of work, which prevents the schedule from meeting other best practices, such as establishing a critical path or developing a schedule risk analysis. Thus, the updated USCIS-approved solutions architect schedule does not fully meet all nine key practices, making meaningful measure and oversight of program status and progress difficult to achieve. USCIS did not provide us with an updated OIT schedule; therefore, we were unable to determine to what extent many of the deficiencies we identified in the earlier versions were addressed.

Further, USCIS established the Transformation Program as a long-term program made up of five releases to procure, test, deploy, and maintain USCIS ELIS, but USCIS officials confirmed in October 2010 that there was no integrated master schedule for the entire Transformation Program. In addition, the schedules we received in August 2011 were also not integrated into a master schedule. According to best practices, an integrated master schedule is to contain the detailed tasks necessary to ensure program execution and is a required document to develop key acquisition planning documents under DHS acquisition management guidance. Among other things, best practices and related federal guidance call for a program schedule to be programwide in scope, meaning that it should include the integrated breakdown of the work to be performed by both the government and its contractors over the expected life of the program. According to program officials, when the Transformation Program’s planning efforts began, USCIS was unable to develop an integrated master schedule for the Transformation Program due to the complexity of integrating the numerous individual schedules and the lack of skilled staff necessary to develop and manage such an integrated master schedule. In addition, program officials explained that scheduling software to develop and maintain individual schedules was not used by every organization performing transformation work, such as OIT, even though the program issued guidance in August 2010 to all organizations on scheduling best practices, including the use of scheduling software.

As an alternative to an integrated master schedule and for ease of reporting to the ARB and other senior officials, TPO developed a high- level tracking tool summarizing dates and activities for the first release of the program and based on individual schedules such as the OIT and solutions architect schedule, which are not directly managed by TPO. According to program officials, in a September 2010 briefing to agency leadership, this high-level tracking tool created capacity for USCIS to analyze the schedule. In this briefing, program officials stated TPO used the high-level tracking tool to ensure coordination and alignment of activities by collaborating with staff responsible for the management of individual schedules. However, this tracking tool is not an integrated master schedule as it does not integrate all activities necessary to meet the milestones for Release A; rather, it is a selection of key activities drawn from the individual schedules maintained by USCIS components and the solutions architect. Moreover, the Transformation Program Manager expressed concern in a May 2011 program management review that the information reported in the high-level tracking tool was not being reported in the individual schedules. In addition, this tracking tool is not an integrated master schedule because it does not show activities over the life of the program. That is, there are no dates or activities for when the other four releases’ set of work activities will occur, how long they will take, and how they are related to one another. As a result, it will be difficult for program officials to predict, with any degree of confidence, how long it will take to complete all five releases of the Transformation Program. It will also be difficult for program officials to manage and measure progress in executing the work needed to deliver the program, thus increasing the risk of cost, schedule, and performance shortfalls. Lastly, USCIS’s ability to accurately communicate the status of Transformation Program efforts to key stakeholders such as its employees, Congress and the public will be hindered.

Because USCIS lacks reliable schedules, its ability to develop reliable life-cycle cost estimates is hampered. As outlined by DHS acquisition management guidance, a life-cycle cost estimate is a required and critical element in the acquisition process. USCIS has developed and updated the life-cycle cost estimate for the Transformation Program, but USCIS’s individual schedules for the Transformation Program do not meet best practices for schedule estimating, thus raising questions about the credibility of the program’s life-cycle cost estimates. For example, neither the OIT nor the solutions architect schedule fully captured all activities to be performed by the government and contractor. Therefore, when USCIS is developing the life-cycle cost estimate there is an incomplete understanding of the work necessary to accomplish the five releases of the Transformation Program. Further, in the case of both individual schedules, the absence of a schedule risk analysis makes it difficult for officials to account for the cost effects of schedule slippage when developing the life-cycle cost estimate. Further, a reliable life-cycle cost estimate is essential for helping the program determine how much funding is needed and whether it will be available to achieve the Transformation Program’s goals.

Best practices that we have previously identified for cost estimation state that because some program costs such as labor, supervision, rented equipment, and facilities cost more if the program takes longer, a reliable schedule can contribute to an understanding of the cost impact if the program does not finish on time. Meeting planned milestones and controlling costs are both dependent on the quality of a program’s schedule. An integrated schedule is key to managing program performance and is necessary for determining what work remains and the expected cost to complete the work.

Conclusions

USCIS’s effort to develop a modern, automated system for processing benefit applications and addressing the many current program inefficiencies has been in progress for nearly 6 years. The program is now more than 2 years behind its planned deployment schedule for implementing the agencywide transformed business process, and given the enormity, significance, and complexity of this transformation, it is essential that USCIS ensures it takes the proper steps for implementation. Although only one benefit type is expected to be available for online account management and adjudication in December 2011, the decision to channel resources and efforts to focus on ensuring the core businesses are ready for a December 2011 launch prior to making other application types available for online-processing appears to be prudent.

Moving forward, it is essential that USCIS consistently follows DHS acquisition management guidance to best position the department to develop and share information within the department and with Congress and the public that can be relied upon for purposes of informed decision making. Moreover, ensuring that the program’s schedules are consistent with schedule estimating best practices and integrated through an integrated master schedule would better position USCIS to reliably estimate the amount of time and effort needed to complete the program. Reliable schedules could also assist USCIS in developing and maintaining a complete and reliable life-cycle cost estimate for the program which is essential for helping the program determine how much funding is needed and whether it will be available to achieve the Transformation Program’s goals.

Recommendations for Executive Action

To help ensure that USCIS takes a comprehensive and cost-effective approach to the development and deployment of transformation efforts to meet the agency’s goals of improved adjudications and customer services processes, we recommend that the Director of USCIS take the following three actions: 1. Ensure program schedules are consistent with the nine estimating best practices. 2. Develop and maintain an Integrated Master Schedule consistent with these same best practices for the Transformation Program. 3. Ensure that the life-cycle cost estimate is informed by milestones and associated tasks from reliable schedules that are developed in accordance with the nine best practices we identified.

Agency Comments and Our Evaluation

We provided a draft of this report to DHS for comment. DHS provided written comments, which are reprinted in Appendix II. In commenting on this report, DHS, including USCIS, concurred with the recommendations.

DHS's letter outlined the actions that USCIS is taking action or has taken to address each recommendation. Regarding the first recommendation to ensure program schedules are consistent with best practices, DHS stated that USCIS is incorporating the nine schedule estimating best practices we identified into Transformation Program management reviews, as well as the Acquisition Review Board review. Regarding the second recommendation to develop and maintain an Integrated Master Schedule consistent with these same best practices for the Transformation Program, DHS stated that USCIS will develop an Integrated Master Schedule to depict the multiple tasks, implementation activities, and interrelationships needed to successfully develop and deploy the Transformation Program. Regarding the third recommendation to ensure that life-cycle cost estimates are developed in accordance with the nine best practices, DHS stated that they will refine the Transformation Program life-cycle cost estimate in accordance with GAO’s 12-Step Process for Cost Estimation. In addition, DHS noted that their revised master schedule will clearly identify work elements to ensure a reasonable and cost-effective timeframe for accomplishing the five releases associated with the program. If fully implemented, we believe that the actions that DHS identified will address our recommendations.

DHS also provided technical comments, which we have incorporated, as appropriate.

We are sending copies of this report to the Secretary of Homeland Security and other interested parties. The report also will be available at no charge on the GAO Web site at http://www.gao.gov.

If you or your staff members have any questions about this report, please contact me at (202) 512-8777 or stanar@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix III.

Appendix I: Detailed Results of GAO Assessment of USCIS’s Detailed Transformation Program Schedules

In prior work, we have identified nine best practices associated with effective schedule estimating. These are (1) capturing all activities; (2) sequencing all activities; (3) assigning resources to all activities; (4) establishing the duration of all activities; (5) integrating activities horizontally and vertically; (6) establishing the critical path for all activities; (7) identifying float time between activities; (8) conducting a schedule risk analysis; and (9) updating the schedule using logic and durations. We assessed the extent to which two detailed schedules, Office of Information Technology (OIT) and the solutions architect, dated November 2010, met each of the nine practices. We characterized whether the schedules met each of the nine best practices as follows:  Not met—the program provided no evidence that satisfies any portion of the criterion.  Minimally met—the program provided evidence that satisfies less than one-half of the criterion.  Partially met—the program provided evidence that satisfies about one-half of the criterion.  Substantially met—the program provided evidence that satisfies more than one-half of the criterion.  Met—the program provided evidence that satisfies the entire criterion.

Tables 7 and 8 provide the detailed results of our analysis of these schedules.

Appendix II: Comments from the Department of Homeland Security

Appendix III: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, Mike Dino, Assistant Director; Kathryn Bernet, Assistant Director; and Carla Brown, Analyst-in-Charge; managed this assignment. Sylvia Bascopé, Jim Russell, and Ulyana Panchishin made significant contributions to the work. Nate Tranquilly and Bill Russell provided expertise on acquisition issues. Tisha Derricotte and Jason Lee provided expertise on scheduling best practices. Frances Cook provided legal support. Linda Miller and Labony Chakraborty provided assistance in report preparation, and Robert Robinson developed the report’s graphics.

Related GAO Products

Acquisition Planning: Opportunities to Build Strong Foundations for Better Services Contracts. GAO-11-672. Washington, D.C.: August 9, 2011.

Opportunities to Reduce Potential Duplication in Government Programs, Save Tax Dollars, and Enhance Revenue. GAO-11-318SP. Washington, D.C.: March 1, 2011.

Secure Boarder Initiative: DHS Needs to Strengthen Management and Oversight of its Prime Contractor. GAO-11-6. Washington, D.C.: October 18, 2010.

Department of Homeland Security: Assessments of Selected Complex Acquisitions. GAO-10-588SP. Washington, D.C.: June 30, 2010.

GAO Cost Estimating and Assessment Guide: Best Practices for Developing and Managing Capital Program Costs. GAO-09-3SP. Washington, D.C.: March 2009.

Department of Homeland Security: Billions Invested in Major Programs Lack Appropriate Oversight. GAO-09-29. Washington, D.C.: November 18, 2008.

Defense Acquisitions: Realistic Business Cases Needed to Execute Navy Shipbuilding Programs. GAO-07-943T. Washington, D.C.: July 24, 2007.

USCIS Transformation: Improvements to Performance, Human Capital, and Information Technology Management Needed as Modernization Proceeds. GAO-07-1013R. Washington, D.C.: July 17, 2007.

Immigration Benefits: Additional Efforts Needed to Help Ensure Alien Files Are Located when Needed. GAO-07-85. Washington, D.C.: October 27, 2006.

Information Technology: Near-Term Effort to Automate Paper-Based Immigration Files Needs Planning Improvements. GAO-06-375. Washington, D.C.: March 31, 2006.

Immigration Benefits: Improvements Needed to Address Backlogs and Ensure Quality of Adjudications. GAO-06-20. Washington, D.C.: November 21, 2005.

Contract Management: Coast Guard’s Deepwater Program Needs Increased Attention to Management and Contractor Oversight. GAO-04-380. Washington, D.C.: March 9, 2004.