Immigration Benefits System: Better Informed Decision Making Needed on Transformation Program
Published: 2015-05-18T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Each year, the Department of Homeland Security's (DHS) USCIS processes millions of applications for persons seeking to study, work, visit, or live in the United States. USCIS has been working since 2005 to transform its outdated systems into an account-based system with electronic adjudication and case management tools that will allow applicants to apply and track the progress of their application online. In 2011, USCIS reported that this effort, called the Transformation Program, was to be completed no later than June 2014 at a cost of up to $2.1 billion.
Given the critical importance of the Transformation Program, GAO was asked to review it. This report (1) discusses the program's current status, including the impact of changes made, and (2) assesses the extent to which DHS and USCIS are executing effective program oversight and governance.
To do so, GAO reviewed DHS and USCIS documents, interviewed relevant officials, and compared program documentation and actions to DHS and USCIS policy and guidance and GAO and industry leading information technology practices.
section_title: What GAO Found
paragraphs: The U.S. Citizenship and Immigration Services' (USCIS) currently expects that its Transformation Program will cost up to $3.1 billion and be fully deployed no later than March 2019, which is an increase of approximately $1 billion and delay of over 4 years from its initial July 2011 baseline. In March 2012, the program began to significantly change its acquisition strategy to address various technical challenges (see table).
Key Changes to the Transformation Program's Acquisition Strategy
Source: GAO analysis of USCIS documentation. | GAO-15-415
These changes have significantly delayed the program's planned schedule, which in turn has had adverse effects on when USCIS expects to achieve cost savings, operational efficiencies, and other benefits. Among other things, USCIS has yet to achieve the goal of enhancing national security by authenticating users and integrating with external agency databases.
While the program's two key governance bodies have taken actions aligned with leading IT management practices, neither has used reliable information to make decisions and inform external reporting (see table). For example, one governing body's vote in March 2013 to migrate to a new architecture was based in part on savings that did not account for the added costs of merging data from the old architecture. The ability of USCIS, DHS, and Congress to effectively monitor program performance may be limited until these bodies more effectively use reliable information to inform their program evaluations.
Extent to Which Program Governance Bodies Met Leading Practices for Oversight
● Implemented ◐ Partially implemented ○ Not implemented
Source: GAO analysis of USCIS documentation. | GAO-15-415
section_title: What GAO Recommends
paragraphs: GAO is making recommendations to DHS components and offices to improve governance and oversight of the Transformation Program. DHS agreed with the recommendations, but did not agree with GAO's evaluation of the impact of changes made to the acquisition strategy. GAO maintains its position on the impact of changes, as discussed in the report.

section_title: Letter
paragraphs: Each year, U.S. Citizenship and Immigration Services (USCIS), within the  Department of Homeland Security (DHS), processes millions of  applications for more than 50 types of immigrant and nonimmigrant- related benefits for persons seeking to study, work, visit, or live in the  United States, and for persons seeking to become U.S. citizens. Having  an efficient system to process immigration benefits and citizenship  requests in a timely manner and accurately identify fraudulent and  criminal activity is essential for ensuring the integrity of the immigration  process. USCIS has long recognized the need to improve the underlying  technology infrastructure that supports its benefits and adjudication  processes.
We have previously reported on inefficiencies in USCIS’s current  benefits processing systems. These systems employ both electronic and  paper records, and therefore make sharing the information with other  government agencies (such as those monitoring for criminal or terrorist  activity) difficult and time-consuming.
To address these inefficiencies, USCIS embarked on a major initiative in  2005 to transform its outdated systems into an account-based system  with electronic adjudication and case management tools that would allow  applicants to apply and track the progress of their application online. In  2011, USCIS reported that this effort, called the Transformation Program,  was expected to be completed no later than June 2014 at a cost of up to  $2.1 billion.
Given the critical nature of the Transformation Program, you asked that  we review it. This report (1) discusses the program’s current status,  including the impact of changes made to the acquisition strategy, and (2)  assesses the extent to which DHS and USCIS are executing effective  program oversight and governance.
To determine the program’s current status and impact of changes, we  examined DHS and USCIS acquisition policy, and DHS and USCIS  documentation specific to the Transformation Program. Among other  things, we analyzed recent program planning documentation such as the  “Transformation Road Map” and compared it to the past acquisition  program baseline to determine cost overruns and schedule delays. In  addition, we reviewed meeting minutes and presentations from oversight  entities such as the Executive Steering Committee to further understand  program changes. We also reviewed related GAO and DHS Inspector  General reports to capture previously identified issues encountered by the  Transformation Program.
To determine the extent to which DHS and USCIS are executing effective  program oversight and governance, we reviewed agency acquisition  management policy, analyzed roles and responsibilities, and reviewed the  execution of these roles and responsibilities against relevant policy,  guidance, and best practices. Specifically, we analyzed DHS and USCIS  policy for acquisition management, such as Acquisition Management  Directive 102-01, to understand the roles and responsibilities of oversight  entities. We also reviewed and analyzed governance and oversight body  charters, meeting minutes, presentation slides and supporting materials,  and after action reports of various oversight entities; and compared these  artifacts and results to leading practices supported in the IT Investment  Management Maturity Framework and the Capability Maturity Model  Integration for Development.
We conducted this performance audit from August 2014 to May 2015, in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives. More details about the  objectives, scope and methodology can be found in appendix I.
subsections: section_title: Background
paragraphs: The mission of USCIS is to adjudicate applications and petitions for  immigration benefits, and requests for other action by individuals seeking  to become either a citizen of the United States or a lawful permanent  resident or to temporarily study, live, or work in this country. To  accomplish its mission, each year USCIS processes millions of  applications, petitions, and requests for more than 50 types of immigrant  and nonimmigrant-related benefits.
As we reported in 2007, the processing of millions of applications,  petitions, and requests has been hindered by inefficient, paper-based  processes, which has resulted in a backlog that peaked in 2004 at more than 3.8 million cases;  tens of thousands of missing or misplaced files; difficulties in verifying the identity of applicants; problems in providing other government agencies with the information  necessary to identify criminals and potential terrorists; and benefits being issued to unverified applicants.
subsections: section_title: USCIS Established a Transformation Program to Modernize Benefits Processing
paragraphs: Begun in 2005, the goals of the Transformation Program are to  modernize the paper-based immigration benefits process to enhance  national security and system integrity, and to improve customer service  and operational efficiency. The program is comprised of many systems,  each of which provides a service to facilitate operations, such as identity  management and risk and fraud analytics. The main component of the  program is the USCIS Electronic Immigration System (ELIS), which is to  provide case management for the adjudication of immigration benefits  and interface with other systems to achieve end-to-end processing of  immigration benefits. Once the system has been implemented, USCIS  expects that
Applicants will be able to establish an account with USCIS to file and  track the status of the application, petition, or request online.
The system will apply risk-based rules automatically to incoming  applications, petitions, and requests to identify potentially fraudulent  applications and national security risks.
Adjudicators will have electronic access to applications, petitions, and  requests, relevant policies and procedures, and external databases to  aid in decision making.
USCIS will have management information to track and allocate  workload.
The system will allow electronic linkages to other agencies, such as  the Departments of Justice and State, for data sharing and security  purposes. Appendix II lists systems internal and external to USCIS  with which USCIS ELIS will interface.
Figure 1 provides an overview of how applicants are to file for immigration  benefits and apply for citizenship using the new system.
Five core operational requirements are expected to form the foundation of  USCIS ELIS, which is to process and manage all applications. Table 1  describes the five core operational requirements.
Each core operational requirement is broken down into a series of  capabilities, features, and sub-features. For example, the Intake and  Account Management operational requirement has five capabilities, such  as the Establish/Authenticate portal account capability, which is broken  down into four features, including Manage Account Access. This feature  is further decomposed into five sub-features, including Provide Account  Management for Customers and Representatives. Appendix III provides  additional information on the planned capabilities for the Transformation  Program.
section_title: Key Aspects of USCIS’ Original Transformation Program Strategy
paragraphs: USCIS began implementing the Transformation Program by awarding  contracts for various phases, including for pilot projects and system  architecture. In fiscal year 2006, USCIS initiated three pilot projects and  one for a proof-of-concept effort.
Enumeration pilot: The pilot was a joint effort by USCIS and the then- United States Visitor and Immigrant Status Indicator Technology (US- VISIT) program to positively identify individuals through the  generation of a unique identifier permanently associated with a  person. The enumerator is created by the submission of fingerprint  and biographic data. Each time the person returns to USCIS for a  subsequent benefit application, biometric information is to be matched  to determine if the individual is the same person. This process is  intended to limit the number of times fingerprints are submitted for  background checks.
Biometric Storage System pilot: The Biometric Storage System was to  help streamline the established USCIS biometric and card production  processes and become the centralized repository for all USCIS  customer biometrics. The system receives the 10-print fingerprints  taken at application support centers along with related biographic  data, and submits them to the Federal Bureau of Investigation for a  fingerprint check and then to the US-VISIT Automated Biometric  Identification System for creation of the enumerator and for  permanent storage. The 10-print fingerprints and photos are then  associated with benefit cards that serve as travel documents. The  system is to store the images only as long as needed to facilitate  adjudication and production of benefits cards.
Digitization pilot: The Digitization pilot was comprised of the  Enterprise Document Management System, which stores electronic  documents along with metadata describing the documents, and the  Records Digitization Facility, which scans documents into electronic  format, creates the metadata, and transfers the document to the  document management system. The pilot tested the use of images by  adjudicators in their day-to-day work environment.
The proof-of-concept was intended to demonstrate the case processing  capability of the case management system. Specifically, it was to  demonstrate that the enumerator (unique identifier based on biometrics)  could link an applicant to related filings for that individual and assess  whether the case management system could be used to view digitized  files. According to the Transformation Program Management Plan, a post- implementation review completed in October 2007 determined that the  proof-of-concept met all of its primary objectives and that the  Transformation Program should continue with this approach.
In November 2008, USCIS awarded a solutions architect contract for  approximately $500 million to be allocated over a 5-year period to design,  develop, test, deploy, and sustain the Transformation Program by  November 2013. The overall strategy was to deliver the solution in two  increments: Increment 1, to include releases A and B, and Increment 2, to  include releases C, D, and E. Releases A and B included making USCIS  ELIS available to applicants applying for nonimmigrant benefits and much  of the functionality needed to operate the five core operational  requirements. Table 2 shows the order in which USCIS ELIS’s five  releases were to be deployed and the types of benefits to be made  available to applicants in each release.
In July 2011, DHS officially approved the Transformation Program’s  acquisition program baseline and supporting operational requirements.  The baseline included approved objectives (targets that reflect the most  likely cost and schedule) and approved thresholds (ceilings which, if  exceeded, initiate official re-planning actions). Table 3 details the cost  and schedule parameters of the program’s approved July 2011 baseline.
On May 22, 2012, USCIS launched the first release of USCIS ELIS  (release A1). This release included capabilities associated with all of the  core operational requirements, such as online account setup, case  management, case acceptance, applicant evidence intake, and notice  generation. Since May 2012, five primary releases, along with a series of  maintenance releases, have been deployed to add functionality to USCIS  ELIS, with new applicant processing added in two of the five releases.  Table 4 shows the date of each functional release and a description of the  capabilities in the release.
section_title: DHS Acquisition Management
paragraphs: USCIS ELIS is to be managed consistent with DHS’s acquisition  management process. DHS’s Acquisition Management Directive 102-01  and its Instruction Manual 102-01-001 (the guidebook) establish the  department’s policies and processes for managing major acquisition  programs. Acquisition Management Directive 102-01 provides high-level  direction to program managers to help determine funding needs,  capability requirements, and schedule, and the guidebook provides  specific direction to program managers on how to implement the directive,  such as how to address a program breach.
DHS’s Deputy Secretary and Under Secretary for Management serve as  the acquisition decision authorities for the department’s largest acquisition  programs, those with life-cycle cost estimates of $1 billion or more. The  Under Secretary for Management also serves as DHS’s Chief Acquisition  Officer, and in this role is responsible for the management and oversight  of the department’s acquisition policies and procedures.
To assist in the acquisition oversight process, the Under Secretary for  Management is supported by two offices within the department. The  Office of Program Accountability and Risk Management (PARM) is  responsible for DHS’s overall acquisition governance process. The Office  of the Chief Information Officer (OCIO) is responsible for, among other  things, setting departmental information technology (IT) policies,  processes, and standards. It is also responsible for ensuring that IT  acquisitions comply with DHS IT management processes, technical  requirements, and the approved enterprise architecture.
Within the OCIO, the Enterprise Business Management Office is to  ensure that the department’s IT investments align with its missions and  objectives. As part of its responsibilities, this office periodically assesses  IT investments to gauge how well they are performing through a review of  program risk, human capital, cost and schedule, and requirements. These  assessments serve as the criteria for reporting to the IT Dashboard for  oversight by the Office of Management and Budget (OMB).
In March 2015, we reported that DHS acquisition policy does not define  the differences in the role of PARM and the OCIO’s Enterprise Business  Management Office in oversight of major IT acquisitions. In particular,  the functions of PARM and the Enterprise Business Management Office  may overlap. Further, we reported that programs report to PARM and the  Enterprise Business Management Office through two separate  information systems, which further complicate the distinction. In order to  ensure consistent, effective oversight of DHS’s acquisition programs, we  recommended that the Secretary of DHS direct the Under Secretary for  Management to develop written guidance to clarify the roles and  responsibilities of PARM and OCIO Enterprise Business Management  Office for conducting oversight of major acquisition programs. DHS  concurred with our recommendations.
section_title: DHS Acquisition Life Cycle
paragraphs: DHS Acquisition Management Directive 102-01 establishes that a major  acquisition program’s decision authority is responsible for reviewing and  approving the movement of the program through four phases of the  acquisition life cycle at a series of five acquisition decision events. These  acquisition decision events, which can be more than 1 year apart, provide  the acquisition decision authority an opportunity to assess whether a  major program is ready to proceed through the life cycle phases.  Following are the four phases of the acquisition life cycle, as established  in DHS acquisition policy:  1.  Need: Department officials identify that there is a need, consistent  with DHS’s strategic plan, justifying an investment in a new capability  and the establishment of an acquisition program to produce that  capability. This phase concludes with the acquisition decision  authority granting the acquisition program approval to proceed at  acquisition decision event 1.  2.  Analyze/Select: A designated program manager reviews alternative  approaches to meeting the need and recommends a best option to  the acquisition decision authority. This phase concludes with the  acquisition decision authority granting the acquisition program  approval to proceed at acquisition decision event 2A.  3.  Obtain: The program manager develops, tests, and evaluates the  selected option. The acquisition decision authority may review the  acquisition program multiple times before granting the acquisition  program approval to proceed with particular acquisition activities.  Acquisition decision event 2B focuses on the cost, schedule, and  performance parameters for each of the program’s projects. The  program may also proceed through acquisition decision event 2C,  which focuses on low rate initial production. The phase concludes with  the acquisition decision authority granting the program approval to  proceed at acquisition decision event 3.  4.  Produce/Deploy/Support: DHS delivers the new capability to its  operators, and maintains the capability until it is retired. This phase  includes sustainment, which begins when a capability has been  fielded for operational use; sustainment involves the supportability of  fielded systems through disposal, including maintenance.
Figure 2 depicts the acquisition life cycle established in the directive.
Two important aspect of the acquisition decision events are the review  and approval of key acquisition documents critical to establishing the  need for a major program, its operational requirements, an acquisition  baseline, and testing and support plans. Table 5 describes the key  acquisition documents requiring department-level approval before a  program can move to the next acquisition phase.
Figure 3 depicts the acquisition life cycle and associated program  documentation requirements.
We have previously reported on the management and development of the  USCIS Transformation Program. In July 2007, we evaluated the  Transformation Program strategic and expenditure plans to determine the  extent to which these plans had prepared USCIS to carry out its program.  We reported that the agency’s plans partially or fully addressed most key  practices but more attention was needed in certain areas such as  performance measurement and IT management. We also reported that  the plans provided some information on costs and revenues, but that  USCIS had not finalized its acquisition strategy and, therefore, cost  estimates were uncertain.
To improve its transformation strategy and fully address congressionally  requested information, we recommended that the Director of USCIS  address gaps in plans in the areas of performance measurement,  strategic human capital management, communications, and IT  management practices. DHS concurred with our recommendations.
Between September 2009 and September 2011, USCIS took steps to  address the gaps identified in our report, such as finalizing a balanced set  of four performance measures and establishing fiscal year 2012 targets  that aligned with transformation goals for customer satisfaction, decisional  accuracy, timeliness, and efficiency.
In November 2011, we assessed the extent to which USCIS had  followed DHS acquisition policy in developing and managing the  Transformation Program. We reported that the agency had not  consistently followed the acquisition management approach that DHS  outlined in management directives in developing and managing the  program. For example, USCIS did not complete several acquisition  planning documents required by DHS policy prior to moving forward with  an acquisition approach, which contributed to schedule delays and  increased program costs. To help ensure that USCIS used a  comprehensive and cost-effective approach to the development and  deployment of transformation efforts to meet the agency’s goals of  improved adjudications and customer services processes, we  recommended that the Director of USCIS develop and maintain an  integrated master schedule consistent with best practices for the  Transformation Program and ensure that the life-cycle cost estimate be  informed by milestones and associated tasks from reliable schedules.  The agency concurred with our recommendations and has begun work to  address them.
section_title: Transformation’s Deployment Schedule Is Significantly Delayed, Causing Missed Cost Savings and Deferred Mission Benefits
paragraphs: The Transformation Program has changed its system acquisition strategy,  which has contributed to significant delays in the program’s planned  schedule. As of March 2015, USCIS ELIS functionality deployed in the  program’s initial releases is still in operation. Moving forward, USCIS  estimates that its Transformation Program will now cost up to $3.1 billion  and be fully deployed no later than March 2019. This is an increase of  approximately $1 billion and a delay of over 4 years from its initial  approved baseline. In addition, several major changes were made to the  Transformation Program acquisition strategy to help address concerns  about delays and cost overruns, including changes to the software  development methodology, contracting approach, and program  architecture. However, the plans for this new approach have yet to be  formally approved. Nevertheless, despite the lack of an updated and  approved program baseline, USCIS has begun system acquisition by  awarding contracts for planning and development. Moreover, the delay in  the program’s planned schedule has in turn impacted USCIS’s ability to  achieve cost savings, operational efficiencies, and other benefits.
subsections: section_title: Transformation Program’s Baseline Estimates and System Acquisition Strategy Have Significantly Changed
paragraphs: As of March 2015, the program had work underway to maintain existing  operations as well as to transition to a new system architecture. More  specifically, USCIS ELIS capabilities developed and implemented in  releases A1 through A2.5 were still in operation. To maintain this  functionality, USCIS extended its initial Solution Architect contract,  originally scheduled to end in 2014, through May 2015. This functionality  is expected to be replaced under a new architecture (as discussed later in  this report). The program also had transition activities underway to test  this new architectural environment. In particular, USCIS conducted a  limited deployment for replacing and renewing permanent resident cards  in November 2014 and fully deployed an initial set of capabilities in March  2015. Following this deployment, USCIS expects to begin  decommissioning the existing capabilities.
With regard to program baseline estimates, the Transformation Program  is expected to cost no more than $3.1 billion and to be deployed no later  than March 2019. This is an increase of approximately $1 billion and  delay of over 4 years from the initial baseline approved in July 2011.  According to program officials, the Transformation Program is expected to  deliver the same system capabilities as initially intended; however, the  plan for deploying such capabilities has been revised.
Table 6 compares the Transformation program’s current draft baseline to  its initial baseline plans.
In addition to the changes made to the program baseline estimates,  several major changes were made to the Transformation Program  acquisition strategy. These changes are summarized in table 7 and  discussed in more detail below.
Software development methodology: The Transformation program  transitioned from a traditional waterfall approach to an Agile one. Agile  software development is consistent with industry best practices and  existing incremental development requirements. Further, our work  has shown that, if performed effectively, this approach can provide  more flexibility to respond to changing agency priorities, and allow for  easier incorporation of emerging technologies and termination of  poorly performing investments with fewer sunk costs.
Under the previous approach, USCIS ELIS was to deploy the program  under five major releases. The non-immigrant line of business was to  be delivered across two releases, A and B. The three remaining lines  of business—immigrant, humanitarian, and citizenship—were to be  delivered through releases C, D, and E, respectively. Under the Agile  approach, the program plans to deliver smaller sets of functionality  every 4 weeks. These shorter-term releases are intended to culminate  in a more complete release of a USCIS business line in generally no  longer than 6 months. The program estimates it will move through 16  releases prior to achieving full operating capability.
Contracting approach: In October 2013, the DHS Deputy Chief  Procurement Officer approved a revision to the program’s acquisition  plan. This updated acquisition plan reflected a transition from a  primary contract for a single solution architect to a series of  contracts for tasks that had previously been provided by the solution  architect (e.g., requirements development and testing).
The Office of Transformation Coordination, with support from the  Office of Information Technology, has assumed responsibility for  integrating and overseeing the work performed under at least nine  different contracts. For example, training of USCIS personnel who will  interact with USCIS ELIS is provided through one contract, while  integration of software code from multiple development teams will be  handled through a different contract. In addition, software  development will be handled by multiple contractors, each operating  on 6-month contracts that can be renewed depending on the  performance of the contractor.
According to USCIS officials and briefings to senior leadership, this  new acquisition plan is to allow the Transformation Program to  transition away from a solution that delivered deficient software code  and an overly complex solution. However, the new acquisition  approach also introduces risk to the program. According to  documentation supporting the Transformation Program acquisition  plan, the primary risk of this approach is that the contract incentive  structure may not lead to cooperative behavior on the part of the  various contractors. Our prior work on approaches of this nature has  also shown that as the number of contracts and contractual  relationships increase, so does program complexity. Further, the  schedule and performance risks that arise from higher program  complexity typically result in greater costs.
Program architecture: In March 2013, the Transformation program  was granted approval by its executive steering committee to move to  a new, more modern, solution architecture. Design and development  of this new architecture began in September 2012 and was intended  to simplify the existing USCIS ELIS architecture by using fewer  commercial products and including open source software; support  enterprise solutions that could be reused across USCIS rather than  limiting their use to USCIS ELIS; and allow for easier scalability to  accommodate surges in the benefits processing by using cloud  services.
According to the USCIS Chief Information Officer, none of the  hardware or software from the existing architecture will be used as  part of the new architecture, with the exception of some licenses  associated with risk and fraud services. Accordingly, the program will  rebuild functionality previously developed in releases A1 through  A2.5, such as support for the I-526 form. The program plans to build  this functionality in releases 5.1, 5.2, and 5.3. Appendix III provides  additional information on the capabilities (releases) of USCIS ELIS as  well as when they are to be made available.
To transition to and test the efficacy of the new architecture, USCIS  planned a limited and full deployment for the replacement and  renewal of permanent resident cards (form I-90) for the first release of  system functionality. The Transformation Program conducted the  limited deployment in November 2014. In May 2015, the Director of  Operational Test and Evaluation signed off on an operational  assessment of the limited deployment.
According to the USCIS Chief Information Officer, the three key changes  made to the acquisition strategy were in conjunction with a March 2012  OMB-facilitated TechStat review with DHS of this program. As part of  the root-cause analysis to inform the TechStat review, the USCIS Chief  Information Officer noted that delays and cost overruns were partly the  result of the solution architect delivering deficient software code and  performing at an unacceptably low rate of productivity.
section_title: Despite a Lack of Formal DHS Approval of Current Plans, Transformation Has Proceeded with System Acquisition Work
paragraphs: The Transformation program’s current baseline estimates and plans have  yet to be formally approved by the DHS Undersecretary for Management  (the acquisition decision authority for this program); as a result, the  program has not been in compliance with DHS acquisition policies and  procedures since 2013. According to these policies and procedures, any  breaches to the approved acquisition program baseline (cost, schedule,  and performance parameters) require that, within 90 days, a new baseline  be approved or a program review be conducted to review the proposed  baseline revisions and recommendations to the acquisition decision  authority.
For the Transformation Program, neither of these actions has taken place  since 2013—when the program breached its approved acquisition  program baseline (from July 2011). According to the component lead in  PARM, a new baseline was not approved because supporting acquisition  planning documentation was not yet mature. Despite the lack of an  updated and approved program baseline, USCIS has begun to implement  its current system acquisition plans. For example, in October 2013 and  February 2014, the Acting Under Secretary for Management allowed  USCIS to award a series of contracts to develop the pilot for the new  architecture and re-build existing functionality, as well as to proceed with  planning and development of future releases. According to the USCIS  component lead for PARM, development of the USCIS ELIS system is  limited under the approved contracts. However, according to USCIS  planning documents, the releases currently being developed make up at  least 66 percent of the total capabilities of USCIS ELIS.
According to program officials, an acquisition decision event is expected  to occur at the beginning of April 2015, which should bring the  Transformation Program back into compliance with DHS policy.  Specifically, the acquisition decision (referred to as 2B in the DHS  acquisition life cycle) is intended to obtain the DHS Under Secretary for  Management’s approval of a new acquisition program baseline. However,  as discussed later in this report, this acquisition decision event has  already been rescheduled twice by the department—initially set to occur  in September 2013 and then in September 2014. Figure 4 provides a  timeline of past and current program events.
Until the Transformation program has a revised program baseline  (reflecting the significant changes in strategy) that has been approved at  the departmental level, program accountability for cost, schedule, and  operational commitments will continue to be limited.
section_title: Changes to Transformation Program’s Acquisition Strategy Have Contributed to Missed Cost Savings and Deferral of Operational Efficiencies and Other Benefits
paragraphs: The changes in the Transformation Program’s acquisition strategy have  significantly impacted the program’s planned schedule targets, which in  turn have had negative effects on USCIS’s ability to achieve cost savings,  operational efficiencies, and other benefits.
In justifying a transition to the new architecture, the program projected an  accelerated completion date of August 2016. Since then, draft acquisition  documents have pushed the completion date out to no later than March  2019. According to program officials, the delay in getting to full  operational capability is due in part to unexpected or greater-than- expected challenges in implementing the software methodology,  contracting, and architecture changes. For example:
The development and test environments took longer than expected  due the complexity of standing up the environments for use.
A bid protest of the flexible Agile development services contract  required the program to adjust the schedule and extend the solution  architect contract and also contract for another team to continue  development work until the bid protest was resolved and development  work could be initiated under the contract. According to the Chief of  the Office of Transformation Coordination, the temporary development  teams that were used while the bid protest was ongoing performed  more slowly than what was projected for permanent development  teams.
The program determined that it would be unable to re-use existing  software and hardware purchased by the solution architect and in use  by USCIS. As a result, work performed under the solution architect  contract would need to be redeveloped as part of future releases. This  additional work was not initially planned, and therefore expected to  require additional effort.
The schedule delays in system delivery have in turn hampered USCIS’s  ability to realize cost savings associated with the Transformation  Program. For example, the business case for the program highlighted  cost savings that would be realized from decommissioning of legacy  systems upon full deployment of USCIS ELIS. Each of these legacy  systems must remain operational to allow USCIS to perform its mission  until an alternative option is available. In fiscal year 2014, the total cost of  maintaining systems that could have been decommissioned if USCIS  ELIS had been fully operational was approximately $71 million. In  addition, the business case for the Transformation Program also identified  cost savings from reducing data entry and mail handling costs. USCIS will  continue to incur such costs while the program awaits full implementation.
In addition, the schedule delays have deferred USCIS’s ability to realize  operational improvements tied to the program. For example, the  Transformation Program is expected to implement organizational and  business process changes to better use IT. According to USCIS, this  increased use of IT should help achieve goals such as reducing the  immigration benefit backlog through business process change; improving  customer service through expanded electronic filing; and enhancing  national security by authenticating users and integrating with external  agency databases. Due to delays in the program, these improvements  have yet to be achieved.
Also, we have previously found weaknesses with existing processes and  systems by which USCIS processes benefits. The agency has cited the  Transformation Program as one effort that will help to address these  concerns. For example, in our December 2014 report, we found that  collection and maintenance of data for the K nonimmigrant visa process  (form I-129F) were not reliable or were not collected or maintained in a  reportable format. However, according to USCIS Service Center  Operations officials, the agency will be able to collect and maintain more  complete data through the deployment of the electronic I-129F petition in  USCIS ELIS. Thus, until the new system is deployed, weaknesses in  these existing processes may continue.
section_title: DHS Oversight Bodies Need to Improve Governance and Oversight of the Transformation Program
paragraphs: DHS acquisition policy identifies a process by which governance bodies  should evaluate cost, schedule, and performance to make decisions  about individual programs. Based on this process, multiple bodies govern  the program, requiring corrective actions when needed. In addition, the  department’s OCIO and PARM offices are tasked with conducting  evaluations of individual programs, which inform congressional and OMB  oversight. However, a lack of reliable information against which to monitor  the Transformation Program has inhibited the ability of these governing  and oversight bodies to make informed decisions.
subsections: section_title: Governance Bodies Have Mostly Taken Actions Aligned with Leading Practices but Decisions Are Based on Unreliable Information
paragraphs: The department’s acquisition policy requires that the DHS Acquisition  Review Board support the Under Secretary for Management by reviewing  major acquisition programs for proper management, oversight,  accountability, and alignment to the department’s strategic functions at  key decision points and other meetings, as needed. DHS acquisition  policy establishes that the review board be chaired by the acquisition  decision authority and consist of individuals that manage DHS’s mission  objectives, resources, and contracts. Table 8 identifies the members of  the Transformation Program’s Acquisition Review Board.
In May 2012, the Under Secretary for Management chartered the  Executive Steering Committee for the Transformation Program to help  improve program governance. In contrast to the Acquisition Review  Board, this committee assumed authority to oversee all aspects of the  execution of the Transformation Program between key decision points.  The Executive Steering Committee includes voting and non-voting  members from DHS and USCIS. Table 9 provides a list of the  committee’s voting and non-voting members.
Leading practices that we and others have identified note that oversight  is a critical element of an investment’s life cycle. To be effective,  governance bodies should, among other things:
Monitor a project’s performance and progress toward predefined cost  and schedule expectations: Governance bodies should measure the  actual value of project planning parameters, compare actual values to  estimates in the plan, and identify significant deviations.
Ensure that corrective actions are identified and assigned to the  appropriate parties at the first sign of cost, schedule, and/or  performance problems: Governance bodies should collect and  analyze issues based on predefined expectations and determine  corrective actions to address them.
Ensure that these corrective actions are tracked until the desired  outcomes are achieved: Governance bodies should track the  implementation of all corrective actions until the desired outcomes  occur.
Rely on complete and accurate data to review the performance of IT  projects and systems against stated expectations: Governance bodies  should integrate, measure, and analyze activities into the processes  of the project, including tracking actual progress and performance  against established plans and objectives. Analysis should account for  the quality (e.g., age, reliability) of all data that are used for the  analysis, regardless of the source of the data. The quality of data  should be considered to help select the appropriate analysis  procedure and evaluate the results of the analysis.
Table 10 shows the extent to which each of the governance bodies met  the leading practices for performing oversight.
As shown, the two governance bodies have implemented three of the  leading practices:
Acquisition Review Board: The board implemented the leading  practice associated with ensuring corrective actions are identified. For  example, during a program review on February 16, 2012, the board  discussed a request to provide the program with an additional $21.5  million to cover unplanned costs resulting from delays in the program  schedule. Based on this discussion, the review board identified a  series of action items, including a request for USCIS to return for an  additional review after delivery of the first release to request approval  to commit additional funding for development and delivery of the next  release. The Acquisition Decision Authority committed additional  funding in May 2012.
Executive Steering Committee: This body implemented two of the  leading practices. In particular, the committee fully ensured that  corrective actions were identified, and that corrective actions were  tracked until the desired outcomes were achieved. As part of its  reviews, the committee assigned corrective actions to specific  individuals and entities and tracked them until they had been  addressed. For example, in July 2013, the Executive Steering  Committee tasked the Office of Transformation Coordination with  forming a working group to propose potential options for sequencing  of forms and associated business processes to be included in  releases 10.0 through 16.0. This corrective action was tracked and  subsequently completed in September 2013.
The governance bodies partially implemented three leading practices:
Acquisition Review Board: The board partially addressed two  practices associated with monitoring cost and schedule against  predefined parameters and following corrective actions through to  completion. For example, with respect to monitoring against  predefined parameters, in February 2012, the review board evaluated  the planned costs of release A against the approved baseline amount  to determine if a request for additional funding was justified. However,  in February 2014, the program approved awarding a series of  contracts but did so without an approved acquisition program  baseline.
With respect to corrective actions, in a July 2013 acquisition decision  memorandum, the Transformation Program was directed to take the  corrective action of returning to the Acquisition Review Board for a re- baseline decision by September 2013 in order to establish new cost  and schedule parameters against which to evaluate the program. A  February 2014 acquisition decision memorandum again directed the  program to return for a re-baseline decision by September 2014.  However, as of February 2015, a re-baseline decision had not  occurred.
As another example, in a May 2012 acquisition decision  memorandum, the Transformation Program was directed to submit a  Systems Engineering Life Cycle Tailoring Plan by June 2012 that was  to document the Agile approach. According to the component lead in  PARM, this document was not submitted for approval.
Executing Steering Committee: The committee partially addressed  one practice associated with monitoring cost and schedule against  predefined parameters. For example, in April 2013, the committee  reviewed and approved the funding of approximately $31 million for IT  to support the Transformation Program. However, this evaluation did  not include an assessment of these costs against an approved  baseline for IT funding. Officials from the Office of Transformation  Coordination explained that the approval of funds was based on a  DHS-approved life-cycle cost estimate from April 2013. However, this  cost estimate was not supported by an approved schedule, current  requirements, and the other documentation required for a complete  understanding of the program, which would be provided by an  approved baseline.
Finally, the program’s two governance bodies did not implement two  leading practices:
Acquisition Review Board: The board did not make decisions based  on reliable cost, schedule, and performance information. For example,  in October 2013, the Acquisition Decision Authority approved a new  contract for continued development of the new architecture. In  addition, in February 2014, the authority approved a bridge contract  for the solutions architect to, among other things, assist in developing  the new architecture. However, operational requirements along with  an integrated master schedule and cost estimates had not been  approved to support these decisions. This is consistent with our  previous findings. For example, in November 2011, we found that  USCIS managed the program without specific acquisition  management controls, such as reliable schedules. As such, the  review board did not have reasonable assurance that it could meet its  future milestones or that it is developing a system that will achieve  performance goals. A summary of the corresponding  recommendations we made and their status is discussed earlier in this  report.
Executive Steering Committee: The committee did not make decisions  based on reliable cost, schedule, and performance information. For  example, in March 2013, the committee voted unanimously to migrate  to a new architecture for the Transformation Program. This approval  was based, in part, on the cost analysis reported by USCIS. However,  this analysis included cost savings that did not fully account for the  added costs for merging and migrating data from the old architecture.  In addition to excluding costs for data migration, the cost analysis also  projected significant cost savings to occur based on an accelerated  date for full operating capability. Specifically, that analysis was based  on an August 2016 date, but the full operating capability date in the  draft baseline is no later than March 2019. Officials from the Office of  Transformation Coordination added that the Executive Steering  Committee decision was also based on how the simplification of the  architecture would reduce risks to performance of the system.  Nevertheless, the decision was not supported by reliable cost and  schedule information.
Further, the committee approved capabilities to develop and deliver in  each release. However, the program did not have approved  documentation tying approved operating requirements to capabilities  and features on which such decisions could be made. Additionally, as  with the Acquisition Review Board, the committee relied on cost and  schedule information that had not been approved.
Until these governance bodies can base their reviews of performance on  timely, complete, and accurate data, they will be limited in their ability to  make timely decisions and to provide effective oversight.
section_title: Key External Reporting to Support Program Oversight is Based on Unreliable Information
paragraphs: In addition to governance from the decision-making bodies previously  discussed, two DHS offices—PARM and OCIO—assist in the oversight of  the Transformation Program. In particular, these offices perform periodic  reviews of the Transformation Program to assess general program health  and risk. These assessments are shared with external stakeholders with  oversight responsibilities, including OMB, Congress, and, in the case of  the OCIO, reported publicly on the Federal IT Dashboard. However, these  assessments of the Transformation Program reflected unreliable and, in  some cases, inaccurate information.
PARM develops periodic reports to ensure that various DHS programs  and their components within the agency satisfy compliance-related  mandates and improve investment management. To support these  efforts, PARM primarily draws on information reported to DHS’s official  system of record for acquisition program reporting. PARM’s assessments  are also informed by its participation in oversight meetings. For the  Transformation Program, the PARM component lead for USCIS attended  a majority of the Executive Steering Committee meetings from 2012  through 2014, among other things.
PARM reported assessments of the Transformation Program in its annual  Comprehensive Acquisition Status Reports for calendar year 2011 and  fiscal years 2013 and 2014, but these assessments were not always  based on reliable information. In its most recent status report, dated June  2014, PARM reported that the program was shown to be on target for  cost and schedule performance targets. The key events for the next 12  months included software development for releases 7 and 8, with  completion dates planned for December 17, 2014, and March 31, 2015,  respectively. However, as of June 2014, completion dates were revised  with only release 5.0 scheduled for deployment in early 2015. Release 7  was re-scheduled for December 2015, and release 8 for April 2016.  According to the USCIS PARM component lead, the report pulled data  from March 2014 and the program had not yet updated its acquisition  program reporting system with revised deployment dates. Further, as  previously discussed, revised cost and schedule baselines were not  established for the program after 2011 so any targets reported in the  status report reflected unapproved parameters reported by the program.
We reported in March 2015 on data issues and reporting limitations  affecting the accuracy of information contained in DHS’s acquisition  program reporting system and the reliability of acquisition status reports  departmentwide. To help address these concerns, we recommended  that DHS determine a mechanism to hold programs accountable for  entering data in its reporting system consistently and accurately, and to  adjust reporting requirements to include, among other things, reasons for  any significant changes in a program’s acquisition cost, quantity, or  schedule from the previous report. DHS concurred with the  recommendations and, if they are appropriately implemented, the  reliability and utility of these reports for decision making purposes should  be improved.
OCIO’s Enterprise Business Management Office also assists in  overseeing the Transformation program. In particular, it performs periodic  reviews that serve as the basis for program ratings that are publicly  reported on the Federal IT Dashboard. To support its assessments, OCIO  primarily draws on information reported to the Investment Management  System—DHS’s official system of record for reporting business cases to  the Office of Management and Budget. For the Transformation Program,  Enterprise Business Management Office representatives also participate  in the Executive Steering Committee and Program Management Review  meetings.
The Enterprise Business Management Office performed four  assessments of the Transformation Program from June 2013 through  June 2014, but these assessments were based on unreliable or  conflicting data. During that period, the program was evaluated as either  a moderately high or medium risk IT program. The most recent  assessment from June 2014 assessed the program as a category 3,  medium risk investment, which is an improvement from the February  2014 rating of 2, moderately high risk. The assessment states that the  program underwent a re-baseline for release 5.0 and, as a result,  reported an acceptable schedule variance and positive cost performance.  Further, it stated that an operational analysis of USCIS ELIS was  completed in August 2013. However, as discussed previously in this  report, the program has experienced over a four-year delay in its  schedule and has not performed a re-baseline to bring it back within cost  and schedule thresholds.
In addition, the operational analysis performed in August 2013 pertained  to the old USCIS ELIS architecture, which is scheduled for  decommissioning, and did not reflect the new USCIS ELIS architecture.  Instead, the Transformation Program has not yet been able to  successfully test to any of its key performance parameters.
Until the OCIO bases its review of performance on timely, complete, and  accurate data, its ability to effectively provide oversight and to make  timely decisions may be limited. In addition, it risks reporting information  on the Federal IT Dashboard that is inaccurate or otherwise misleading,  thus limiting OMB and congressional oversight.
section_title: Conclusions
paragraphs: The Transformation Program presents a significant opportunity for USCIS  to address long-standing challenges by reengineering the manner in  which it conducts business. However, the program has experienced  delays of over four years and a projected cost overrun of approximately  $1 billion since it first established its initial baseline. Each delay means  that more time will pass until USCIS is fully prepared to provide better  service to those applying for immigration and citizenship benefits and until  USCIS realizes the many benefits that the program is intended to provide,  such as improved customer service through expanded electronic filing  and enhanced national security by authenticating users and integrating  with external agency databases.
Changes made to the program’s acquisition strategy were intended to  help mitigate past technical and programmatic challenges; however, the  current plans have yet to be approved in accordance with departmental  policy. Among other things, the Transformation Program is proceeding  with a substantial amount of system development work without a current  and approved acquisition program baseline. As we have previously  concluded, developing a system without first establishing parameters  around which the system will be developed is risky and can result in  additional cost overruns and schedule delays. Moreover, program  accountability will continue to be limited in the absence of such  parameters.
While Transformation’s various oversight bodies are active in their  respective roles, decisions about the program had been made with  incomplete and inaccurate data. In addition, key program health  assessment reports shared with OMB and Congress are also unreliable.  The ability of USCIS, DHS, Congress, and OMB to effectively monitor  program performance and make informed program decisions will continue  to be limited until department-level governance and oversight bodies  more effectively use reliable program information to inform their program  evaluations.
section_title: Recommendations for Executive Action
paragraphs: To more fully understand the impact of recent changes to the  Transformation Program and help ensure improved Transformation  Program governance and oversight, we are making four  recommendations for the Secretary of DHS to direct to components within  DHS.
To help ensure that progress made by the Transformation Program can  be monitored against established and approved parameters, we  recommend that the Secretary of DHS direct the department’s Under  Secretary for Management to re-baseline cost, schedule, and  performance expectations for the remainder of the Transformation  Program.
To improve Transformation Program governance, we recommend that the  Secretary of DHS direct the Under Secretary for Management to ensure  that the Acquisition Review Board is effectively monitoring the  Transformation Program’s performance and progress toward a predefined  cost and schedule; ensuring that corrective actions are tracked until the  desired outcomes are achieved; and relying on complete and accurate  program data to review the performance of the Transformation Program  against stated expectations.
To improve Transformation Program governance, we further recommend  that the Secretary of DHS direct the DHS Under Secretary for  Management, in coordination with the Director of US Citizenship and  Immigration Services, to ensure that the Executive Steering Committee is  effectively monitoring the Transformation Program’s performance and  progress toward a predefined cost and schedule and relying on complete  and accurate program data to review the performance of the  Transformation Program against stated expectations.
To help ensure that assessments prepared by OCIO in support of the  department’s updates to the federal IT Dashboard more fully reflect the  current status of the Transformation Program, we recommend that the  Secretary of DHS direct the department’s Chief Information Officer to use  accurate and reliable information, such as operational assessments of the  new architecture and cost and schedule parameters approved by the  Under Secretary of Management.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this product to DHS for comment. In its written  comments, reproduced in appendix IV, DHS disagreed with our  conclusion about the impact of the changes made to the Transformation  acquisition strategy on the program’s cost and schedule, but concurred  with all four of our recommendations. DHS fully addressed one of our  recommendations and provided plans of action for the remaining three.  DHS also provided technical comments that we have incorporated into  the report as appropriate.
Regarding our conclusion, the department disagreed that changes to the  acquisition strategy delayed the program and added $1 billion to the  overall cost, citing changes in the time period covered by each program  cost estimate. Further, DHS did not believe we adequately considered the  potential benefits and lower risks that resulted from revising the  acquisition strategy for the program, stating that the original approach  would have far exceeded initial cost estimates and schedule.
Our report did not assess how much the program would have cost or  when it would have delivered a complete solution if USCIS continued to  pursue its original approach. Nevertheless, we maintain, as stated earlier  in this report, the acquisition program baseline approved in May 2015  reflects delays of nearly four years and approximately $1 billion in  additional cost when compared to the program’s July 2011 program  baseline. Our report also documents that cost increases and delays in  achieving full operational capability were due, in part, to unexpected or  greater-than-expected challenges in implementing the program’s new  approach. In response to the department’s comment about the time  period covered by the new cost estimate, we added additional detail to  the report to reflect that the newly-approved program baseline covers an  additional 11 years beyond what was addressed in the initial program  baseline. Even taking this into account, these additional years only  account for about $536 million of the increased costs.
Concerning the statement that we did not adequately consider the  potential benefits and lower risks of revising the acquisition strategy, we  disagree and stand by our analysis and findings. With respect to potential  benefits, we reported that some of the changes have the potential to  result in improvements. For example, our work has shown that, if  performed effectively, an Agile approach to software development can  provide more flexibility to respond to changing agency priorities and allow  for easier incorporation of emerging technologies. However, other  changes may introduce additional program risk. For example, our prior  work has shown that program complexity increases as the number of  contracts and contractual relationships increase. Further, the schedule  and performance risks that arise from higher program complexity can  result in greater program costs.
Regarding our recommendation to re-baseline cost, schedule, and  performance expectations for the remainder of the Transformation  Program, DHS provided evidence that it has been fully implemented.  Specifically, DHS provided an approved acquisition program baseline and  supporting documents. These documents demonstrate that the  department has approved a re-baseline of cost, schedule, and  performance expectations for the remainder of the Transformation  Program.
DHS concurred with our recommendation that the Acquisition Review  Board improve its governance of the program and described specific  actions taken that they believe will fully address it. For example, DHS  cited a signed memorandum of agreement outlining specific metrics  related to cost, schedule, and technical performance. It also cited  procedures to ensure acquisition decision memorandum actions are  tracked until desired outcomes are achieved and approval of all required  acquisition documentation prior to a re-baseline. If implemented  effectively, these actions have the potential to fully address our  recommendation.
DHS concurred with our recommendation on improving the governance of  the Transformation Program’s Executive Steering Committee. The  department also described planned actions to address the  recommendation, including ensuring that cost and schedule data are  presented and evaluated against the Acquisition Program Baseline. The  department estimated that it will be able to demonstrate successful  resolution of this recommendation by December 31, 2015. If implemented  effectively, these actions have the potential to fully address our  recommendation.
DHS also concurred with our recommendation to ensure that the  department’s Chief Information Officer updates the federal IT Dashboard  using accurate and reliable information about the program. The  department described actions it has taken and plans to take to address  this recommendation. For example, the DHS Chief Information Officer  has established an Integrated Product Team with PARM to identify gaps  in its assessment processes and establish better coordination to ensure  the timely availability of updated Acquisition Program Board information.  In addition, the department stated it would take steps to improve oversight  and data quality by consolidating two automated tools into a single  enterprise information management and repository system. The  department estimated that it will be able to demonstrate successful  resolution of this recommendation by December 31, 2015. If implemented  effectively, these actions have the potential to fully address our  recommendation.
As agreed with your offices, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 30 days from the  report date. At that time, we will send copies to interested congressional  committees and the Secretary of Homeland Security. This report will also  be available at no charge on our website at http://www.gao.gov.
If you or your staffs have any questions on matters discussed in this  report, please contact me at (202) 512-4456 or chac@gao.gov. Contact  points for our Offices of Congressional Relations and Public Affairs may  be found on the last page of this report. GAO staff who made major  contributions to this report are listed in appendix V.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: Our objectives were to (1) determine the status of the Transformation  Program, including the impact of changes made to the acquisition  strategy and (2) assess the extent to which the Department of Homeland  Security (DHS) and the U.S. Citizenship and Immigration Services  (USCIS) are executing effective program oversight and governance.
To determine the status of the Transformation Program and the impact of  changes made to the acquisition strategy, we reviewed and analyzed  recent program planning documentation and compared it to past  documentation and previously reported issues. Specifically, we reviewed  recent documentation such as a draft acquisition program baseline, test  and evaluation master plan and operational requirements, the acquisition  plan, annual budget figures, budget justification reviews, the annual  operating plan, capabilities and constraints documents, contracts and  associated statement of work, life-cycle cost estimate, privacy impact  assessments, system design documents, team process agreement, and  the Transformation road map.
To identify differences in past and current program expectations, we  reviewed past program documentation, such as the acquisition program  baseline, acquisition plan, business case analysis for process  alternatives, concept of operations, cost estimating baseline document,  exhibit 300 submissions to the Office of Management and Budget, life- cycle cost estimate, mission needs statement, operational assessments  and the associated operational requirements document, program  management plan, and solution architect contract and statement of work.  For example, we compared the operational requirements document  approved in 2011 to a draft operational requirements document from  2014 to determine changes in operating requirements and measures  used in operational testing. We also compared the cost savings and  benefits in the business case of process alternatives to release-specific  capabilities and constraints documents along with other planning  documentation to determine if recent changes to the program would  impact cost savings and benefits intended for the program.
In addition, we reviewed information from DHS and USCIS oversight  entities, such as meeting minutes and slide decks from the Executive  Steering Committee to further understand program changes. We also  reviewed related GAO and DHS Inspector General reports to capture  previously identified issues encountered by the Transformation Program.
To determine the extent to which DHS and USCIS are executing effective  program oversight and governance, we reviewed DHS acquisition  management policy, analyzed roles and responsibilities, and reviewed the  execution of these roles and responsibilities against relevant policy,  guidance, and leading practices. Specifically, we identified DHS and  USCIS policy for acquisition management such as Acquisition  Management Directive 102-01 to understand the DHS and USCIS  program oversight entities and expectations. We also identified criteria for  effective governance and oversight based on practices captured in the IT  Investment Management Maturity Framework and the Capability Maturity  Model Integration for Development. We reviewed and analyzed charters  of oversight bodies, meeting minutes, presentation slides and supporting  materials, and after action reports of various oversight entities, including  the Acquisition Review Board, Executive Steering Committee, the Office  of the Chief Information Officer (OCIO), the Office of Program  Accountability and Risk Management (PARM), the Component  Acquisition Review Board and the Product Management Team.
Based on this information, we assessed the extent to which the  Acquisition Review Board and Executive Steering Committee had  executed key governance and oversight practices. We assessed a  governance body as having implemented a practice if the practice was  shown to have been consistently applied (at least 80 percent of the time),  partially implemented if the practice was applied but on an inconsistent  basis (at least a quarter of the time), and not implemented if the practice  was not applied.
To determine the reliability of assessments produced by PARM, we  reviewed policy for the comprehensive acquisition status reports to  understand the criteria for assessing major acquisition programs. We  reviewed the master acquisition oversight list from 2010 through 2014 to  confirm that the Transformation Program was properly included in such  reviews and included in that list. We analyzed comprehensive acquisition  status reports in which the Transformation Program was identified. These  reports include calendar year 2011, fiscal years 2013 and 2014. We  compared these reports against current acquisition planning  documentation to gauge the accuracy of the information. For example, we  compared the tasks reported on the comprehensive acquisition status  reports that were planned for the next 12 months to the program schedule  presented to the Executive Steering Committee for that same time period.
We interviewed PARM officials to discuss any gaps. We also reviewed a  prior GAO report covering PARM oversight and reporting for the  comprehensive acquisition status report and the reliability of information  in the system supporting the process. In that report, GAO determined that  the data in the system were not sufficiently reliable for our purposes, as  also discussed in this report. Moreover, PARM officials acknowledged  that there were data accuracy issues with the system.
To determine the reliability of assessments produced by OCIO, we  reviewed the office’s program assessment scoring and Federal IT  Dashboard reporting guides to understand the criteria for assessing major  IT programs. We analyzed all reports and underlying scorecards for the  Transformation Program. These reports covered from June 2013 through  June 2014. We compared these reports against current acquisition  planning documentation to gauge the accuracy of the information. For  example, we compared information contained in the assessment narrative  to presentations made before oversight bodies for that same time period.  We interviewed OCIO officials to discuss any gaps. We determined these  assessments were unreliable as discussed in the report.
We conducted this performance audit from August 2014 to May 2015 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
section_title: Appendix II: USCIS ELIS Interfaces
paragraphs: The Transformation Program is to link developed systems, enterprise  services, and existing systems and capabilities to enable end-to-end  processing. The central project in this portfolio is the USCIS Electronic  Immigration System (USCIS ELIS), which is to enable electronic filing and  adjudicative case management. USCIS ELIS is to function as a  component of a larger architecture consisting of multiple systems,  services and interfaces. In particular, according to USCIS, USCIS ELIS is  to interface with existing systems, some of which are to be  decommissioned as the program is fully deployed. These components  must all function together as a whole to enable USCIS to operate in a  person-centric, paperless environment. Table 11 describes systems and  services with which USCIS ELIS is planned to interface.
section_title: Appendix III: Planned Transformation Program Capabilities to be Delivered by USCIS
paragraphs: The Transformation Program plans to be fully operational no later than  March 2019. In order to be fully operational, the program intends to  develop and deploy capabilities incrementally across various releases,  beginning with release 5.0. Table 12 describes the planned capabilities  to be delivered by release according to the Transformation Program road  map, as of December 2014.
section_title: Appendix IV: Comments from the Department of Homeland Security
section_title: Appendix V: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, individuals making contributions  to this report included Michael Holland (assistant director), Mathew  Bader, Kathryn Bernet, Nancy Glover, Martin Skorczynski, Nathan  Tranquilli, and Johnathan Wall.