TSA Modernization: Use of Sound Program Management and Oversight Practices Is Needed to Avoid Repeating Past Problems
Released: 2017-10-17T00:00:00

Letter

Background

As we have previously reported, transportation systems and facilities are vulnerable and difficult to secure given their size, easy accessibility, large number of potential targets, and proximity to urban areas. TSA’s mission is to protect the nation’s transportation systems by providing effective and efficient security to ensure freedom of movement for people and commerce. Accordingly, TSA is responsible for managing vetting and credentialing programs to ensure that individuals that transport hazardous materials or have unescorted access to secure or restricted areas of transportation facilities at maritime ports and TSA-regulated airports do not pose a security threat. In order to carry out this responsibility, TSA conducts background checks—known as security threat assessments— on individuals seeking an endorsement, credential, access, and/or privilege (hereafter called a credential).

Specifically, TSA reviews applicant information and searches government databases, such as criminal history records from federal, state, and local sources in the Federal Bureau of Investigation’s National Crime Information Center database and Terrorist Screening Database, which is the federal government’s consolidated terrorist watchlist. This information is used to determine whether the applicant has known ties to terrorism and whether the applicant may be otherwise precluded from obtaining a credential based on his or her immigration status and criminal history, among other factors. If TSA determines that an applicant does not pose a security threat, a credential may be supplied by an issuing entity. If it determines an applicant should be denied, the agency issues a preliminary determination of ineligibility letter to the applicant. The applicant may seek redress by appealing the determination or requesting a waiver.

TSA’s security threat assessments support over 30 credentialing programs in the maritime, surface, and aviation transportation segments. The largest programs include the Transportation Worker Identification Credential program for maritime workers, Hazardous Materials Endorsement program for commercially licensed drivers, the Aviation Worker program, and TSA Pre® for travelers at airport checkpoints. According to TIM program officials, these transportation programs are collectively estimated to have processed about 12.8 million enrollments by October 2017. Table 1 describes the largest transportation credentialing programs, by segment, and purpose of each.

TSA Established the TIM Program to Address Shortcomings with Security Threat Assessments and Credentialing Systems

TSA’s legacy IT systems that are currently used to help conduct its security threat assessment and credentialing functions are an aggregation of stove-piped solutions that were developed over a period of time to support individual transportation screening programs. These systems are duplicative and lack needed sophistication to effectively detect, for example, if an individual is attempting to gain access to multiple facilities across different transportation programs in an effort to find any successful entry point. Early detection of this type of threat is difficult and time consuming because many aspects of the current systems are not fully automated.

Additionally, we and the DHS Office of Inspector General (OIG) have previously reported numerous shortfalls with TSA’s security threat assessment and credentialing systems.

We reported in 2011 that the demand for security threat assessments is expected to continue to grow and the existing credentialing systems will not be able to accommodate this growing enrollment demand.

In July 2013, we reported on functional limitations and technical problems with TSA’s legacy credentialing systems that were to be addressed by the TIM system. These limitations included the inability to run reports to measure TSA response times to applicants, track adjudication of cases, and address case workload backlogs. We also reported on delays in processing new cases. We made recommendations to address these issues and DHS agreed with our recommendations. DHS has taken several actions to implement the recommendations, such as establishing a process for developing accurate workload projections and hiring additional adjudicators.

In June 2015, DHS’s OIG reported on issues with TSA’s lack of continuous vetting once a credential was issued, referred to as recurrent vetting. For example, the OIG reported on the need for recurrent vetting of aviation workers. Specifically, it found that TSA did not have effective controls in place for ensuring that aviation workers had not committed crimes that would disqualify them from having unescorted access to secure areas of airports, and that they had lawful immigration status and were authorized to work in the United States. Instead, TSA depended on the commercial airports and air carriers to verify criminal histories of workers who already hold credentials, and on the credential holders themselves to report disqualifying crimes to the airports where they worked. The DHS OIG recommended that TSA pilot the Federal Bureau of Investigation’s Rap Back program and take steps to institute recurrent vetting of criminal histories at all commercial airports. TSA concurred with the recommendation and stated that it planned to initiate a pilot Rap Back program to help ensure full implementation across all eligible TSA- regulated populations in the future.

In September 2016, DHS’s OIG reported that, although TSA required Transportation Worker Identification Credential cardholders to self- report to the administration and surrender their card when charged with a disqualifying offense, this self-reporting occurred only once between 2007 and 2016. The report also stated that TSA was testing two methods to implement recurrent vetting into its credentialing programs—the Federal Bureau of Investigation’s Rap Back program to check for criminal violations and the use of DHS’s Automated Biometric Identification System to check for both criminal and immigration violations. However, TSA’s plans did not include a method for determining the best approach, and the OIG reported that this would impede TSA’s ability to implement recurrent vetting successfully and efficiently. Accordingly, the OIG recommended that TSA establish measurable and comparable criteria to use in evaluating and selecting the best criminal and immigration recurrent vetting option, and TSA concurred with this recommendation.

Also, in September 2016, the DHS OIG reported that the background checks for the Transportation Worker Identification Credential program were not as reliable as they could be. For example, the OIG found that TSA did not have processes in place to ensure the proper separation of duties for adjudicators, who had the ability to assign, review, and perform quality assurance on the same case. The OIG also found missing supervisory review controls in the terrorism vetting process. Accordingly, the OIG recommended that TSA identify and implement additional internal controls and quality assurance procedures; TSA agreed with the recommendation. In response, TSA planned to make improvements to the TIM system to include an additional quality assurance component in which the system would automatically select cases for senior adjudicators to review and to incorporate into the overall reporting and monitoring activities.

The TIM system is intended to address the shortfalls identified in these prior reports by providing a modern and centralized end-to-end credentialing system. The system is also intended to provide counter- terrorism and trend analytic capabilities to help identify unusual activities (e.g., credential shopping and using multiple aliases) across the entire credentialing process and all transportation populations supported by TSA’s security threat assessments. In addition, the system is expected to enable automated recurrent vetting of individuals against criminal and immigration databases to ensure that a credential or endorsement is revoked if an individual commits a disqualifying act.

The planned credentialing process that is to be supported by the TIM system includes:

Registration and enrollment: Individuals seeking a credential or endorsement under one of the transportation programs supported by the system are expected to be able to apply for a security threat assessment at a Universal Enrollment Center or via the system’s online portal. The biographic and biometric information collected from the applicant is to be received and processed by the system.

Eligibility vetting and risk assessment: The system is to conduct automated vetting of the applicant’s information against criminal, immigration, and terrorism watchlists to determine the security risk associated with allowing access privileges based on the criteria for the credential or endorsement that the individual is seeking to obtain. If the results return a flag for a potentially disqualifying factor, the applicant’s case is to be sent for adjudication. TSA adjudicators are to use the system to review and adjudicate cases that did not pass automated vetting by comparing the applicant’s information to the criteria for the credential or endorsement that the individual is seeking to obtain. The adjudicators are to determine the applicant’s eligibility for the credential or endorsement, and approve or deny the individual’s application.

Issuance: When an applicant is approved through eligibility vetting or adjudication, the system is to notify the applicant of approval and provide instructions on how to receive the credential, which is to be activated by the system and supplied by the issuing entity. The applicant also is to be able to login to the online portal to view the status of the application.

Verification and use: Use of the credential in secured areas is to be verified, including determining that the credential is authentic, that the individual is the correct recipient of the credential, and that the credential’s status is valid (not revoked or expired).

Revocation and expiration: The system is expected to conduct subsequent automated recurrent vetting of individuals who previously had been approved against criminal, immigration, and terrorist databases on an ongoing basis. If, as a result of recurrent vetting or self-reporting, there is new information indicating that an applicant’s credential should be revoked, the system is to alert the adjudicators who are then to determine if revocation is needed. The system is to prompt credential expiration at the end of a specified period of time.

Redress or waiver: An applicant that is denied a credential is to be able to apply to TSA to either appeal the decision, to include providing documentation to prove that he/she is eligible, or request a waiver from having to meet the eligibility criteria.

Trend analytics: The system is to allow TSA’s Office of Intelligence and Analysis users to select from a standardized suite of analysis tools that would allow them to identify unusual activities across transportation populations. A key objective would be to identify through analysis those adversaries and terrorists who may attempt to hide behind multiple personas and aliases.

Figure 1 provides an overview of the intended future credentialing process which the TIM system is expected to support.

Agile Software Development

TIM program officials decided to adopt an Agile software development approach—a type of incremental development—which calls for the rapid delivery of software in small, short increments rather than in the typically long, sequential phases of a traditional waterfall software development approach. This decision is consistent with OMB’s guidance as specified in its IT Reform Plan, as well as the legislation commonly referred to as the Federal Information Technology Acquisition Reform Act. Agile emphasizes early and continuous software delivery, as well as using collaborative teams and measuring progress with working software. Figure 2 provides a depiction of software development using the Agile approach compared to a waterfall approach.

The Agile approach significantly differs in several ways from traditional waterfall software development. Table 2 highlights major differences between the Agile and waterfall software development approaches.

Additionally, Agile practices integrate planning continuously throughout the life-cycle. Although Agile requires some high-level, up front planning, in general, planning in Agile focuses on the near term of the next few software releases. Planning sessions are conducted to support each release, iteration, and every work day. For example, development teams have daily meetings, where the team members discuss what they did yesterday and what they plan to do that day. Frequent planning is aimed at ensuring the program is delivering the needed capabilities to the end users.

As we have previously reported, numerous frameworks are available to Agile practitioners to guide their Agile software development activities. Scrum is one common framework, which is widely used in the public and private sectors and its terminology is often used in Agile discussions. The following are key Scrum terminology and concepts:

Product owners represent the end user community and have the authority to set business priorities, make decisions, and accept completed work.

Scrum iterations (also called sprints) are where development teams build a piece of working software during a short, set period of time (e.g., 2 weeks). A collective set of sprints is bundled into a software release.

Sprint teams (or development teams) conduct the Agile software development and testing work. These teams collaborate with minimal management direction, often co-located in work rooms. They meet daily and post their task status visibly, such as on wall charts.

Scrum masters, similar to project managers, are responsible for removing impediments to the sprint teams’ ability to deliver the product goals and deliverables.

User stories convey the customers’ requirements at the smallest and most discrete unit of work that must be done to create working software. Each user story is assigned a level of effort, called story points, which is a relative unit of measure used to communicate complexity and progress between the business and development sides of the project.

To ensure that the product is usable at the end of every iteration, teams adhere to an agreed-upon definition of what constitutes acceptable, completed work.

Backlogs are lists of requirements, such as user stories, to be addressed by working software. If new requirements or defects are discovered, these can be stored in the backlog to be addressed in future iterations.

Velocity is a metric which is used to track the rate of work completed using the number of story points completed or expected to be completed in an iteration (i.e., sprint), or release. For example, if a team completed 100 story points during a 4-week iteration, the velocity for the team would be 100 story points every 4 weeks.

Another framework, referred to as the Scaled Agile Framework (SAFe), is a governance model for organizations to use to align and collaborate the product delivery for modest to large numbers of Agile software development teams. The framework is intended to be applied to several organizational levels, including the development team level, the program level, and the portfolio level. It is also intended to provide a scalable and flexible governance framework that defines roles, artifacts, and processes for Agile software development across all levels of an organization.

DHS has sought to establish Agile software development as the preferred method for acquiring and delivering IT capabilities. Specifically, in February 2016, the DHS Under Secretary for Management initiated an Agile software development pilot to improve the execution and oversight of the department’s IT acquisitions. The Under Secretary for Management selected five DHS programs that were in various stages of the acquisition life-cycle, including the TIM program, to be part of the pilot.

As part of this pilot initiative, DHS established integrated product teams designed to support each of the five programs in their efforts to adopt Agile practices. These teams were directed to focus on effectively planning and executing the pilot programs, as well as developing appropriate documentation to support program execution. According to the Under Secretary for Management, the department plans to use lessons learned from the pilots to develop and update policies and procedures for executing the pilot programs and future IT acquisitions. As of May 2017, department officials had not determined a completion date for the pilot.

Additionally, DHS established a headquarters-level Agile team intended to collaborate across the department on improvements to policy, governance, and acquisition guidance. This group is intended to support Agile delivery; codify and publicize process improvement artifacts generated by the program-level integrated product teams; and eliminate redundancies and conflicting guidance so that oversight groups speak with one voice, reducing time through the acquisition process.

DHS Oversight Framework

In addition to the use of Agile software development principles, the TIM program is subject to the department’s oversight framework. Specifically, the program is to adhere to DHS’s acquisition policy, including its systems engineering life-cycle framework, which is intended to support efficient and effective delivery of IT capabilities. The Under Secretary for Management serves as the decision authority for the program, and is responsible for overseeing adherence to DHS’s acquisition policies for the department’s largest acquisition programs (i.e., those with life-cycle cost estimates of $1 billion or more).

The Under Secretary for Management is supported by two offices within the department. The first of these offices—the Office of Program Accountability and Risk Management (PARM)—is responsible for DHS’s overall acquisition governance process. PARM is responsible for, among other things, periodically conducting program health assessments to evaluate acquisition programs, in terms of a program’s management, resources, planning and execution activities, requirements, cost and schedule, and how these factors are impacting a program’s ability to deliver a capability.

The other key supporting office—the DHS Chief Information Officer (CIO)—is responsible for, among other things, setting departmental IT policies, processes, and standards. The CIO is also responsible for ensuring that acquisitions comply with the department’s IT management processes, technical requirements, and the approved enterprise architecture. Within the Office of the Chief Information Officer (OCIO), the Enterprise Business Management Office is to ensure that the department’s IT investments align with its missions and objectives. As part of its responsibilities, this office periodically assesses investments to gauge how well they are performing through a review of program risk, human capital, cost and schedule, and requirements—referred to as the CIO’s program health assessment.

According to the CIO, the Chief Technology Officer, which is responsible for leading the development of IT and standards across the department, and for management of the Agile pilot initiative, offers guidance and assistance to programs to help improve their execution. In addition, the Director of the Office of Test and Evaluation is to provide oversight of components’ independent test and evaluation activities.

The DHS Acquisition Review Board is chaired by the Under Secretary for Management and is made up of many executive level members including the CIO, the Executive Director of the Office of PARM, and the Chief Procurement Officer. The board is to meet periodically to oversee programs’ business strategies, resources, management, accountability, and alignment to strategic initiatives. Additionally, the department has established executive steering committees, which generally are comprised of component and DHS executive-level members, such as the component CIO and Chief Financial Officer, as well as the DHS Chief Technology Officer and the Executive Director of the Office of PARM. The committees are to provide governance, oversight, and guidance to programs and their related projects and initiatives to help ensure successful development and operations.

Figure 3 shows the organizational structure of the key DHS organizations with IT acquisition management responsibilities.

TSA Organizations Involved with the TIM Program

The TIM program office resides within the Mission Operations component of TSA’s Office of Information Technology. The expected users of the TIM system come from multiple offices under the Office of Intelligence and Analysis, including the Security Threat Assessment Operations office, which is responsible for conducting the security threat assessments, and the Program Management office, which is responsible for managing TSA’s maritime, surface, and aviation credentialing programs. The TIM program’s Executive Steering Committee is chaired by the TSA CIO, who is the head of the Office of Information Technology, and the TSA Deputy Component Acquisition Executive, and meets quarterly. In addition, the TSA Operational Test Agent is to perform operational testing and evaluation of the TIM system’s operational effectiveness, interoperability, cybersecurity, and suitability. As previously mentioned, the DHS Director of the Office of Test and Evaluation is to provide oversight of these test and evaluation activities.

Figure 4 shows the key TSA organizations involved with the TIM program.

After Experiencing Significant Cost, Schedule, and Performance Issues with the Initial TIM System Deployment, TSA Implemented a New Strategy

The TIM program experienced significant cost, schedule, and performance issues during its initial implementation efforts. Specifically, in May 2014, TSA launched an initial version of a commercial-off-the-shelf (COTS) system for the maritime transportation segment of TIM that was to support the Transportation Worker Identification Credential program. However, as we previously reported, in September 2014, TSA reported to DHS that the program had breached its baseline because it had significant cost, schedule, and performance issues due to, among other things, the addition of newly created credentialing programs that were added to the program’s scope, such as TSA Pre® and Chemical Facility Anti-Terrorism Standards.

TIM program officials also reported in the breach remediation plan other issues that led to the breach, including different expectations between TSA officials and the contractor regarding the extent of reuse of system functionality among the different transportation segments. Specifically, TSA expected that it would be able to reuse more of the maritime functionality for the surface and aviation populations, while the contractor expected there to be less reuse.

In January 2015, the Acting Under Secretary for Management directed program officials to suspend all planning and development efforts related to the other two segments of the program—surface and aviation—until the issues with the maritime segment could be resolved. In August 2015, program officials prepared a revised life-cycle cost estimate which increased costs to approximately $1.34 billion (about $713 million more than the original 2011 estimate), and delayed full deployment of the TIM system (to include all three transportation segments) to fiscal year 2022 (7 years later than originally planned).

Also, in September 2015, the Director of the Office of Test and Evaluation issued a letter of assessment which concluded that initial operational testing of the COTS system for the maritime segment had determined that the system was not operationally effective and not operationally suitable. The Under Secretary for Management directed the DHS CIO to conduct a thorough review of the proposed plans for moving forward with the TIM program.

After conducting the review, the CIO did not support the program’s proposal. As a result, in November 2015, the Under Secretary for Management continued the suspension of all developmental efforts for the surface and aviation transportation segments, but authorized the program to continue resolving problems that were identified during initial operational testing for the COTS system being used by the maritime segment. The Under Secretary for Management also directed the CIO to form and lead an integrated product team with senior TSA representatives and the TIM program office to develop a new strategy for the program.

In March 2016, DHS and TSA officials completed a new strategy for delivering TIM capabilities. This strategy included the following changes: replace proprietary COTS applications with custom-developed applications using open source code; transition traditional, large development teams using a waterfall system development methodology to an Agile software development framework to enable rapid, incremental development and deployment; and migrate from a defined, fixed data center environment to a scalable Federal Risk and Authorization Management Program (FedRAMP) certified cloud computing environment.

Also, according to the new strategy, the move from the COTS product to an open source solution is to include replacing the COTS product that had already been deployed to the maritime segment with the open source solution. It is also to include replacing the legacy systems that support the credentialing programs from the other two transportation segments (surface and aviation) with the open source solution. TSA plans to incrementally transition the program from these legacy systems between fiscal years 2018 and 2021.

Additionally, the system is expected to interface with at least 19 other information systems, including the following key systems:

TSA’s Transportation Vetting System, which conducts initial and recurrent name-based matching against defined terrorist related data sets.

The Federal of Bureau of Investigation’s National Crime Information Center, which is an electronic clearinghouse of crime data.

DHS’s Automated Biometric Identification System, also referred to as IDENT, which is the central DHS-wide system for storage and processing of biometric and associated biographic information for national security, law enforcement, immigration and border management, intelligence, and other background investigative purposes.

TSA’s Secure Flight, which identifies individuals who may pose a threat to aviation or national security and designates them for enhanced screening or prohibition from boarding an aircraft, as appropriate.

The U.S. Citizenship and Immigration Service’s Systematic Alien Verification for Entitlements, which is the primary data source for government agencies to verify legal entry and presence in the United States of a non-U.S. citizen or naturalized U.S. citizen.

In April 2016, the Under Secretary for Management approved the TIM program’s new strategy and, in September 2016—almost 2 years after the program was initially suspended—the program was rebaselined to reflect the new strategy. As we previously reported, the estimated cost and schedule in the revised baseline was significantly different than the initial baseline. The revised baseline estimate was for about $1.27 billion (a $74 million decrease from the previous 2015 cost estimate and an overall increase of $639 million from the original 2011 estimate), with full deployment planned for 2021 (a 1-year acceleration from the previous 2015 schedule and an overall delay of 6 years from the original 2011 schedule). Table 3 shows the estimated costs and schedules reflected in the initial and revised estimates.

According to TIM officials, in the program’s first 8 years (between October 2008 and September 2016), TSA spent over $280 million to deploy the initial COTS solution to the maritime segment and address critical fixes in the solution (i.e., the solution that TSA determined it needs to replace). Also during 2016, TSA began transitioning to an Agile software development framework. In September 2016, TSA issued two task orders to a contractor to provide Agile software development services. The orders were issued to the same design and development contractor that had assisted with the initial deployment of the TIM COTS solution.

From October 2016 to June 2017, the program deployed four software releases using Agile software development practices. These releases were focused on, for example, deploying new functionality to the COTS system to enhance the criminal and immigration vetting data provided to adjudicators.

In December 2016, between the first and second Agile releases, the program suspended new development for 1 month while officials reconsidered the order in which they would deliver functionality. Also during this period, the program developed and deployed a smaller release which program officials refer to as a “half release.” According to program officials, this release did not produce any new capabilities and instead addressed operations and maintenance-related fixes to the deployed COTS system.

After development of the second software release, at the end of March 2017, the program was reviewed by DHS’s Acquisition Review Board. The purpose was to review the results of follow-on operational testing that was performed to determine whether the program had adequately addressed the prior system and usability issues and implementation of the program’s new strategy. The meeting was also intended to discuss the status of several action items from a prior review board meeting that occurred in September 2016, such as finalizing a test and evaluation master plan, conducting a cybersecurity threat assessment, updating the program’s mission needs statement and concept of operations, and establishing software development cost metrics. Implementation of the new strategy continues to be monitored by DHS and TSA oversight bodies.

The New Strategy for the TIM Program Has Addressed Selected Prior Challenges, but Concerns Remain

The new strategy for the TIM program addressed a number of major challenges that the program faced during earlier efforts to develop and deploy the system; nevertheless, key challenges remain. Specifically, of the seven major challenges that the program faced during its initial implementation of a COTS solution for the maritime segment, four challenges have been addressed related to (1) system performance and usability issues, (2) data migration issues, (3) information security testing, and (4) the inadequacy of the program’s previous hosting facility. However, the remaining three challenges regarding constraints with COTS product, significant addition of new transportation programs (e.g., TSA Pre®), and insufficient stakeholder coordination and communication have not been fully addressed.

Fixes Have Been Implemented to Address Critical System Performance and Usability Issues

According to DHS guidance, among other things, an operational test and evaluation examines systems for operational effectiveness. Specifically, it tests for the ability of a system to accomplish a mission when used by representative users in the expected environment.

The 2015 initial operational testing of the maritime segment (supporting the Transportation Worker Identification Credential program) found that the COTS system was extremely unreliable due to frequent critical failures, and had several system performance and usability issues that limited users’ ability to execute tasks in a timely and accurate manner. These issues included lags, freezes, the need for excessive refreshes, inadequate reporting and case management functionalities, as well as an interface that was not user-friendly. For example, the system was unable to produce accurate reports on case workload and status, so users expended significant effort creating spreadsheets to manually assign cases and manage their progress. The system was also unable to perform certain waiver functions in a timely and complete manner, which resulted in a significant backlog.

The program office has addressed the issues identified in the initial operational test report by first identifying a list of over 900 action items. According to TIM officials, they validated this list with the operational test agent and prioritized the action items with the product owners (i.e., end users) to identify which were the most critical to complete. For example, critical items included addressing issues with the waiver functions, assigning cases, and issuing credentials. The program implemented the critical fixes by developing seven software releases from September 2015 to October 2016. In January 2017, the TSA operational test agent reported that follow-on operational testing of the COTS system confirmed that the program had adequately addressed the prior system and usability issues. As a result, according to the test agent, the program’s previously deployed maritime segment of the system performed as intended.

Actions Have Been Taken to Better Account for the TIM Program’s Future Data Migration Efforts

According to leading practices, IT programs should identify potential problems before they occur. This allows programs to plan and execute activities to mitigate the risk of such problems having adverse impacts on the program.

When the TIM program transitioned maritime users from the legacy system to the COTS system, according to TSA’s breach remediation plan, program officials found that cleaning and properly migrating data was very difficult and time consuming because the legacy systems were old and the data mapping information was not readily evident. Program officials stated that the data migration efforts were also difficult because of the proprietary nature of the COTS product, which impacted the ability to effectively migrate data from legacy systems. The additional time needed for data migration resulted in higher than anticipated costs for the maritime transportation segment.

Program officials have taken action to better account for the TIM program’s future data migration efforts. Specifically, as part of the new strategy, the officials plan to defer legacy data migration until after system deployment efforts are complete to avoid disrupting deployment efforts. The strategy focuses on the program migrating only closed case data from the legacy systems to the new system. As such, adjudicators are to continue to complete and close any security threat assessment cases opened in the legacy system even after the new system is deployed, and the new system is to only handle newly opened security threat assessment cases. Once final disposition of the cases in the legacy system is complete, those cases would then be included in the closed case data migration effort, which is planned to occur at the end of development, around fiscal years 2020 to 2021.

In addition, the new strategy includes streamlining the data migration by using the open source solutions to help simplify the migration of data on transportation populations from the legacy systems. As a result of the new approach, the program should be better positioned to more effectively migrate data during future transitions between the legacy systems and new system.

Prior Information Security Weaknesses in the TIM System Have Been Addressed, and Deferred Cybersecurity Threat Testing Is Planned

According to DHS guidance, the operational test and evaluation also should examine the department’s systems for operational suitability, which is the degree to which a system is deployable and sustainable. The evaluation is to take into account factors such as reliability, maintainability, availability, and interoperability.

The 2015 initial operational testing of the COTS system found that it was not suitable because the system had significant information security weaknesses. Specifically, the system inappropriately provided users with greater access than was necessary to do their jobs, which undermined the security benefits of controlling what different users were able to do in the system based on their role. The COTS system also contained critical and high-risk system security vulnerabilities which could result in the compromise of sensitive system information, such as passwords, and could hinder TSA officials’ ability to effectively respond to incidents.

Program officials took actions to address the security weaknesses previously identified. For example, in response to the findings from the initial operational testing, between September 2015 and October 2016, they developed and released fixes to the significant security weaknesses. In April 2017, the results of the follow-on operational testing confirmed that the COTS system was free of critical or high-risk system security vulnerabilities and that it appropriately restricted access to the system by only allowing users to access areas of the system needed to support their specific business tasks.

In addition, critical steps to evaluate the system’s cybersecurity have been planned, but not yet completed. Specifically, testing for realistic cybersecurity threats which is used to help categorize the system’s risk- level in terms of confidentiality, integrity, and availability, was deferred until March 2018. Program officials decided to defer this test until new hosting environments for TIM are implemented, rather than testing TIM in an environment that will soon be retired. These environments are intended to enable the development, testing, and production of the system. However, implementation of those environments has been delayed until December 2017, and as a result, the cybersecurity vulnerability assessment has been deferred to March 2018. The identification of a time frame in which the program plans to conduct this important cybersecurity test is a step in the right direction, and avoiding additional delays will be important.

TSA Decided to Discontinue Use of a DHS-Provided Cloud, and Recently Took Actions to Address Delays in Implementing Interim Hosting Environments

According to OMB, a hosting facility or data center is to process or store data and must meet stringent availability requirements. Additionally, cloud computing can be used as a means for enabling on-demand access to shared and scalable pools of computing resources.

During the initial implementation of TIM, the system was hosted in a cloud that operated out of a DHS data center (referred to as DHS Data Center 1). However, the DHS cloud was higher in operations and maintenance costs than the program originally planned, which presented a challenge for the program.

To address this challenge, in 2016, TIM program officials decided to move the COTS system that was previously deployed (the maritime segment) out of the DHS cloud and set it up in a public cloud environment. They also planned to use the public cloud environment to develop, test, and operate the future TIM open-source based system. The officials planned to use a phased migration that consisted of first establishing hosting environments at two data centers—DHS Data Center 1 and TSA Colorado Springs Operations Center. The officials planned to use the data centers for the development, testing, and production of the future TIM open-source based system, and then eventually transition to a public or hybrid cloud once the system reaches full operational capability in fiscal year 2021. As part of this approach, officials planned to establish 10 development, testing, and production environments at these data centers from January to July 2017, so that TIM’s development teams did not have to compete for the same environments during Agile software development and testing efforts.

While the program experienced delays in setting up its production environment, officials recently took actions to address these delays. Specifically, the program was expected to have a new production environment available at the TSA Colorado Springs Operations Center by March 2017; however, it was delayed until May 2017. Additionally, while migration of the TIM system to the new hosting environments was planned to occur by September 2017, it has been delayed. These delays have contributed, in part, to delays in other aspects of the program, including the execution of the cybersecurity vulnerability assessment, as well as delays in the implementation of automated testing and deployment tools (discussed later in this report). In response to these delays, program officials recently established a revised schedule in May 2017 for setting up the new environments by December 2017.

Effectively executing against this updated schedule should help to keep the program on track with delivering these important environments and fully addressing the related challenge that the program experienced during its prior implementation efforts.

TSA Decided to Move the TIM System from COTS to Open Source, but Implementation Plans Continue to Significantly Change

According to leading practices and guidance, technology decisions should seek to enable services to scale easily and cost-effectively and to avoid vendor lock-in by, for example, using open source solutions. The benefits of using open source solutions can include improved software reliability and security through the identification and elimination of defects from continuous and broad peer review of publicly available source code that might otherwise go unrecognized by a more limited core development team; unrestricted ability to modify software source code; no reliance on a particular software vendor due to proprietary restrictions; reduced software licensing costs; and the ability to “test drive” the software with minimal costs and administrative delays in a rapid prototyping and experimentation environment.

Also, according to leading practices, IT programs should ensure that their plans include how they will transition from the current state to the final state of system operations. Such planning provides a mutual understanding to relevant stakeholders of how programs are to accomplish the transition.

According to TSA’s breach remediation plan, the TIM program’s use of a COTS solution led to several challenges. For example, program officials reported that the COTS product restricted their ability to make changes to the product to improve system usability and, as previously discussed, impacted the ability to effectively migrate data from legacy systems because of the proprietary COTS product. Program officials also reported that they were highly dependent on the COTS vendor to remediate compatibility issues and resolve problems, which required additional time. The plan also stated that the COTS product required a complex system architecture which prevented the program from implementing modern software development and testing tools. Finally, use of the COTS product resulted in higher software licensing costs.

The TIM program’s new strategy is intended to address these challenges by moving away from using a COTS product to a custom-developed open source solution. However, the program’s approach for developing and delivering this new solution has been in a continual state of fluctuation and implementation plans have not been defined. As such, this challenge has yet to be fully addressed. Specifically, In September 2016—after the 2-year pause in the program and completion of its extensive rebaselining effort—DHS and TSA officials decided that TSA would incrementally retire legacy systems as the transportation programs that use those systems are migrated to the open source solution; they also decided to eventually replace the COTS system that was previously deployed to support the maritime Transportation Worker Identification Credential program and migrate to the open source solution. This was to be completed using a staged approach between the migrations, and also by using two versions of the COTS system as well as the open source system. However, the program lacked a plan detailing how it was going to migrate from the current legacy state, to the interim environment (with the two versions of COTS plus an open source system), to the final state.

As previously mentioned, in December 2016, new development for the TIM system was paused once again to, among other things, further evaluate the transitioning approach that was agreed to 3 months prior. Four months later (in mid-March 2017), program officials decided to continue pursuing the approach that was agreed to in September. Subsequently, the high-level implementation schedule was revised to adjust for delays that this most recent replanning effort contributed to (other contributing factors for the delay are discussed later in this report). The revised schedule delayed deployment of the initial Pre® capabilities by 6 months and other key functionality up to 12 months.

Further adding to the fluctuation in the program, at the end of March 2017, the DHS Acquisition Review Board requested that the program’s implementation approach be revised to accelerate the delivery of the TIM program’s front-end interface for adjudication and redress functions. However, it is unclear how the acceleration of the development and implementation of these functions will impact the delivery of the other planned functionality, and what tradeoffs the program will need to make. Program officials were expected to develop an overview of the acceleration efforts associated with cost, schedule, risk, and impacts on the program and deliver it to PARM and the Office of the Chief Technology Officer in August 2017.

As a result, while it has been 8 months since the TIM program was rebaselined, the details of how the program will transition from its current state, to an interim state, then to the final state of full open source, have yet to be determined. This is contrary to leading practices that we have previously identified, which state that when pursuing an IT modernization effort, organizations should develop a plan for transitioning from the current to the target environment.

In response to our concerns, program officials stated that after they determine how they will adjust to incorporate the Acquisition Review Board’s recent acceleration request, they will determine the details of how the program will achieve the desired final state. However, until the program establishes and implements specific time frames for determining key implementation details, including how it will transition the program from its current state to an interim state and to the final state, the TIM program office, and TSA and DHS oversight bodies cannot be certain about how the program will ultimately deliver its complete open source solution.

New Transportation Programs Have Been Incorporated in TIM’s Rebaselined Schedule, but the Program Is Experiencing Significant Delays

According to leading practices, programs should manage changes to requirements as they evolve during the project. Programs should also ensure that planned schedules provide a realistic forecast for completion of activities, including providing reasonable slack (i.e., flexibility in the schedule).

After the TIM program was initiated in 2008, it experienced significant increases in scope, such as the addition of TSA Pre® and Chemical Facility Anti-Terrorism Standards populations in 2012, which required more functionality and considerably more processing demands than originally planned. The TIM program was challenged to accommodate the additional work needed to incorporate these new transportation populations and capabilities, and, in part, contributed to a significant breach in its original cost and schedule estimates.

To address the challenge, the TIM program incorporated the additional functionality and processing requirements into its cost and schedule rebaseline that was approved in September 2016. In addition, the program’s new strategy addressed the need to be adaptable to accommodate any new transportation populations and capabilities that could be added in the future by taking an enterprise-level approach to providing capabilities.

Nevertheless, while the TIM program incorporated TSA Pre® into its new plans, the implementation schedule for the program was very compressed and program officials did not establish a schedule that realistically forecasted when activities would be completed. Specifically, program officials planned to deploy initial TSA Pre® capabilities by May 2017 without any slack in the schedule. According to program officials, the reason for this approach, was because TSA Pre® was considered a high priority for migrating from its legacy system in order to accommodate an expected influx of applicants during the summer months. However, slack was not incorporated in the implementation schedule; therefore, when the program experienced schedule delays, it resulted in the program missing the May 2017 implementation deadline and being rescheduled to November 2017.

The 6-month delay in delivering initial Pre® capabilities was due to the delays discussed in the prior section associated with replanning the strategy for transitioning to the open source system, as well as delays in onboarding additional development team members and setting up new development and production environments. The delay in delivering Pre® capabilities is especially problematic because program officials have reported that the legacy system is at risk of exceeding its processing capacity.

Additionally, as previously mentioned, the program’s revised schedule shows the delivery dates for almost all (8 of 10) capabilities being significantly pushed back—with 2 capabilities being delayed up to 12 months. Moreover, not only were the implementation dates delayed for these efforts, the time to complete a number of these efforts was reduced by about 1 to 12 months—thus further exacerbating our concerns about unrealistic schedules. Without a schedule that realistically forecasts when activities will be completed, TIM program officials cannot ensure that they will meet the dates that they have committed to, such as when key capabilities for TSA Pre® are to be deployed.

Efforts to Improve Stakeholder Coordination and Communication for the TIM Program Have Begun, but Key Actions Have Not Been Implemented

According to leading practices, programs should coordinate and collaborate with relevant stakeholders (i.e., those that are affected by or in some way accountable for the outcome of the program, such as program or work group members, suppliers, and end users). Stakeholder coordination includes, for example, involving stakeholders in reviewing and committing to program plans, agreeing on revisions to the plans, and identifying risks. Programs should also identify the needs and expectations of stakeholders and translate them into end user requirements.

However, during prior implementation efforts with the COTS solution, the program experienced challenges with effectively coordinating and communicating with end-users. For example, according to program documentation, it had not adequately collaborated with end users in developing and implementing business requirements and conducting post-deployment user satisfaction assessments. This led to frustration among end users who felt inadequately informed and prepared for the new COTS system.

To address this challenge, the TIM program’s new strategy includes establishing a product owner role, which, as previously mentioned, is intended to represent the end user community and have the authority to set business priorities, make decisions, and accept completed work. The program’s adoption of the Agile software development approach has also significantly increased the frequency of the program’s engagement with stakeholders to define, test, and implement software releases.

In addition, program officials established an organizational change management strategy in October 2016 that is intended to, among other things, focus broadly on establishing overall communication processes for program stakeholders. This strategy identifies key steps such as, establishing a communication team and hiring a communication lead to oversee the development and execution of the communication action plans, establishing a communication working group, and serving as chair of the communication working group. This group is to be responsible for developing four communication action plans for key stakeholder groups (e.g., new transportation populations, existing transportation populations, and management). These particular steps were to be completed from November 2016 through January 2017.

However, while as of May 2017, the TIM program had implemented certain steps from the organizational change management strategy, such as establishing a communication team, the program has been delayed in implementing other steps. Specifically, the communication lead position was to be filled in November 2016. However, in March 2017 TIM program officials stated that the position had not yet been filled due to the federal hiring freeze. Additionally, because of the vacancy in the communication lead position, other key actions have been delayed, such as the development and execution of the communication action plans.

Program officials have not established new time frames for completing the remaining steps outlined in the organizational change management strategy. Until these time frames are established and effectively executed, program officials will have less assurance that there will be effective communication with stakeholders and customers to ensure that the program is meeting their needs.

The TIM Program Has Not Fully Implemented Leading Practices for Transitioning to Agile Software Development

As discussed previously, transitioning a program from waterfall development to Agile software development is a significant effort, and requires the implementation of fundamental practices to ensure that the transition is successful. According to leading guidance, an organization transitioning to Agile software development should establish critical practices to help ensure successful adoption of the Agile approach, such as obtaining full support from leadership to adopt Agile processes, enhancing Agile knowledge, ensuring product owners are engaged with the development teams and have clearly defined roles, establishing a clear product vision, prioritizing backlogs of requirements, and implementing automated tools to enable rapid system development and deployment.

While the TIM program has fully implemented the first two of these leading practices necessary to ensure the successful adoption of Agile, the remaining four practices have not been fully implemented. The gaps we have identified with the program’s implementation of Agile are concerning given that it did not follow key IT acquisition best practices when using its waterfall development approach during the program’s first 8 years and spent over $280 million on a system that TSA has determined it needs to replace.

The TIM Program Has Received and Maintained Support from TSA and DHS Leadership to Adopt Agile Practices

According to leading practices and guidance, an organization transitioning to Agile software development should get and maintain full support from the organization’s leadership to adopt Agile processes. Leadership support helps empower employees to continuously improve the use of Agile software development practices.

DHS and TSA leadership have approved the TIM program’s adoption of Agile software development, and continue to support the transition. For example, the DHS OCIO worked closely with TSA officials in 2015 and 2016 to develop the new strategy for the program which included moving away from a waterfall development approach to Agile software development. As previously mentioned, the Under Secretary for Management selected the TIM program to be part of the DHS Agile pilot initiative in February 2016 and approved the program’s new strategy in April 2016.

Moreover, the DHS Office of the Chief Technology Officer has continued to provide guidance and resources to the program since it adopted Agile. For example, TIM program officials stated that the DHS Chief Technology Officer added two of the office’s full-time and one part-time staff members to the TIM program. DHS and TSA officials stated that the Chief Technology Officer also provided an Agile coach to assist the TIM Program Manager about 3 days per week with establishing an Agile governance framework. Finally, DHS established an Agile Integrated Product Team that is co-chaired by PARM and the TIM Program Manager. The team meets bi-weekly to provide guidance on adopting Agile processes. As a result of the sustained leadership commitment, the program is better positioned to continuously improve its Agile practices.

Key TIM Program Staff Have Received Agile Training to Enhance Knowledge

According to leading practices and guidance, an organization transitioning to Agile software development should ensure that the entire program team receives Agile training. This allows organizations to achieve a faster shift away from the previous culture and processes and toward a more agile culture.

Toward this end, the TIM program requires its Agile contractor to ensure that development teams are trained and skilled in Agile methods, as well as in the specific Agile frameworks the program has adopted, which include the Scrum and SAFe frameworks. Additionally, the program provided initial Agile training for key program staff when it began transitioning to Agile software development. Specifically, the program provided a mandatory 2-day Agile workshop in October and December 2016 which covered basic Agile principles and the Scrum and SAFe frameworks. This training was provided to many key staff members, including contractor support staff, a contracting officer representative, and product owners.

Further, in December 2016, the program began providing training on the SAFe framework to its government employees. This training was tailored based on different roles, such as Agile practitioner, program manager or product owner, and scrum master. The training courses were provided to key staff members, including TIM program leadership, team leads, branch managers, and scrum masters. As a result of providing Agile training, the program’s staff should be able to more effectively adopt and apply Agile software development processes.

TIM Program Product Owners Frequently Engage with Development Teams, but Roles and Responsibilities Are Not Clearly Defined

According to leading practices and guidance, an organization transitioning to Agile software development should designate a product owner who represents the user community and establishes priorities based on business needs, approves user stories and their acceptance criteria, and decides whether completed work meets the acceptance criteria and can be considered done. The product owner should also maintain close collaboration with the development teams by, among other things, providing daily support to help clarify requirements and attending key Agile meetings, such as sprint- and release-level planning sessions and system demonstrations. Additionally, roles and responsibilities among relevant stakeholders, such as the product owner, should be clearly defined and documented by the organization that is transitioning to Agile software development, so that the stakeholders are aware of their responsibilities and given the authority to perform their roles.

The TIM program has two different groups of individuals that collectively share the responsibilities of product owner, and while these groups frequently engage with the development teams, program officials have not yet clearly defined the groups’ roles and responsibilities. Specifically, according to program officials, the first group consists of five product owners that represent end users and are collectively responsible for supporting all development teams, attending all Agile meetings, and prioritizing and approving planned and completed work. In addition, according to program officials, these five individuals are also responsible for approving user stories associated with new system functionality.

The other group is referred to as the solutions team, which includes, for example, the TIM Chief Architect and Chief Engineer. According to program officials, the technical work (which is to help enable the system functionality, such as ensuring network connectivity and proper software licenses) is approved by the solutions team.

Nevertheless, while program officials told us about these high-level roles and responsibilities, the program’s documentation does not clearly define them among the five product owners and the solutions team. Moreover, program officials have not defined the rules of engagement for these product owners, such as how competing priorities among different product owners should be handled.

According to program officials, the lack of clearly defined roles and responsibilities has not been a problem for the program because the product owners and the solutions team regularly communicate and coordinate with each other, and thus far, have been in agreement on the priorities for the program. However, the program recently scaled up the amount of work being conducted simultaneously, which adds to the volume of the decisions that need to be made and the coordination that has to occur among the five product owners and solutions team. Thus, even if the program has not yet experienced issues with coordination, without more clarity in the roles and responsibilities among the groups that are responsible for prioritizing and accepting work, the program risks facing challenges in establishing priorities, approving user stories, and deciding whether completed work meets the acceptance criteria.

The TIM Program Established a Vision, but It Does Not Always Align to the Requirements; Recent Corrective Actions Should Yield Improvements

According to leading practices and guidance, a program transitioning to Agile software development should have a clearly defined vision. This can be in the form of a product roadmap, to guide the development of the product and to help inform the planning and requirements development of Agile software development releases.

Consistent with leading practices, TSA established a vision for the TIM program. This vision is articulated in multiple documents—including the Mission Needs Statement, Concept of Operations, and Operational Requirements Document. Officials also use a strategic roadmap to articulate the program’s vision, which specifies the high-level system capabilities that are to be deployed over the life-cycle of the program through 2021.

However, the program’s vision has not always informed the planning of requirements for the software releases, as intended by leading practices. Specifically, the capabilities outlined in the program vision documents, such as the strategic roadmap, do not consistently map to program requirements. While 5 of the 10 capabilities in the strategic roadmap align to the high-level and large scope requirements, referred to as epics, the other half of the capabilities do not clearly align to the epics. For example, the adjudication and redress capabilities that are in the strategic roadmap do not align to any epic. In addition, the capability for public-facing portals does not clearly track to any epic.

TIM officials recognized the alignment issues, and in August 2017, stated that they are in the process of establishing alignment from the program’s vision down to the lowest level of requirements, by refining the program’s vision and requirements. Officials also stated that they expected this effort to be completed by 2018. Effective execution of this effort should help ensure the program’s vision is informing requirements planning.

Requirements for the TIM System Have Not Been Fully Prioritized

According to leading practices and guidance, a program transitioning to Agile software development should have a prioritized list of the requirements that are to be delivered—referred to as the backlog. This backlog should be maintained so that the program can ensure it is always working on the highest priority requirements that will deliver the most value to the users. In addition, according to TIM Agile management documentation and program officials, the program’s backlog of features (i.e., mid-sized requirements) is expected to represent the features that are to be delivered over the next several software releases. These features are to be assigned priority levels to help determine which should be selected for development when planning the next release.

According to TIM Agile management documentation, the TIM program is expected to manage a backlog for each software release, which is to identify the features and their derived user stories (i.e., the smallest and most detailed requirements) that are to be delivered in a specific release. The documentation also indicates that each feature and user story is to be assigned priority levels to determine which should be included in the development of the next release and associated sprint. Figure 5 illustrates the intended prioritization in the features, releases, and user stories backlogs.

However, as of July 2017, the program’s backlogs did not contain specific prioritization levels for each of the features and user stories, as called for in DHS guidance. According to program officials, instead of assigning specific prioritization levels, they had more generally identified which features should be developed within the near-term (e.g., in the next several Agile releases). Program officials recognized that they still needed to prioritize their backlogs by assigning priority levels to all features and user stories, but they did not have a time frame for completing this effort.

Without ensuring full prioritization of current and future features and user stories, the program is at risk of delivering functionality that is not aligned with the highest needs of those that are responsible for conducting security threat assessments to protect the nation’s critical transportation infrastructure.

The TIM Program Has Been Delayed in Implementing Many of the Planned Automated System Development and Deployment Tools

According to leading practices and guidance, automating system development and deployment work and avoiding manual work is especially important for Agile programs, as it enhances the ability for rapid development and delivery of high quality software. Specifically, a program transitioning to Agile software development should use an automated tool for managing Agile activities, such as maintaining the product backlog and tracking the status of completed work. The program should also establish automated testing and deployment capabilities to improve the quality of the system. For example, according a DHS’s Agile development instruction manual, the vast majority of software defects are discovered during system integration testing, and—if automated—this testing can be run multiple times on a sprint or release in order to identify more defects sooner. In addition, automated tools can enable more efficient processes for frequently integrating computer code that is developed by different team members (e.g., hourly or daily), in order to quickly detect any code integration errors. Automation of testing can also help decrease the risk of introducing security flaws due to human error.

However, program officials deferred implementation of an automated Agile program management tool and many other testing and deployment tools. Specifically, while the program had been using Agile software development practices since October 2016, the program has not used an automated management tool for tracking the status of completed work for its first three Agile software releases. Instead the program has used spreadsheets that require TIM program officials to manually populate and track large amounts of program status information.

Program officials had planned to implement an automated management tool by October 2016, but did not do so until the end of April 2017. According to the officials, the delay occurred because they were in the process of tailoring the SAFe governance framework and the management tool needed to be customized to reflect the tailored approach.

Regarding tools for testing and deployment, as of May 2017, the program was only using 4 of the16 automated tools that program officials planned to use. These included tools that enable the management of software code development, defect tracking, and components of automated functional testing. However, the remaining 12 testing and deployment tools had not yet been implemented. These include, among others, tools that enable the automated building of software code, frequent merging of an individual piece of software code with the main code repository so that new changes are tested continuously (referred to as continuous integration), small automated tests to verify that each individual unit of code written by the developer works as intended, and installation of application patches to protect against known vulnerabilities. TIM program officials stated that these testing and deployment tools are not expected to be implemented until the new development, testing, and production environments are set up. However, as previously mentioned, the program has experienced challenges in implementing these environments.

As a result, the program’s use of manual processes have been time consuming, impeded visibility into the process, and hindered software testing. In addition, without automated tools, program performance metrics were being manually calculated and this increases the risk for incomplete and inaccurate data. While the automated Agile management tool has just been implemented, until the remainder of the automated Agile testing and deployment tools are implemented, the program is likely to continue to operate at reduced efficiency levels, and be limited in its ability to ensure product quality.

TSA and DHS Have Not Fully Implemented Most Key Practices for Overseeing the TIM Program’s Cost, Schedule, and Performance

According to leading practices, to ensure effective program oversight of cost, schedule, and performance, organizations should: ensure that corrective actions are identified and tracked until the desired outcomes are achieved, document relevant governance and oversight policies and monitor program performance and progress, and rely on complete and accurate data to review performance against expectations.

While TSA fully implemented the first practice, the remaining three practices were not fully implemented by DHS and TSA. As a result, the effectiveness with which the governance bodies oversee and monitor the program has been limited.

TSA Established a Process for Ensuring Corrective Actions Are Identified and Tracked for the TIM Program

According to leading practices, effective program oversight includes ensuring that corrective actions are identified and tracked until the desired outcomes are achieved. In this regard, governance bodies should collect and analyze data on program risks and issues and determine corrective actions to address them and track them to completion.

TSA has established a process for ensuring that corrective actions are identified and tracked. Specifically, the program has a process for identifying corrective actions and monitoring the status of these actions in its weekly program status reviews. The program also uses an automated tool to track and maintain a complete list of all actions that have been identified. As of February 2017, the list contained 89 actions and included the status of the actions—83 of which had been tracked to completion. As a result of the program having a process that can identify and track corrective actions, it is better positioned to address significant deviations in cost, schedule, and performance parameters.

TSA and DHS Have Documented Selected Oversight and Governance Processes for the TIM Program, but Other Key Processes Are Underdeveloped

According to leading practices, effective program oversight includes the use of documented policies and procedures for program governance and oversight, such as reporting and control processes. These processes may include, among others, requiring programs to report on the status and progress of activities; expected or incurred program resource requirements; known risks, risk response plans, and escalation criteria; and benefits realized. Oversight and governance documentation may also include threshold criteria to use when analyzing performance, and the conditions under which a program or project would be terminated.

TSA and DHS have documented selected policies and procedures for governance and oversight of the TIM program. Specifically, DHS documented procedures for its Acquisition Review Board and its Executive Steering Committee for the TIM program on how these governance bodies are to review the cost, schedule, and performance of the program. For example, according to the Committee’s charter, it is responsible for assessing the health of the program and identifying major issues and risks, utilizing a standard reporting format at oversight meetings.

TSA has also documented processes for the program’s Agile milestone reviews, such as conducting workshops at the end of the release cycle to perform a system demonstration, review qualitative metrics, and promote continuous quality improvement. TSA also developed a risk management plan tailored for the Agile approach to guide TIM staff members in identifying, managing, and mitigating risks and issues impacting cost, schedule, and performance of the program. The agency also developed a test and evaluation master plan that outlines how it and DHS will conduct and oversee testing and evaluation of the program’s capabilities under the new Agile software development approach.

However, TSA and DHS have not developed or finalized other key oversight and governance documents. Specifically, three oversight and governance policies have not been finalized and/or appropriately updated: the TIM program’s tailoring plan for SAFe, a DHS-level oversight policy for Agile programs, and DHS Office of the Chief Technology Officer’s guidance for Agile programs to use for collecting and reporting on performance metrics.

The TIM program has not updated its Systems Engineering Life Cycle Tailoring Plan (which outlines the Agile governance process and all milestone reviews that are required for planning and deploying Agile releases), to reflect changes in the way officials have reported using the SAFe governance framework. As a result, there are inconsistencies in the governance documentation. For example, the Systems Engineering Life Cycle Tailoring Plan describes four levels of governance—portfolio, value stream, program, and team—while program officials have reported omitting the value stream level from the governance framework. According to TSA officials in May 2017, they planned to update the Systems Engineering Life Cycle Tailoring Plan to reflect the revised governance framework, but they did not have a specific time frame for completing the revision. Until the TIM program fully updates its Systems Engineering Life Cycle Tailoring Plan to reflect the revised governance framework, the program lacks a clearly documented and repeatable governance process to effectively oversee the program.

DHS officials stated that they plan to conduct biannual oversight reviews of the five Agile pilot programs (including TIM), instead of the annual reviews that are typically conducted for traditional waterfall development programs. According to the officials, the purpose of moving to biannual reviews is to better ensure cost, schedule, and performance remain on track for these Agile programs. However, officials in the Office of the Chief Technology Officer stated that DHS- level Agile governance and oversight policies and procedures have not been revised to reflect this new oversight approach because consensus among DHS leadership on related changes needs to be established before this new oversight approach can be documented in the department’s guidance. As of May 2017, officials had not specified a time frame for reaching such consensus. Until DHS leadership reaches consensus on needed oversight and governance changes, and then documents and implements associated changes, the program continues to plan as though it is undergoing annual oversight reviews, versus biannual reviews.

As of early May 2017, officials in the Office of the Chief Technology Officer were also in the process of drafting guidance for Agile programs to use for collecting and reporting on performance metrics, but did not know when this guidance will be finalized. According to TSA officials, in the absence of complete Agile guidance, the TIM program receives support from DHS’s Agile team supporting the pilot initiative, which, as specified in the team’s charter, is intended to help the program (as well as the other four pilot programs) facilitate Agile software development. However, this team is not intended to perform oversight functions to ensure that the program is meeting cost, schedule, and performance targets. Thus, until the Office of the Chief Technology Officer completes guidance for Agile programs to use for collecting and reporting on performance metrics, TIM program officials may not report the most informative Agile performance metrics to oversight entities.

TSA and DHS Consistently Conduct Program Performance Reviews, but Lack Insights from Key Performance Metrics

According to leading practices, effective program oversight includes monitoring program performance and progress by comparing actual cost, schedule, and performance data with estimates in the plan and identifying significant deviations from established targets or thresholds for acceptable performance levels. Program reviews are to be conducted at predetermined checkpoints or milestones in order to determine progress by measuring programs against cost, schedule, and performance metrics. In addition, Agile programs should be measured on, among other things, velocity (i.e., number of story points completed per sprint or release), development progression (e.g., the number of features and user stories planned and accepted), product quality (e.g., number of defects and unit test coverage), and user satisfaction.

The TIM program management office conducts frequent and regular performance reviews and focuses on several important Agile release- level metrics. Specifically, program management officials monitor TIM’s performance and progress during weekly program status review meetings and in periodic Agile reviews that are conducted at the end of each release. These reviews also include officials from the development teams and program stakeholders. The reviews focus on, among other things, velocity, progress, and product quality. They also include the status of key activities and risks impacting cost, schedule, and performance.

Nevertheless, while the program management office uses performance metrics, the program has not established thresholds or targets for acceptable performance levels for these metrics. For example, program status reports showed that about 47 percent of the work that was planned to be completed in the first Agile release was accepted by the product owners. While the program appears to have been improving in this metric—74 percent was accepted in the second Agile release and 94 percent in the third Agile release—program officials have not established the thresholds or targets to determine the acceptable level of performance.

Program officials stated that they considered the performance in the first Agile release to be low, but they have not yet established targets or thresholds. According to program officials, they planned to establish targets based on the capacity of work that development teams are expected to complete in a release, which can be better predicted as the teams spend more time together. However, the program has since developed three releases and continues to lack performance thresholds and targets. Until program officials establish performance thresholds or targets, oversight bodies may lack important information to ensure the program is meeting acceptable performance levels.

In addition, the program management office’s performance reviews have included limited information on program cost. According to TIM officials, the program manager holds weekly meetings with the contract, finance, and budget groups to review costs associated with TIM’s contracts. However, management does not review or produce reports on overall life- cycle cost performance for the program or Agile software development cost performance. Program officials said they have not yet determined how best to measure cost performance in an Agile software development environment. In September 2016, the Under Secretary for Management instructed the program to collaborate with DHS’s Cost Analysis Division and the headquarters-level Agile integrated product team to establish agreed-upon software development cost metrics as well as a method for collecting and reporting on those metrics by the end of the March 2017. However, as of May 2017, this effort was still in progress. Until the TIM program begins collecting and reporting on Agile-related cost, oversight bodies will have limited information by which to monitor TIM costs.

Department-level oversight bodies have focused on reviewing certain program life-cycle metrics for the TIM program. Specifically, the DHS Acquisition Review Board conducts periodic reviews of the program to monitor the program’s performance and hold the program accountable.

Since the program was rebaselined in September 2016 and transitioned to Agile software development, the Acquisition Review Board has conducted one review. In addition, the Executive Steering Committee, which is chaired by the TSA CIO and Deputy Component Acquisition Executive, and includes representatives from the DHS Chief Technology Officer and PARM, reviews the program quarterly. As of July 2017, the Executive Steering Committee had conducted three reviews of the TIM program since implementing its new development approach. These oversight bodies reviewed, for example, performance information such as comparisons of the dates that milestones were actually achieved, against the planned schedule, and the burnup charts for the program (i.e., graphical representations of accumulated story points planned and completed per release).

However, the Acquisition Review Board and the Executive Steering Committee have not been measuring the program against the rebaselined life-cycle costs, or important Agile release-level metrics, which are essential for providing early indicators of issues with the program. For example, these oversight bodies did not review the program’s velocity, number of features/user stories planned and accepted, product quality, or Agile software development cost metrics.

In addition, while we have previously reported that there was overlap in the DHS OCIO’s and the PARM office’s assessments of certain IT programs, neither of these offices assessed the TIM program’s progress against key Agile performance metrics or cost performance. Specifically, the DHS OCIO and the PARM office conducted periodic (monthly or quarterly health assessments) of the program that included, among other things, schedule and system performance indicators for the entire life- cycle of the program (similar to what is used to review traditional waterfall programs). While these metrics are useful for understanding the program’s progress against the full schedule (60 months to full operational capability, or 30 Agile releases), they do not offer insight into the progress of individual Agile releases, which are deploying high-priority capabilities for the TIM program every 2 months. For example, as of April 2017, these two oversight bodies did not include Agile performance metrics which would have offered important insights into the progress of individual releases, such as velocity, progress metrics, quality metrics, post-deployment user satisfaction, or Agile software development costs. Thus, until DHS-level oversight bodies review key Agile performance and cost metrics and use them to inform management oversight decisions, the oversight bodies will be limited in their ability to obtain early indicators of any issues with the program, and to call for course correction, if needed.

Recently, the TIM program also began measuring user satisfaction. Specifically, in April 2017, the DHS Acting Under Secretary for Management directed TSA’s Operational Test Agent to implement a continuous evaluation dashboard based on the results from the program’s third Agile release by the end of June 2017. This dashboard was to measure, among other things, post-deployment user satisfaction. TSA subsequently implemented the continuous evaluation dashboard in June 2017.

Table 4 summarizes the extent to which performance metrics are reviewed by various oversight bodies.

TSA and DHS Do Not Always Rely on Complete and Accurate TIM Performance Data

According to leading practices, effective program oversight includes relying on complete and accurate data to review program performance against stated expectations. Complete and accurate data allow oversight bodies to have transparency into the performance of programs and helps them identify when course correction is needed.

However, TIM’s reported performance data were not always complete and accurate. Specifically, when reporting on the velocity (i.e., total number of story points completed per sprint and/or release across the development teams) of TIM’s first release after it was deployed, program officials inconsistently reported velocity among the program’s performance reports, thus calling into question the accuracy and completeness of the information. Since the data were being reported on a completed release, the velocity should have been reported as one consistent number that did not change. According to program officials, the reason for inconsistent reporting was that, despite best practices, the program’s methodology for measuring velocity was not consistent and was calculated differently each time. For example, table 5 shows three different numbers that were to represent the collective velocity across the development teams, and that officials reported to program management after the deployment of the first software release.

While there was less variation in the velocity data reported after the second software release was deployed, discrepancies were still present. For example, table 6 shows the different numbers that officials reported to TIM program management after the deployment of the second software release.

Program officials stated that the reason for the inconsistencies in reported velocity data was that during the first release they were still in the process of adapting Agile and were working to determine how best to calculate velocity. However, as shown in table 6 inconsistent data continued to occur beyond that first release.

These inconsistencies in reported data call into question the completeness and accuracy of the velocity numbers reported, and the potential impact on oversight bodies’ ability to hold the program accountable. For example, velocity is most useful when tracked over time to ensure consistent performance and for forecasting how quickly development teams can work through the items in a backlog. However, without a complete and accurate velocity number from each release, it is difficult for oversight bodies to ensure the program is producing work at an acceptable pace to enable the program to meet its cost, schedule, and performance targets.

In addition, the program had been reporting inaccurate unit test coverage data using a manual measurement approach. Specifically, from December 2016 to March 2017, program officials were reporting that, for each release, they tested every line of code, based on a manual estimate (i.e., 100 percent). However, testing each line of code manually is unrealistic because with manual tests, it is difficult to determine which function, line of code, or logic decision is executed, and which is not. As such, program officials were reporting that they were testing every line of code, even though they were unable to confirm that they were actually doing so, thus calling into question the reliability and accuracy of the data reported.

In response to our concerns, program officials acknowledged that they could not confirm whether they had tested every line of code. Accordingly, program officials stopped estimating this metric manually and stated that they planned to begin measuring unit test coverage again once lines of code could be tracked using automated tools. As previously discussed, program officials stated that the testing and deployment tools are not expected to be implemented until the new development, testing, and production environments are set up. However, until the program has complete and accurate unit test code coverage data, program officials will not know if portions of its code are going untested, which could lead to undetected issues and impact the quality of the product.

Conclusion

TSA’s TIM program has taken notable steps to address several of the major issues it faced during prior system development and deployment efforts, such as implementing system fixes to address critical performance and usability issues found in the maritime segment. Nonetheless, a number of significant challenges have not been fully addressed. In particular, until the TIM program establishes specific time frames for determining key implementation details, ensures its schedule provides planned completion dates based on realistic estimates, and establishes new time frames for implementing the actions identified in the strategy, it is at significant risk of repeating past mistakes and experiencing the same pitfalls as it did during its initial implementation attempts. An indication of concern is that the program is currently experiencing a delay of at least 6 months in the rebaselined schedule for delivering TSA Pre® capabilities.

While the program has also taken certain steps to successfully make the transition from a waterfall development approach to Agile software development—a substantial and complex effort—TIM has not defined key roles and responsibilities, prioritized features and user stories, or implemented automated capabilities that are essential to ensuring effective adoption of Agile. The gaps we identified with the program’s implementation of Agile are concerning given that it did not follow key IT acquisition best practices when using its waterfall development approach, in which the program spent approximately 8 years and over $280 million on a system that TSA has determined it needs to replace. While selected corrective actions have been taken, until the TIM program is implemented in accordance with leading practices, the program will be putting at risk its ability to deliver a quality system that strengthens and enhances the sophistication of TSA’s security threat assessment and credentialing programs.

In addition, while TSA and DHS have implemented certain practices for overseeing and governing the program, the lack of other practices has impeded their oversight effectiveness, including the lack of thresholds or targets for acceptable performance levels, the lack of reporting on Agile- related cost metrics, and inconsistent measuring and reporting of program velocity and unit test coverage for software releases. These gaps limit the ability of DHS oversight bodies to obtain early indicators of any issues with the program, and to call for course corrections, if needed.

Further, until DHS leadership reaches consensus on needed oversight and governance changes related to Agile programs, and then documents and implements associated changes to align oversight reviews with the timing of Agile software releases, the department will not be well positioned to hold the program accountable. Moreover, until the Office of the Chief Technology Officer completes guidance for Agile programs to use for collecting and reporting on performance metrics, and DHS-level oversight bodies require the TIM program to report on key Agile performance and cost metrics and use them to inform management oversight decisions, the department will also be limited in its ability to hold the TIM program accountable and ensure that it is meeting its cost, schedule, and performance targets.

Recommendations for Executive Action

We are making the following 14 recommendations to DHS: The TSA Administrator should ensure that the TIM program management office establishes and implements specific time frames for determining key strategic implementation details, including how the program will transition from the current state to the final TIM state. (Recommendation 1)

The TSA Administrator should ensure that the TIM program management office establishes a schedule that provides planned completion dates based on realistic estimates of how long it will take to deliver capabilities. (Recommendation 2)

The TSA Administrator should ensure that the TIM program management office establishes new time frames for implementing the actions identified in the organizational change management strategy and effectively executes against these time frames. (Recommendation 3)

The TSA Administrator should ensure that the TIM program management office defines and documents the roles and responsibilities among product owners, the solution team, and any other relevant stakeholders for prioritizing and approving Agile software development work. (Recommendation 4)

The TSA Administrator should ensure that the TIM program management office establishes specific prioritization levels for current and future features and user stories. (Recommendation 5)

The TSA Administrator should ensure that the TIM program management office implements automated Agile management testing and deployment tools, as soon as possible. (Recommendation 6)

The TSA Administrator should ensure that the TIM program management office updates the Systems Engineering Life Cycle Tailoring Plan to reflect the current governance framework and milestone review processes. (Recommendation 7)

The TSA Administrator should ensure that the TIM program management office establishes thresholds or targets for acceptable performance-levels. (Recommendation 8)

The TSA Administrator should ensure that the TIM program management office begins collecting and reporting on Agile-related cost metrics. (Recommendation 9)

The TSA Administrator should ensure that the TIM program management office ensures that program velocity is measured and reported consistently. (Recommendation 10)

The TSA Administrator should ensure that the TIM program management office ensures that unit test coverage for software releases is measured and reported accurately. (Recommendation 11)

The Secretary of Homeland Security should direct the Under Secretary for Management to ensure that appropriate DHS leadership reaches consensus on needed oversight and governance changes related to the frequency of reviewing Agile programs, and then documents and implements associated changes. (Recommendation 12)

The Secretary of Homeland Security should direct the Under Secretary for Management to ensure that the Office of the Chief Technology Officer completes guidance for Agile programs to use for collecting and reporting on performance metrics. (Recommendation 13)

The Secretary of Homeland Security should direct the Under Secretary for Management to ensure that DHS-level oversight bodies review key Agile performance and cost metrics for the TIM program and use them to inform management oversight decisions. (Recommendation 14)

Agency Comments and Our Evaluation

DHS provided written comments on a draft of this report, which are reprinted in appendix II. In its comments, the department concurred with all 14 of our recommendations and described actions it has planned or taken to address them. For example, with regard to recommendation 6, which calls for DHS to implement automated Agile management testing and deployment tools, the department stated that TSA plans to implement such tools by June 30, 2018. Additionally, for recommendation 14, the department stated that DHS intends to ensure that oversight bodies review key Agile performance and cost metrics for the TIM program by June 30, 2018. If implemented effectively, these actions should address the weaknesses we identified.

The department also described recent actions that it and TSA had taken to address three of the recommendations, and requested that we consider these recommendations resolved. Specifically, in response to recommendation 9, calling for TSA to ensure that the TIM program management office begins collecting and reporting on Agile-related cost metrics, the department stated that the program is now reporting these metrics on a monthly basis. In response to recommendation 10, calling for TSA to ensure that the program’s velocity is measured and reported consistently, the department stated that velocity is now being reported consistently and in accordance with DHS guidelines. Further, in response to recommendation 13, which calls for DHS to complete guidance for Agile programs to use for collecting and reporting on performance metrics, the department stated that the guidance had recently been published and provided to us. However, to date, we have received only draft versions of the guidance. We will work with the department to obtain finalized documentation related to the three recommendations, to determine if the recent actions fully address the recommendations.

In addition to the aforementioned comments, we received technical comments from DHS and TSA officials, which we incorporated, as appropriate.

We are sending copies of this report to the Secretary of Homeland Security and interested congressional committees. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-4456 or harriscc@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III.

Appendix I: Objectives, Scope, and Methodology

Our objectives were to (1) describe the Transportation Security Administration’s (TSA) past implementation efforts for the Technology Infrastructure Modernization (TIM) program and its new implementation strategy; (2) determine the extent to which TSA’s new strategy for the program addresses the challenges encountered during earlier implementation attempts; (3) determine the extent to which TSA has implemented selected key practices for transitioning to an Agile software development framework for the program; and (4) determine the extent to which the TSA and the Department of Homeland Security (DHS) are effectively overseeing and governing the TIM program to ensure that it is meeting cost, schedule, and performance requirements.

To address our first objective, we reviewed program documentation, such as initial and current acquisition program baselines, initial and current life- cycle cost estimates, acquisition decision memorandums, and program plans documenting a new strategy for implementing the program. We used the information in this documentation to summarize the program’s earlier attempts to implement TIM capabilities and its new implementation strategy for delivering the program, including estimated costs, schedule, and key decisions made. We also interviewed TSA officials, including the TIM Director and Deputy Director, on the status of TIM program office efforts.

To determine the extent to which the TIM program’s new strategy addresses the challenges encountered during earlier implementation attempts, we reviewed documentation on the challenges the TIM program faced when it breached cost and schedule thresholds and experienced system performance issues, such as those described in initial operational test reports, the breach remediation plan, and the results of a technical evaluation of program challenges. We synthesized the information in these documents to identify a consolidated list of key challenges the program had faced. We did not include challenges that were already being evaluated as part of other objectives, such as the use of the waterfall software development approach. We then reviewed documentation on the program’s new strategy, such as plans documenting the new strategy, follow-on operational test reports, program schedules, program status reports, and identified risks. We assessed the extent to which the new strategy outlined in these documents addressed the prior challenges by comparing them against criteria identified in leading practices and guidance, such as DHS’s Systems Engineering Lifecycle Guide and the Software Engineering Institute’s Capability Maturity Model® Integration for Development. In addition, we conducted a site visit at the TSA Adjudication Center in Reston, Virginia. During this site visit, we observed demonstrations of the current commercial-off-the- shelf system and legacy systems for TSA Pre® and Aviation Workers, and we interviewed adjudicators and supervisors on current security threat assessment processes and limitations. Further, we interviewed TSA officials, including the TIM Director and Deputy Director, on the program office’s efforts to address prior challenges.

To determine the extent to which the program has implemented selected key practices for transitioning to an Agile software development framework, we identified leading practices and guidance outlined in the following sources:

GAO, Software Development: Effective Practices and Federal Challenges in Applying Agile Methods

Software Engineering Institute, Agile Readiness and Fit

TechFAR handbook

TSA Agile Scrum guidance

CMMI® for Development, version 1.3

Software Engineering Institute, Agile Metrics After reviewing the sources listed, in consultation with our internal expert, we grouped practices that were identified as being critical to establish when transitioning to an Agile software development framework, and selected the practices that were most relevant based on the status of the program’s transition and we discussed the practice areas with TSA officials. The practices included: full support from leadership to adopt Agile processes, enhancing Agile knowledge, ensuring product owners are engaged with the development teams and have clearly defined roles, establishing a clear product vision, prioritized backlogs of requirements, and implementing automated tools to enable rapid system development and deployment.

We reviewed program management documentation against these practices, such as Agile training records, Agile contracts, program roadmaps, backlogs, test plans, Agile release artifacts, program status reports, and identified risks. Additionally, we observed Agile release and sprint development activities at TSA facilities in Annapolis Junction, Maryland, and at a contractor’s facilities in Beltsville, Maryland, and we observed a demonstration of how user stories map from high-level capabilities and tracked through development and testing.

We also interviewed TSA officials, including the TIM Director and Deputy Director and the five TIM product owners, on their efforts to transition the program to an Agile software development framework. Further, we interviewed DHS officials, including the Chief Technology Officer, on their efforts to conduct an Agile pilot to assist programs like TIM in adopting Agile software development processes. We assessed the evidence against leading practices to determine the extent to which TSA met the practices.

To determine the extent to which TSA and DHS are effectively overseeing and governing the program to ensure that it is meeting cost, schedule, and performance requirements, we identified leading practices and guidance outlined in the following sources:

TSA Agile Scrum guidance

CMMI for Development, version 1.3

Software Engineering Institute, Agile Metrics After reviewing the sources listed, we grouped practices related to oversight and governance for programs using Agile software development into four key practice areas and we discussed the practices with DHS and TSA officials. These areas included:

Document relevant governance and oversight policies and procedures.

Monitor program performance and progress.

Rely on complete and accurate data to review performance against expectations.

Ensure that corrective actions are identified and tracked until the desired outcomes are achieved.

To assess the extent that TSA and DHS had addressed these key practices, we reviewed the most current program management and governance documentation as of April 2017. Specifically, we analyzed documentation on program management processes, such as TIM’s Systems Engineering Life Cycle Tailoring Plan, TIM Agile and Technical Strategy, TIM Agile software development contract, and draft DHS Agile Acquisition Program Delivery Metrics Playbook; and artifacts from TIM’s program execution and review, such as Agile release artifacts, program status reports, contractor status reports, program schedules, life-cycle cost estimates, risk registers, TSA Executive Steering Committee reviews, DHS program health assessments, DHS Agile pilot integrated product team meetings, DHS Office of the Chief Technology Officer Agile pilot reviews, and DHS Acquisition Review Board reviews.

Additionally, we interviewed TSA officials, including the TIM Director and Deputy Director, on their efforts to oversee TIM’s development. Further, we interviewed DHS officials, including the Chief Technology Officer, on their efforts to oversee the program’s Agile software development activities. We compared this evidence against leading practices to determine the extent to which TSA and DHS met the practices.

To assess the reliability of the data that we used to support the findings in this report, we reviewed relevant program documentation to substantiate evidence obtained through interviews with agency officials. We determined that the data used in this report were sufficiently reliable for the purposes of our reporting objectives. We made appropriate attribution indicating the sources of the data.

We conducted this performance audit from September 2016 to October 2017 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix III: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, the following staff made key contributions to this report: Shannin G. O’Neill (Assistant Director), Jeanne Sung (Analyst in Charge), Jennifer Beddor, Rebecca Eyler, Bruce Rackliff, and Dwayne Staten.