Agile Software Development: DHS Has Made Significant Progress in Implementing Leading Practices, but Needs to Take Additional Actions
Released: 2020-06-01T00:00:00

Letter

Background

DHS and its components invest billions of dollars each year to acquire IT and other capabilities to support the department’s critical functions. The department plans to spend approximately $2.3 billion on major IT investments in fiscal year 2020.

However, DHS has faced long-standing challenges in acquiring and managing IT. We have highlighted the department’s IT management issues on our high-risk list since 2003 and have made numerous recommendations to improve its IT management practices. For example, in 2013, we testified that, out of 68 major IT investments that the department had in development, 21 had one or more subsidiary projects that were not meeting cost and/or schedule commitments due to technical issues in the development phase, changes in agency priorities, or a lack of understanding of user requirements, among other things.

Overview of Incremental and Agile Software Development

Many federal agencies, including DHS, are accustomed to using a waterfall software development model. This type of model typically consists of long, sequential phases, resulting in product delivery years after program initiation. With many federal IT investments in a development phase, it is important to ensure that agencies are making the most efficient use of their financial resources through effective management practices. However, as we have previously reported and testified, historically federal IT projects often fail—that is, even after exceeding their budgets by millions of dollars and delaying the schedules by years—and the results do not meet requirements.

Recognizing the severity of challenges related to the government-wide management of IT, in December 2014, federal IT acquisition reform provisions (commonly referred to as FITARA) were enacted as a part of the Carl Levin and Howard P. ”Buck” McKeon National Defense Authorization Act for Fiscal Year 2015. One of the provisions requires that the Office of Management and Budget (OMB) require in its annual IT capital planning guidance that each covered agency’s chief information officer (CIO) certify that IT investments are adequately implementing incremental development, as defined in capital planning guidance issued by OMB.

Agile software development—one form of incremental development— calls for the rapid delivery of software. Probably the most well-known feature of Agile software development is iterative product development and delivery; that is, development of software in segments that are continuously evaluated against requirements. This method is well suited for programs in which the final product is to include distinct features, some of which may be discovered during the process rather than planned at the beginning. These frequent iterations can effectively measure progress and allow developers to respond quickly to feedback from customers, thus reducing technical and programmatic risk. With its emphasis on early and continuous delivery of working software, Agile can be a valuable tool for agencies in mitigating schedule and budget risks.

Figure 1 compares requirements, design, development, and testing using Agile software methods versus a traditional waterfall approach; illustrating how requirements, design, development, and testing are performed concurrently in smaller time-boxed iterations for Agile and sequentially in waterfall development. As a result, using an Agile framework should result in producing high-quality software with frequent reviews and customer feedback to ensure that the highest value requirements are being met. The figure assumes that planning for both Agile and waterfall development has already occurred.

DHS Adopted Agile Software Development to Address IT Challenges

In February 2016, the DHS Under Secretary for Management announced an effort to pilot the use of Agile development methodologies to improve the department’s execution and oversight of IT acquisitions. This resulted in five Agile pilot programs. Each pilot program was overseen by a component integrated program team. Collectively, the first pilot programs were also overseen and supported by a DHS integrated program team. In April 2016, the department issued an Agile instruction, which identified Agile software development as the preferred approach for all DHS programs and projects that are to deliver an IT, or embedded-IT, capability. The department also set an expectation for its component CIOs to develop plans to increase the use of Agile development and justify any major IT programs that did not intend to use Agile development practices. Many DHS programs were already using Agile or similar incremental development methods before the department identified it as the preferred approach.

Roles and Responsibilities for Agile Programs

The DHS CIO, as the individual delegated departmentwide responsibility for approving, managing, and overseeing all of the department’s IT programs, sets the policies and procedures to help ensure Agile practices meet the department’s goals and comply with acquisition management policy. The DHS CIO is supported in this effort by the heads of other major DHS lines of business, such as the Chief Procurement Officer.

Table 1 describes the roles and responsibilities that support Agile development within the department.

Additionally, DHS established a headquarters-level team—the ITPM COE—to collaborate across the department on improvements to policy, governance, and acquisition guidance. In April 2017, the ITPM COE assumed responsibilities for the department’s transition to Agile development. The Office of the Chief Technology Officer (OCTO) Strategic Technology Management (STM) division within the OCIO facilitates the ITPM COE and serves as the official liaison between other OCIO divisions, other partner headquarters directorate and management offices, and operational components as needed.

GAO Previously Reported on Challenges in DHS’ Management of Agile Programs

We have reported on various programmatic and technical challenges that were limiting DHS’ efforts on Agile programs. For example, In 2016, we reported that the U.S. Citizenship and Immigration Services Transformation program, which was using Agile software development to modernize citizenship and immigration benefits processing, needed to improve testing of its software code and ensure its approaches to interoperability and end user testing met leading practices. We made 12 recommendations to improve Transformation program management, including ensuring alignment among policy, guidance, and leading practices in areas such as Agile software development and systems integration and testing. DHS concurred with the recommendations and has thus far implemented eight of them.

We reported in October 2017 that the Transportation Security Administration Technology Infrastructure Modernization program had not defined key roles and responsibilities, prioritized system requirements, or implemented automated capabilities that were essential to ensuring effective adoption of Agile. We made 14 recommendations including that DHS should prioritize requirements and obtain leadership consensus on oversight and governance changes. DHS concurred with the recommendations and to date has implemented 13 of them.

In November 2018, we reported that the U.S. Secret Service OCIO did not fully measure post-deployment user satisfaction with one project supporting the Information Integration and Technology Transformation investment. We made 13 recommendations to the U.S. Secret Service including that the Secret Service establish a process that ensures the CIO reviews all IT contracts, as appropriate; and identify the skills needed for its IT workforce. DHS concurred with the recommendations but has not yet implemented them.

We reported in April 2019 that the Federal Emergency Management Agency Grants Management Modernization program had not yet fully established plans for implementing new business processes or established completed traceability of IT requirements. We made eight recommendations to implement leading practices related to reengineering processes, managing requirements, scheduling, and implementing cybersecurity. DHS concurred with the recommendations and has thus far implemented two of them.

Organizational Change Management

According to the Project Management Institute, the practice of change management is a comprehensive, cyclic, structured approach for transitioning individuals, groups, and organizations from a current state to a future state with intended business benefits. It helps organizations to integrate and align people, processes, structures, culture, and strategy. The Project Management Institute and GAO have both described leading practices for effective organizational change management.

DHS Has Made Progress in Implementing Leading Practices, but Has Not Fully Addressed Others

Leading practices in organizational change management advise an agency to (1) plan for, (2) implement, and (3) measure the impact when undertaking a significant change, such as a transition from one software development approach to another. Since DHS committed to its transition to Agile software development in policy in April 2016, the department has fully developed plans to facilitate the transition. However, DHS has not fully implemented these plans and has experienced challenges in measuring progress against its intended goals. In addition, many of the plans are part of a larger effort to improve overall IT acquisitions rather than specific to a transition to Agile development, an approach that may delay DHS’s execution of these plans.

Leading practices for Agile software development adoption advise an agency to focus on three organizational levels of adoption: (1) agency environment, (2) program processes, and (3) team activities and dynamics. DHS has partially adopted practices at all three organizational levels. For example, the agency activities fully supported Agile methods through actions such as developing policies and procedures that called for the alignment of software, program goals, and agency goals. However, the department’s culture can better support Agile methods by, among other things, demonstrating an incentives and rewards structure to incentivize Agile teams.

DHS Has Made Progress in Implementing Nine Leading Practices for Agile Software Development Adoption, but Has Not Fully Implemented All

Leading practices that we developed for Agile software development adoption are organized into three areas, called organizational levels: agency environment, program processes, and team activities and dynamics. The organizational levels are further divided into nine leading practices. Table 2 identifies the three organizational levels and nine leading practices associated with these levels (three practices within each area). A detailed assessment of DHS’s implementation of each of the nine leading practices can be found in appendixes III, IV, and V.

We refer to the leading practices related to an agency’s processes, culture, and acquisition strategies as agency environment practices. For an agency to successfully transition from an agency that supports traditional development methods, it should ensure that its activities, culture, and acquisition policy and procedures support Agile methods. DHS partially implemented the agency environment practice level by fully implementing two leading practices and partially implementing the remaining one. A more detailed assessment of DHS’s agency environment leading practices can be found in appendix III.

Agency activities support Agile methods–fully implemented. DHS established appropriate life cycle activities to support Agile methods. For example, the department has outlined its policies, procedures, and guidance in several documents to assist its components in the acquisition and implementation of Agile software development. The department also developed policies and procedures that called for the alignment of software, program goals, and agency goals.

Agency culture supports Agile methods–partially implemented. DHS established an environment that supported Agile development, and senior stakeholders supported its development throughout the agency. However, DHS did not take sufficient steps to ensure that senior stakeholders serving as executive sponsors understood Agile development, as called for by leading practices that are described in further detail in appendix III. The Director of STM stated that Agile sponsors were considered to be chief executive officers (e.g. Executive Director of PARM and the Deputy Under Secretary for Management). These parties oversaw the actions of the ITPM COE and approved the Agile action plans in June 2017.

In addition, the department did not require training for senior stakeholders serving as executive sponsors, as called for by leading practices. In a written response, the Office of the Chief Human Capital Officer said that there were no Agile training requirements for officials at this level. By training executive-level sponsors in Agile development, the department can mitigate the risk of setting expectations for programs and projects that do not align with the values and principles of Agile software development.

DHS also did not demonstrate that it established an incentives and rewards structure to incentivize Agile teams, as called for by leading practices. Officials from the Office of the Chief Human Capital Officer stated that the department’s existing rewards structure allowed for incentivizing team and individual performance even though it was not focused specifically on Agile methods. These officials stated that they did not believe that additional policy, guidance, or modifications to their existing policy were necessary. The Director of STM within OCIO stated that rewarding Agile teams was not a topic the ITPM COE was currently considering, but that OCIO might be interested in pursuing the topic after completing existing, higher-priority activities. By considering modifications to policy and guidance governing the incentives and rewards structure to promote team performance, DHS could improve team productivity and output.

Agency acquisition policies and procedures support Agile methods–fully implemented. DHS guidance for acquisition strategies supported the unique needs of Agile programs. For example, DHS offered guidance for preparing acquisition strategies through its Procurement Innovation Lab and published Agile guidance that discussed contracting and acquisition strategies.

Program processes involve staff being appropriately trained in Agile methods, technical environments enabling Agile development, and project planning controls being compatible with Agile development. DHS partially implemented the program processes practice level by fully implementing one leading practice and partially implementing the remaining two. A more detailed assessment of DHS program process leading practices can be found in appendix IV.

Staff are appropriately trained in Agile methods–partially implemented. DHS training policy and guidance called for some of the acquisition management program staff to be trained in Agile methods. DHS has also taken steps to incorporate Agile concepts into required training for members of the acquisitions workforce. In addition, DHS offered elective training covering Agile methods and guidance for Agile teams, including contractors, to have the appropriate technical expertise needed to perform their role.

The department also took steps to identify the necessary competencies for Agile teams and individuals. In April 2019, the Strategic Workforce Planning team within OCIO published a white paper identifying 27 competencies necessary for teams and individuals to use and training courses associated with the competencies. The white paper also made recommendations to help DHS address challenges in implementing Agile methods, such as establishing communities of practice for Agile practitioners to identify best practices and provide workshops. According to a written response by OCIO, the Strategic Workforce Planning team will create an implementation and communication plan for any deliverables associated with the white paper.

However, the department did not provide policy or guidance to ensure that all program staff were trained in Agile methods, as called for by leading practices described in further detail in appendix IV. Existing Agile training requirements covered only the acquisitions workforce. DHS did not establish training requirements for program staff outside of the acquisitions workforce—such as a product owner or other staff—who may be assigned to an Agile program. As a result, individual programs must independently decide on and enforce training requirements if they want to ensure that all staff receive the needed training.

DHS officials stated that the department focuses on key acquisition career fields in part because those career fields are defined in policy and procedures. According to the Director of STM, the department also encourages programs to independently find coaching and training because the components are more likely to have funding. By providing policy or guidance to ensure that all personnel staffed to an Agile program or project receive appropriate training, the department can better prepare program staff to plan and execute appropriately, and increase the likelihood of achieving the expected outcomes of the transition to Agile.

Technical environments enable Agile development–fully implemented. DHS guidance called for technical and project tools to be available to support Agile development. For example, DHS test and evaluation guidance stated that automated testing should be implemented where practical.

In addition, DHS guidance called for system designs that will support iterative delivery. For example, DHS enterprise architecture guidance and supplementary design considerations for acquisition programs discussed loose coupling and different methods for establishing a modular system.

Project planning controls are compatible with Agile development–partially implemented. DHS guidance called for defining and incorporating non-functional requirements and critical features throughout development. In addition, DHS provided guidance for establishing a sustainable development pace. For example, the Agile instruction manual identified the benefits of monitoring the amount of work completed by Agile teams across each iteration in order to monitor ongoing team progress.

However, DHS was not tracking and monitoring the pace of Agile team development as called for by DHS guidance and described further in appendix IV. According to the Director of STM, programs were not consistently reporting the Agile core metrics associated with development team pace as required. The Director of STM stated that the department was taking steps to begin tracking and monitoring the pace of Agile teams. In addition, the Director stated that he allocated staff to assist programs with consistently reporting the Agile core metrics.

According to the Director of STM, the department was in the process of updating the core metrics and intended to publish a new version of them in the future, which would include tracking the pace. Nevertheless, DHS did not provide assurance that the metrics associated with development pace would be included in this revised set of metrics or that programs would consistently report that information in order for the department to track and monitor the pace of Agile teams. Until the department consistently tracks and monitors Agile programs and projects, it will not have the information needed to help ensure the development pace is maintained.

Practices at the team activities and dynamics level include team composition supporting Agile methods, work being prioritized to maximize value for the customer, and repeatable processes being in place. DHS partially implemented the team activities and dynamics practice level by fully implementing one leading practice and partially implementing the remaining two. A more detailed assessment of DHS team activity and dynamics leading practices can be found in appendix V.

Team composition supports Agile methods—fully implemented.

DHS established guidance that called for self-organizing teams and defined the role of a product owner. For example, the Agile instruction and Agile instruction manual both explain that collaborative, self- organizing, and cross-functional teams help achieve the flexibility needed for the iterative development that characterizes Agile development methods. In addition, the Agile instruction manual states that the product owner is responsible for representing stakeholders and should be available to the development team throughout the iteration to answer questions and clarify requirements on behalf of the stakeholders.

Work is prioritized to maximize value for the customer—partially implemented. DHS guidance called for Agile teams to craft user stories to define work. The guidance also called for user stories to be prioritized in a backlog based on value.

However, the guidance did not describe how Agile teams can estimate the relative complexity of the user stories as called for by leading practices and described in further detail in appendix V. The Director of STM stated that relative estimation is a basic exercise and that guidance on this topic can be found in a number of sources outside of DHS. However, without providing guidance or directing Agile teams to external sources for additional information on relative estimation, OCIO risks that teams supporting Agile projects will not appropriately estimate user stories relative to each other.

By providing guidance on estimating the relative complexity of user stories, the department can help Agile teams to effectively commit to an appropriate amount of work during a given iteration.

Repeatable processes are in place—partially implemented. DHS guidance addressed holding daily meetings to review progress and discuss impediments, using a demonstration for the acceptance of a user story and conducting a retrospective to evaluate progress. In addition, the department’s guidance called for Agile programs to employ continuous integration and emphasized the need for mechanisms to help ensure code quality.

However, DHS did not set expectations for automated testing and code quality, as called for by DHS guidance and described further in appendix V. DHS’s Agile core metrics included a series of metrics that addressed automated testing and code quality. The core metrics included targets but the targets were notional and, therefore, not expectations that DHS required a program to meet. According to the Director of STM, the initial core metrics were intended to assess the level of DHS team achievement without imposing artificial industry- based target measures for each. The Director stated that, on receiving the metrics for a period of time, the department would then adjust the core metrics and begin to include target measures based on the results achieved. According to the Director, this effort is currently underway and an updated set of core metrics will be distributed in early fiscal year 2020.

Moreover, the department did not track and monitor automated testing or code quality against expectations. As discussed under project planning controls, DHS intended to track and monitor Agile practices, such as automated testing and code quality, through the Agile core metrics. However, according to the Director of STM, programs and projects were not consistently reporting these core metrics and those that were reporting did not collect data or report on particular metrics.

By setting expectations for automated testing and code quality and beginning to track and monitor project performance against these expectations, DHS can increase the likelihood that Agile programs and projects are delivered within cost, schedule, and performance estimates.

Conclusions

DHS has taken many positive steps in its transition to Agile software development. It has implemented activities and artifacts that support all levels of adoption, from the department and component offices to Agile programs, projects, and teams. These activities and artifacts include providing opportunities for Agile programs and projects to streamline acquisition and life cycle processes to allow for iterative delivery and exhibiting senior support for the transition to Agile.

The department successfully planned for the transition to Agile software development and completed many of its intended implementation activities. However, because DHS did not assess the skills and resources needed to complete deferred activities, it risks continued delays in completing these. In addition, without identifying target measures tied to expected outcomes, the department is limited in determining whether the transition is achieving its desired outcomes. Moreover, until DHS can ensure that all programs are consistently reporting on Agile core metrics, the department will not be able to track programs’ development techniques. Further measuring and communicating the benefits of the transition can enable the department to know whether Agile programs are performing better than those used prior to the transition.

DHS has demonstrated significant progress in implementing leading Agile practices. The department can further improve its performance through full execution of the remaining partially implemented practices. At the agency environment level, DHS can mitigate risk and improve productivity through executive level training and modifications to policy to incentivize Agile teams. For program level practices, addressing training requirements for all necessary staff and tracking and monitoring the pace of Agile team development can help ensure teams’ success.

With respect to team-level practices, DHS has not established guidance for estimating the relative complexity of user stories. As a result, Agile teams are hampered in effectively committing to an appropriate amount of work during a given period of time. Finally, because DHS has not set expectations for performance metrics for monitoring and tracking the use of automated testing and code quality, DHS is at a greater risk for programs breaching their cost and schedule expectations.

Recommendations for Executive Action

We are making the following 10 recommendations to the Secretary of the Department of Homeland Security (DHS).

The Secretary should ensure that the Director of Strategic Technology Management (STM), in collaboration with other members of the Information Technology Program Management Center of Excellence (ITPM COE), identifies the skills and resources needed to complete the work intended for the upcoming fiscal year, including the availability of supplementary staff, such as subject matter experts. (Recommendation 1)

The Secretary should ensure that the Executive Steering Committee overseeing the activities of the ITPM COE establishes target measures for the department’s desired outcomes of its transition to Agile development. (Recommendation 2)

The Secretary should ensure that the DHS Chief Information Officer (CIO) defines a process and associated set of controls to ensure that Agile programs and projects are reporting a set of core required performance metrics for monitoring and measuring Agile adoption. (Recommendation 3)

The Secretary should ensure that the ITPM COE, in coordination with the CIO, begins measuring results associated with the transition to Agile and the success of the transition based on its impact on the department. (Recommendation 4)

The Secretary should ensure that the CIO, in collaboration with the Chief Procurement Officer, through the Homeland Security Acquisition Institute, establish Agile training requirements for senior stakeholders. (Recommendation 5)

The Secretary should ensure that the Chief Human Capital Officer, in collaboration with the CIO, consider modifications to the current employee recognition and performance management governance to ensure that teamwork and team performance of Agile programs and projects are incentivized. (Recommendation 6)

The Secretary should ensure that the CIO, in collaboration with the Chief Procurement Officer, through the Homeland Security Acquisition Institute, establish Agile training requirements for staff outside of the acquisition workforce but assigned to Agile programs. (Recommendation 7)

The Secretary should ensure that the CIO, upon establishing a set of core performance metrics, tracks and monitors the pace of Agile team development. (Recommendation 8)

The Secretary should ensure that the CIO, in collaboration with the Executive Director of the Office of Program Accountability and Risk Management (PARM), update or develop new guidance on Agile methodologies to describe how Agile teams can estimate the relative complexity of user stories. (Recommendation 9)

The Secretary should ensure that the CIO, upon establishing a set of core performance metrics, sets expectations for automated testing and code quality, and tracks and monitors against those expectations. (Recommendation 10)

Agency Comments and Our Evaluation

DHS provided written comments on a draft of this report. In its comments (reproduced in Appendix VI), the department agreed with our 10 recommendations and described actions that it had completed and planned to address them.

Based on the actions DHS said it had taken, the department requested that we close the first three recommendations as implemented. For example, the department described steps it had taken to address our recommendation that it identify the skills and resources needed to complete the work intended for the upcoming fiscal year, including the availability of supplementary staff such as subject matter experts. In addition, the department stated that it had addressed our recommendation to define a process and controls to ensure that Agile programs and projects are reporting a set of core required performance metrics for monitoring and measuring Agile adoption. We plan to follow up with DHS to assess the sufficiency of its actions to address our recommendations.

The department also described actions that it plans to take to address the other seven recommendations. For example, DHS stated that it will use the results of its Agile core metrics and Agile Software Delivery Maturity Model to measure the success of the transition to Agile and its impact on the department. According to the department, it expects this action to be completed by June 30, 2021.

Further, DHS stated that it will identify Agile training requirements for staff in Agile programs, and will use that to establish Agile training requirements for staff outside of the acquisition workforce but assigned to Agile programs. Specifically, DHS stated that the DHS OCIO will gather requirements from components via its IT workforce planning integrated project team to identify training resources available across the department that also address the skill sets needed for Agile programs. The department added that the DHS OCIO will utilize information from the April 2019 white paper, titled “OCIO Agile White Paper” to inform proposed Agile program training requirements. The department estimated that these actions are to be completed by September 30, 2020.

DHS also provided technical comments, which we have incorporated as appropriate.

We are sending copies of this report to the Acting Secretary of Homeland Security and interested congressional committees. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-4456 or harriscc@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix VII. In addition, the report is available at no charge on the GAO website at http://www.gao.gov.

Appendix I: Objective, Scope, and Methodology

Our objective was to assess the extent to which the Department of Homeland Security (DHS) addressed selected leading practices for its transition to the use of Agile software development. To accomplish this objective, we assessed the extent to which the department adhered to leading practices in two specific areas: organizational change management and Agile software development adoption.

With regard to organizational change management, we reviewed leading practices published by the Project Management Institute and GAO. Based on this review, we identified 15 leading practices. We then grouped these 15 practices in three broad organizational change management areas: planning, implementing, and measuring change.

To determine the extent to which DHS addressed leading practices for organizational change management in its transition to Agile development, we assessed DHS policies, procedures, guidance, plans, and other working group artifacts and compared them against leading practices. In particular, we reviewed working group charters for the DHS headquarters Agile Acquisition Integrated Program Team and IT Program Management Center of Excellence (ITPM COE), and any plans developed by these working groups, including the DHS Agile Action Plans and associated implementation plans. We then reviewed working group meeting minutes, presentation slides, and status update charts to assess the progress of the transition to Agile, identified artifacts prepared to support the transition to Agile, and assessed the status of plans for the transition to Agile. We reviewed all Agile artifacts prepared by or supporting the Agile working groups, such as a preliminary software development maturity model, the DHS Agile Acquisition Software Delivery Core Metrics (Agile core metrics), and an updated test and evaluation master plan template for Agile, among other artifacts.

We also interviewed officials from DHS headquarters line of business representatives explicitly identified in the Agile Development and Delivery for Information Technology instruction (Agile instruction). This included officials from the Office of the Chief Procurement Officer, Office of the Chief Financial Officer, Office of the Chief Information Officer (OCIO), Office of Program Accountability and Risk Management (PARM), and the Science and Technology Directorate, and offices of Test and Evaluation and Systems Engineering. Within OCIO, we interviewed officials from the Office of the Chief Technology Officer (OCTO) within the Strategic Technology Management (STM) division, among others, as STM is the entity tasked with facilitating the ITPM COE and serves as the official liaison between other OCIO divisions, other partner headquarters directorate and management offices, and operational components. We also interviewed representatives from groups participating in ITPM COE activities but not explicitly called out in the Agile instruction, including the Privacy Office and Joint Requirements Council. In addition, we interviewed representatives from other groups not represented on the ITPM COE but potentially impacted by the transition to Agile. This included officials from the Office of the Chief Readiness Support Officer and Office of the Chief Human Capital Officer.

With regard to leading practices for Agile software development adoption, we reviewed work performed by GAO to develop generally accepted leading practices. In developing these leading practices, GAO reviewed information from a variety of sources related to Agile adoption and compiled a draft of leading practices commonly mentioned across these different sources. We then convened a working group of experts from the public and private sectors and academia. This working group met three times a year between August 2016 and August 2019 to review and discuss these leading practices. More than 200 experts participated in the meetings, including more than 20 officials from DHS. GAO received comments from some of these experts both during these meetings and by email after the meetings.

Based on this work, GAO developed a set of nine leading practices for Agile adoption. GAO grouped these leading practices into three organizational levels: (1) agency environment, (2) program processes, and (3) team activities and dynamics. The leading practices were further described by a series of core elements and core element expectations that, collectively, can be used to assess the status of an agency’s implementation.

To determine the extent to which the department had implemented the leading practices for the adoption of Agile development, we obtained and assessed DHS policies, procedures, guidance, plans, and other documentation and compared them against the nine leading practices. In particular, we reviewed department acquisition policy, procedures, and guidance, such as acquisition management directive 102-01; software engineering life cycle policy, procedures, and guidance, such as those published in the software engineering life cycle guidebook; requirements policy, procedures, and guidance, such as the Joint Requirements Integration and Management System and Requirements Engineering User’s Guide; testing policy, procedures, and guidance, such as the Test and Evaluation Master Plan template and Test and Evaluation Management Guide; technical assessment and enterprise architecture policy, procedures, and guidance; program health assessment policy, procedures, and guidance such as the Acquisition Program Health Assessment instruction and CIO Program Health Assessment Scoring Guideline; and Agile-specific policy, procedures, and guidance, such as the Agile instruction and the Agile Development and Delivery for Information Technology Instruction Manual (Agile instruction manual), among other policy, procedures, and guidance.

In addition to reviewing the department policy, procedures, and guidance, we obtained and assessed supplementary Agile documentation. In particular, we reviewed training materials prepared by the Homeland Security Acquisition Institute for acquisition workforce certifications and webinars offered by the Procurement Innovation Lab; ITPM COE Agile artifacts discussed under our assessment of the implementation of organizational change management leading practices, such as the Agile core metrics; and Agile-specific technical review completion letters, such as the release planning review.

We also interviewed officials from the components responsible for the associated policy, procedures, and guidance and those specifically cited in the Agile instruction. This included officials from the Office of the Chief Procurement Officer, Office of the Chief Financial Officer, OCIO, PARM, Science and Technology Directorate, offices of Test and Evaluation and Systems Engineering, the Joint Requirements Council, Office of the Chief Readiness Support Officer, and Office of the Chief Human Capital Officer. As with our assessment of DHS implementation of organizational change management practices, within OCIO, we interviewed officials from the OCTO STM division, among others.

We assessed a core element as being “met” if the department provided supporting documentation that demonstrated it met all of the expectations associated with the core elements. We assessed a core element as being “partially met” if the department provided supporting documentation that demonstrated some, but not all, aspects of the underlying expectations. We assessed a core element as “not met” if the officials did not provide any supporting documentation for the core element, or if the documentation provided did not demonstrate any aspect of the underlying expectations. The expectations associated with each core element are described more fully in appendixes III, IV, and V.

We assessed each leading practice and practice level as being “fully implemented” if DHS provided evidence that it had met all of the core elements. We assessed each leading practice and practice level as being “not implemented” if DHS did not provide evidence that it had met or partially met any of the core elements. We assessed each leading practice and practice level as being “partially implemented” if DHS provided evidence that it had not met all core elements and partially met at least one core element.

To supplement our assessment of the department’s implementation of the leading practices for adopting Agile development, we also assessed selected projects’ implementation of selected program process and team activity and dynamics leading practices. We updated the core element test plans to include general control objectives, associated controls, and associated test steps in order to reach a determination on the extent to which these projects implemented a particular aspect of a leading practice.

We identified potential case study projects based on data provided by DHS from the Investment Evaluation, Submission, & Tracking system. We determined that the data in the Investment Evaluation, Submission, & Tracking system was sufficiently reliable for our use in selecting projects for our case studies. We selected case study projects, rather than programs, because, according to DHS officials from OCIO, programs report software development life cycle data to the Investment Evaluation, Submission, & Tracking system at the project level only.

We selected only the projects supporting programs on the Major Acquisition Oversight List because these programs are expected to comply with the Agile instruction and acquisition management policy. We then further limited the scope of projects to those within components where GAO has not previously assessed a program using Agile methods or was not in the process of assessing such a program. This excluded the U.S. Citizenship and Immigration Services, Federal Emergency Management Agency, Transportation Security Administration, and U.S. Secret Service.

We then further refined our selection based on the following criteria:

Software development life cycle methodology (iterative development only)

Project completion date (in-progress only)

DHS component (selection of only one project per component)

We then selected a random sample of three projects, with no more than one project selected from a component. The three case study projects we selected were the U.S. Coast Guard (USCG) Command, Control, Communications, and Computers, Intelligence, Surveillance and Reconnaissance (C4ISR) program New Asset Acquisition Offshore Patrol Cutter project, with particular attention to the SeaWatch portion of this project; the U.S. Customs and Border Protection (CBP) Biometric Entry Exit (BEE) program Air Exit project, with particular attention to the Traveler Verification Services portion of this project; and the U.S. Immigration and Customs Enforcement (ICE) Student and Exchange Visitor Information System (SEVIS) program 8001 project, with particular attention to the SEVIS modernization portion of this effort. In preliminary interviews, we confirmed that these projects were applying Agile practices in order to validate data reported to the Investment Evaluation, Submission, and Tracking system.

These case studies were used to supplement our findings from our program process and team activity and dynamics-level evaluations of the department’s implementation of leading practices for adopting Agile development. To evaluate case studies’ implementation of these leading practices, we reviewed artifacts from the selected projects. In particular, we reviewed artifacts demonstrating a project’s use of Agile including testing metrics, evidence of Agile ceremonies, the existence of user stories and a backlog, and the availability of Agile coaching and training.

We then interviewed officials responsible for program and project management and representatives of groups responsible for software development for the three selected case study projects to discuss gaps we identified. We shared our initial assessment with DHS, USCG, CBP, and ICE and obtained feedback and additional supporting documentation.

Regarding our analysis of project implementation of the program process and team activity and dynamics core elements, we followed the aforementioned process in assessing a core element as being “met”, “partially met”, or “not met”. These assessments were used to gain insight into the extent to which DHS policy, procedures, and guidance prepared programs and projects for the successful adoption of Agile leading practices. We did not evaluate the projects in order to make specific recommendations to the individual projects.

We conducted this performance audit from December 2017 through April 2020, in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: DHS Agile Action Plans

In June 2017, Department of Homeland Security (DHS) senior stakeholders endorsed Recommendations Action Plans: Agile Acquisition Pilots, developed by the Agile Acquisitions Working Group. These recommendations were an effort to sustain the success of the information technology (IT) acquisition and delivery pilot program. The action plans were developed in response to the February 18, 2016, Acquisition Decision Memorandum from the Under Secretary for Management, which recognized the expressed need for both components and headquarters directorates to continue driving organizational change and process improvement to DHS IT acquisitions and delivery. The action plans were intended to codify lessons learned and recommendations based on independent interviews and retrospective meetings with those who participated in the five acquisition pilots. These plans were organized by priority: 12 critical, three high, and three moderate. The recommendations were weighted against one another based on impact, level of difficulty, and alignment with the original five goals of the Agile acquisition pilot program charter: reduce risk, increase customer value, faster time to market, economic value, and increased accountability and oversight. Table 3 describes the DHS Agile action plans, including the associated goal, primary organization(s), level of difficulty, impact, and executive priority.

Each DHS action plan included a problem statement and recommendation, as detailed in table 4.

Appendix III: Leading Practices for Adopting Agile Development—Agency Environment

This appendix describes in detail our evaluation of the three leading practices for agency environment when adopting Agile development, including a further explanation of expectations for each practice as well as some of the findings associated with each practice. We do not present any additional recommendations from these findings; this information is intended to assist the Department of Homeland Security (DHS) in implementing the recommendations described in our report.

Agency activities support Agile methods

Establish appropriate life cycle activities Agency activities should support Agile methods by allowing for incremental and iterative software delivery that is tailored to the cadence of Agile software development and by incorporating technical reviews that occur throughout the development process. These activities and supporting policy and guidance should allow for requirements to be changed during development and the requirements change approval process should not impede the cadence of iterative and incremental development. Life cycle activities should also be user-focused and call for collaboration between the development team and users.

To manage its multi-billion dollar investments, DHS has established policies, procedures, and guidance for IT program management. These publications govern the complete life cycle of a system, from technology development through integration and testing and, finally, implementation and operations and maintenance.

DHS has outlined its policies, procedures, and guidance in several documents to assist its components in the acquisition and implementation of software development. Policies for managing its major acquisition programs are primarily set forth in a directive and supporting instruction. These policies outline an acquisition life cycle framework (ALF) that includes a series of predetermined milestones—known as acquisition decision events—at which the Acquisition Decision Authority reviews a program to assess whether it is ready to proceed to the next phase of the ALF. DHS’s Under Secretary for Management serves as the Acquisition Decision Authority for the department’s major acquisition programs.

A separate DHS instruction and associated guidebook outline a framework of major systems engineering activities and technical reviews, collectively considered the systems engineering life cycle (SELC), which should be conducted by all DHS programs, both major and non-major. The SELC helps to ensure that appropriate systems engineering activities are planned and implemented and that a program’s development effort is meeting business needs. The SELC consists of major activities and a set of related technical reviews and artifacts that fit within the acquisition life cycle.

Figure 2 depicts the acquisition life cycle and associated technical reviews established in DHS acquisition management policy.

DHS provided programs with flexibility in their SELC technical reviews. Within the ALF, Agile processes are applied primarily within the obtain phase, where design, development, testing, and implementation of a system takes place. Prior to entering the obtain phase, a program selects its software development approach, such as Agile. The agreed-upon approach is then codified in an SELC Tailoring Plan, which is approved at acquisition decision event 2A. The SELC Tailoring Plan identifies the technical reviews and artifacts that the program is responsible for completing based on its unique characteristics (e.g., scope, complexity, and risk).

To assist in tailoring efforts and further guide the implementation of Agile, DHS published an Agile instruction that includes the scope, definitions, roles and responsibilities, and procedures for establishing an Agile framework for developing all DHS IT acquisitions. DHS supplemented this instruction with an Agile instruction manual and provided a template that Agile programs can follow to tailor their activities. For example, instead of holding a system definition review, an Agile program is encouraged to conduct a release planning review (which encompasses the development and release of a segment of software). This optional approach to tailoring a technical review is depicted in figure 3.

Outside of technical reviews, DHS updated acquisition policy in February 2019 and associated guidance in May 2019 to allow programs greater flexibility in the larger ALF. The Director of Strategic Technology Management (STM) stated that, under the previous acquisition policy and guidance, IT programs were using in-house expertise due to limited funding to prepare for the 2B decision, when full program funding was received. He noted that, by the time a contract was awarded for development following a 2B decision, the contractor might or might not have been using planning artifacts developed by the program and instead might have recreated them, thereby rendering 2 to 3 years of work useless. The Director stated that programs were unable to fully flesh out the program architecture and other key aspects of the program because programs did not receive funding until the 2B decision and in-house expertise was limited. For example, if a program had not proven out its architecture prior to a 2B decision, it could continue to refine and modify the architecture during the course of development, thereby impacting productivity and quality. DHS updated acquisition management policy and guidance to modify the requirements for the acquisition decision events and addressed a related GAO recommendation.

DHS policy and guidance also allowed for programs to modify requirements over the course of development. The traditional process for requirements may be modified as part of tailoring the SELC in order to allow for increased flexibility. The DHS Requirements Engineering User’s Guide detailed requirements engineering steps, activities, and methods for performing those steps.

DHS developed this user guide to supplement SELC policy and guidance. One section of this guide focused on Agile development. According to the guide, requirements are broken down over the course of the ALF and commitments are made at different levels of specificity. Fundamental capability gaps are defined in the mission needs statement presented at acquisition decision event 1. Subsequently, the analyze/select phase would ultimately define the high-level features and functions of each required capability, define the fundamental performance of those high- level features and functions, and establish the business case to support approving the acquisition at an acquisition decision event 2A. Often, a preliminary concept of operations is developed and delivered with the mission needs statement.

The guide also states that the activities to evaluate these potential alternatives will ultimately result in a preferred solution with defined business practices, methods, and processes that allow the development of business epics and associated architecture epics. Business epic is an Agile term that defines the high-level “stories” that describe a capability, or what the new system is required to perform. Architectural epic is an Agile term that defines the architecture the system will be incorporated into. In addition, the preferred solution would have defined high-level performance requirements (stated from the operational perspective) in terms of how well the solution must perform to be operationally effective and suitable. Key constraints such as security, Section 508 compliance, privacy, reliability, etc., should also be identified. These top-level requirements will be documented in the operational requirements document.

According to the guide, Agile teams capture the capabilities and constraints (essentially the functional and non-functional requirements that reflect the business epic level of performance) in an artifact called the capabilities and constraints document. Requirements statements in this document should follow the standard “shall statement” format for ease of translation between the operational requirements document and the capabilities and constraints document. The capabilities and constraints document and its contents mature over time and, as the document matures, business and architectural epics decompose to features/functions or themes, and ultimately to user stories that reflect the specific tasks that users will perform. Officials within the DHS Joint Requirements Council noted that headquarters involvement occurred at this level to approve the high-level operating requirements.

After headquarters oversight and approval, the program may then decompose requirements as part of planning for and executing technical reviews. If tailored into an Agile program, the capabilities and constraints document should drive the development of a backlog. The backlog is a list of all the user stories that describe what the system needs to do. The backlog should become more refined as the program decomposes the high-level features (a service that fulfills a user need) and functions down to specific stories that an individual software developer will code and test during a specific iteration.

To prevent the backlog from becoming unmanageable, DHS guidance stated that backlogs may be established at different levels. For example, the business and architectural epics along with the associated operating requirements would constitute the “program backlog.” Sub-epics are usually broken down into “high-level features” with business epics broken down into business features and architectural epics broken down into architectural features. Features or functions are decomposed into detailed stories that are then allocated to a “release”. The list of user stories in a specific release constitutes the release backlog. This process of decomposing stories continues to the iteration backlog.

DHS guidance places an emphasis on end user needs. The Requirements Engineering User’s Guide raised the importance of identifying stakeholders, including system users, and capturing the needs of those users via requirements or, in the case of Agile, user stories. The Agile instruction manual placed an emphasis on the importance of users to a program and articulated that the product owner represent the user community and was expected to continually seek ongoing feedback and elicit requirements from users. The Agile core metrics also strongly recommended the use of a net promoter score. This score was one mechanism for measuring customer satisfaction through asking users to rank how likely they would be to recommend the system or application to a friend or colleague, based on a score of 1 to 10.

Clearly align goals and objectives Program goals should clearly reflect stakeholder needs and concerns based on input from stakeholders and stakeholder review and approval. Program goals should align with strategic IT objectives. Software-related goals should be defined and clearly aligned with program goals. The agency should collect objective measures that are well defined to track progress towards achieving software goals so the agency knows which features and capabilities have been achieved.

The Requirements Engineering User’s Guide described program expectations for tracing from mission needs to operating and functional requirements, or user stories. The guide recognized that, as a program progressed through the ALF and SELC, it was important to trace requirements from the top-level mission needs or capabilities and/or business requirements down to the system/sub-system, component, or configuration item level that enabled those requirements to be met. This helped ensure continuity across various DHS artifacts, such as the program’s mission need statement, concept of operations, and operational requirements document, to vendor specifications (or applicable equivalent artifacts). Although an Agile program will modify the SELC to accommodate its needs, generally programs were expected to follow the same conceptual approach to the requirements of planning, development, and management.

The user’s guide stated that collaboration among the various stakeholders was important and the program requirements team must continuously work to establish partnerships and networks. To do so, the guide stated that the program team must identify all individuals and organizations that may be impacted by their program and ensure those stakeholders were engaged throughout development to facilitate understanding of their perspectives and needs. The first step was to identify applicable stakeholders, which would include end users, program sponsors, developers, maintainers, trainers, and other affected individuals or organizations. The program requirements team then solicited input from these stakeholders to understand their needs, policies, processes, and operations to begin the requirements definition effort. It identified some ways a team might begin the process of eliciting requirements from the stakeholders.

After collaborating with stakeholders, the stakeholder needs must be translated into the program requirements, or goals. The guide stated that the program requirements team should take the inputs from the various stakeholders and decompose, prioritize, de-conflict, and validate the needs identified. It clarified that a “good” requirement was achievable, testable, clear, concise, technology-independent, feasible, and able to stand alone.

The guide grounded all of the requirements elicitation and development process in the overall contribution to the agency mission, recognizing the need for general strategic alignment. In particular, the guide noted that requirements were “mission need” driven as opposed to “solution” driven. Requirements were developed throughout the life of a program, with the first formal requirements being the operating requirements documented in the operational requirements document.

To ensure that DHS’s mission or strategic goals were key inputs for decision making, DHS relied, in part, on its enterprise architecture process. DHS policy for enterprise architecture stated that the enterprise architecture program provided a vehicle to tie the strategic mission goals and objectives of DHS to the business processes, information resources, and technology investments necessary to reach key performance outcomes. This methodology was intended to integrate IT into the mission and strategic priorities of DHS, which provided the core foundation for all subsequent processes. DHS capital planning and investment control guidance reinforced this fact, stating that the Federal Acquisition Streamlining Act of 1994 required capital investments to align with mission and strategic goals. This included the framework within which the department formulated, managed, and maintained its portfolio of investments as critical assets for achieving success in the DHS mission and alignment to the DHS IT Strategic Plan and the DHS Strategic Plan.

Agency culture supports Agile methods

Cascading sponsorship for Agile software development Senior stakeholders should support and model the use of Agile, along with its values and principles, through explicit policy or guidance impacting the business and should take steps to complete responsibilities defined in agency Agile policy or guidance. Agile should also be supported in all relevant areas of the business impacting a software development project through the use of Agile sponsors. These sponsors should represent the lines of business in key agency decisions on Agile.

Senior stakeholders at DHS demonstrated support for Agile through the publication of policy and guidance that established Agile development as the department’s preferred approach for software development. As discussed previously, the department published Instruction 102-01-004 Agile Development and Delivery for Information Technology (Agile instruction), which provided the scope, definitions, roles and responsibilities, and procedures to establish an Agile framework for the development of IT acquisitions at DHS. Specifically, the Agile instruction established responsibilities for the CIO, the Chief Procurement Officer, the Chief Financial Officer, the Director, Office of Test and Evaluation within the Science & Technology Directorate, and the Executive Director of PARM.

Each of these five stakeholders and their associated components demonstrated their support for Agile development by taking steps to complete their responsibilities defined in the Agile instruction. For example, the Office of the Chief Information Officer (OCIO), the Office of the Chief Procurement Officer, and the Director, Office of Test and Evaluation within the Science and Technology Directorate all had responsibilities related to providing guidance for the implementation of Agile within their specific area of expertise. All three components had taken steps to execute these responsibilities, such as by publishing the Agile instruction manual, providing supplementary guidance for test and evaluation in an Agile environment, and offering elective training on contracting strategies for Agile services.

Representatives from offices with a role in software development also supported Agile via membership in the IT Program Management Center of Excellence (ITPM COE). In addition to the stakeholder organizations identified in the Agile instruction, the ITPM COE membership included representatives from the Joint Requirements Council and the Chief Privacy Officer. According to the ITPM COE charter, the ITPM COE served as a cross-functional team to identify and promote best practices, provide tools, and coordinate assistance for programs and projects to maximize the successful management of DHS IT investments. This included making progress towards the 18 Agile action plans that resulted from the Agile acquisition pilots.

The ITPM COE membership requirements called for representatives of the member organizations to be involved in key decisions regarding Agile. According to the Director of STM within OCTO, ITPM COE members were selected and approved by their organization’s executives. The ITPM COE charter stated that these representatives must be authorized to represent or make decisions on behalf of their officers or organizations. Officials from all ITPM COE member organizations expressed support for the ITPM COE and confirmed that their component was appropriately represented in decision making. This was represented, in part, by the fact that at least one representative for each ITPM COE member group attended at least half of the meetings. For example, at least one representative from the Science and Technology Directorate attended approximately 95% of the meetings.

Sponsor understanding of Agile software development Sponsors should understand and communicate changes resulting from Agile development. Sponsors should attend training or receive coaching on Agile and the agency’s framework, the agency should monitor completion of training, and sponsors should transmit learning from training to staff. Sponsors should also commit to achieving those intended results and sponsor performance should be tied to achieving those intended results.

The Director of STM stated that Agile sponsors were considered to be chief executive officers (e.g. Executive Director of PARM and the Deputy Under Secretary for Management). They oversaw the actions of the ITPM COE and approved the Agile action plans in June 2017.

DHS did not ensure that Agile sponsors attended training or received coaching in Agile development. The department made training available for Agile, including courses such as those required for acquisition professionals. However, in a written response, the Office of the Chief Human Capital Officer stated, and the Director of STM confirmed, that the department did not administer mandatory training on Agile for Agile sponsors.

The department also did not monitor the completion of sponsor training in Agile. Although DHS employees leveraged the Federal Acquisition Institute Training Application System to track their training and certifications, the department was not using this system to monitor sponsor training in Agile. According to a written response from the Office of the Chief Human Capital Officer, the department did not keep a record of whether sponsors completed training in Agile because the department did not require Agile training specifically for sponsors.

DHS Agile sponsors exhibited support for achieving the intended results from the transition to Agile. Agile sponsors committed to achieving these results through an endorsement of the 18 Agile action plans and the associated implementation plans.

However, DHS did not demonstrate that Agile sponsor performance was tied to achieving the intended results of the transition to Agile. According to a written response from the Office of the Chief Human Capital Officer, the department’s employee performance management policy did not specifically address Agile. This written response further stated that addressing Agile in these policies was unnecessary because the Office of the Chief Human Capital Officer incorporated goals derived from project plans in individual performance plans. DHS policy and guidance for performance management identified individual performance goals as a component of employee performance, but the department did not provide evidence that specific performance plans for the sponsors were linked to such goals.

Establish an environment supportive of Agile software development Team dynamics should be facilitated through access to common team rooms and/or modern communication and social media methods and headquarters infrastructure operations should allow for communal spaces and co-location in program offices. A headquarters technical environment should allow access to tools by programs to foster distributed communication, and there should be a process for continuous feedback on the Agile environment and modifications to that process (e.g. communities of practice, routine working group sessions). Agency governance bodies should allow programs greater autonomy and flexibility within existing acquisition processes through the modification of gate reviews and other touchpoints in the acquisition process for Agile projects and increased transparency for governance bodies into project operations when necessary.

DHS policy and guidance allowed for team dynamics to be facilitated through access to common team rooms and modern communication methods. In addition, department policy promoted and allowed program offices to support team dynamics through the use of communal spaces and co-location. Specifically, the Director of Systems and Information Integration within the Chief Readiness Support Office confirmed that DHS had modified policy related to infrastructure operations to allow any office to reorganize their space, citing the USCIS Transformation program as an example of this reorganization. The Director of Systems and Information Integration also noted that he was not aware of any restrictions to this practice in policy. With respect to facilitating access to modern methods of communication, DHS offered programs the option of using a suite of tools that included those for distributed communication.

DHS took multiple steps to establish a process for continuous feedback related to the department’s Agile environment and process modifications. According to the Director of STM, OCIO built support for Agile through the Centers of Excellence, communities of interest, brown bag lunches, and public speaking engagements. The Director added that these sessions facilitated the discussion of Agile and could be used to compile feedback.

The Director of STM explained that, as this feedback came in, it was either addressed immediately or put into a backlog. Efforts to further streamline the acquisition process were tracked via Agile action plan 6.

The department’s governance bodies also increased transparency into project operations when necessary. The Agile Development and Delivery for Information Technology Instruction Manual (Agile instruction manual) stated that the program or project manager should coordinate with the various oversight bodies that govern IT development. These bodies varied depending on the level of investment, but, for major programs, executive steering committees were often established to oversee all aspects of program planning and execution between major acquisition decision events. In addition, PARM officials stated that DHS increased the frequency of acquisition review board reviews and modified the content presented at the reviews to allow it to be more actively involved with projects earlier in the acquisition life cycle. Specifically, PARM updated the Acquisition Review Board slide templates and informed us of its intent to update acquisition management policy to require Agile projects to hold Acquisition Review Board reviews once every six months, as opposed to once every 12 months.

Align incentives and rewards to Agile methods The agency should establish an incentive and reward structure promoting team successes and the value of individuals within those teams. Management should establish agency goals to align incentives and rewards with Agile methods. Goals for incentives and rewards should align with the agency’s goal(s) and focus on team success. The agency should allocate incentives and rewards based on team success.

DHS did not establish an incentives and rewards structure that promoted team successes and did not demonstrate that management had established agency goal(s) to align incentives and rewards with Agile methods. Furthermore, the department did not demonstrate that human resources and others were actively involved in setting goals for incentives and rewards alignment.

DHS guidance specifically discussed contract incentives for Agile projects. For example, the Agile instruction manual suggested that consideration be given to address the duration of the base term and options, scalability, deliverables, and pricing with a mindset that contractors need appropriate incentives to encourage them to perform well. The manual also stated that contract award terms could provide a greater incentive for contractors working on longer-term Agile projects.

Although the department made efforts to adapt incentives and rewards for contractors supporting Agile projects, it acknowledged that it did not update existing incentives and rewards for federal employees working on Agile projects. Officials within the Office of the Chief Human Capital Officer stated that existing human capital and performance plan policy allowed for rewarding and incentivizing Agile teams as well as individuals. These officials further noted that DHS had numerous opportunities to recognize and reward team or individual performance, regardless of the development methodology a program relied on. Specifically, these officials clarified that the Office of the Chief Human Capital Officer used project plans to set goals and included those goals in employee performance plans. As these officials felt the existing performance plan policy was sufficient, they did not believe additional guidance or modifications to existing policy were necessary.

Agency acquisition policies and procedures support Agile methods

Guidance is appropriate for Agile acquisition strategies Agency acquisition policy and guidance should support awarding contracts for the unique needs of an Agile program. Acquisition strategies should recognize the need for interim delivery of software, allow for close coordination between the contracting office and program office staff, and allow for changing requirements and contract oversight mechanisms to be tailored to support Agile development methods.

DHS offered guidance for preparing acquisition strategies through its Procurement Innovation Lab. Webinars offered by the Procurement Innovation Lab on acquisition strategies for Agile programs discussed the need for interim delivery of software, close coordination between contractors and program office staff, contract oversight mechanisms that were tailored to support Agile development, and changing requirements. For example, the “Transportation Security Administration Agile Services Procurement” webinar discussed planning, executing, and de-briefing technical demonstrations used to select the contract recipient, paying particular attention to the value of transparency and modifying contract oversight mechanisms. Officials from the Office of the Chief Procurement Officer clarified that the webinars were available as needed and were not required training.

DHS also published Agile guidance that discussed contracting and acquisition strategies. From an oversight perspective, according to the Agile instruction manual, DHS executive steering committees oversee all aspects of program planning and execution between acquisition decision events. This authority extends to assisting programs in developing acquisition strategies where appropriate. The manual included a section that specifically called out Agile contracting considerations that pointed back to Office of Management and Budget Contracting Guidance to Support Modular Development, the TechFAR handbook, the Digital Services playbook, and innovative contracting case studies.

Among other useful information in the Agile instruction manual were key contracting considerations for an Agile program or project manager. These considerations included, among other things, frequent, iterative deliveries of software, an ability to monitor changes to maintain contract and project scope, flexibility to accommodate refinement of requirements, transparency and collaboration, and prior experience in the Agile methodology. The manual also highlighted goals for the acquisition to discuss with a contracting officer, such as rapid contracting processes to keep pace with Agile development, contracting to accommodate incompletely defined scope and requirements, and the ability to respond to requirements changes without requiring extensive change orders.

According to officials within OCTO and the Office of the Chief Procurement Officer, the department also supported Agile programs in preparing acquisition strategies through the IT acquisition review process. This process was established to provide a mechanism for the DHS Chief Information Officer to review and guide agency IT expenditures. The process was intended to analyze IT acquisitions to ensure alignment with DHS missions, goals, policies, and guidelines. This process relied on subject matter experts to assist in the review of IT acquisitions, including one for Agile reviews. According to the IT Acquisition Review Essentials Guide, Agile reviews occurred where software was being developed to ensure development activities adhered to Agile best practices and DHS SELC guidance. The Agile subject matter expert was expected to review acquisition materials against an established set of criteria for both the acquisition plan and the requirements document. For example, when reviewing the acquisition plan for approval, the subject matter expert should consider if the statement of need adequately addresses Agile or iterative project-specific activities and/or deliverables.

The Director of STM stated that there is one staff member in STM who actively participates in the IT acquisition review process and was responsible for ensuring Agile language was correctly implemented in contract statements of work. The Director also added that they were willing to help teams that were having trouble providing explanations of Agile processes in their statements of work. The Director of STM stated that there was no policy to guide his staff member reviewing Agile language in the statement of work, but that he asked his division to put together a checklist review to govern this process. The Director added that the department sent programs and projects requiring assistance with Agile contracting to the Procurement Innovation Lab by request to streamline the acquisition plan.

According to the Leader of the Procurement Innovation Lab Team, the Office of the Chief Procurement Officer was primarily focused on supporting Agile pilot programs, such as the Federal Emergency Management Agency Grants Management Modernization program. The team leader noted that, while the procurement office supported these programs, it relied on the program offices to ensure accuracy. For example, the program management office ensures that the requirements are structured and delivered, which could be challenging for Agile programs. The team leader mentioned that a particular focus at the moment was defining the pricing for contract line item numbers in such a way as to afford the flexibility needed for Agile development while still holding contractors accountable.

Appendix IV: Leading Practices for Adopting Agile Development—Program Processes

This appendix describes in greater detail our evaluation of the three leading practices for program processes when adopting Agile development. It does not present new findings; rather, the information is intended to assist the Department of Homeland Security (DHS) in implementing the recommendations described in this report.

Program processes refer to leading practices related to the program office and technical environment. For programs to successfully transition from processes used for traditional development projects, programs should ensure that staff are appropriately trained in Agile methods by ensuring Agile teams have the appropriate technical expertise needed to perform their roles technical environments enable Agile development through making technical and project support tools available, and designing a system that supports iterative delivery project planning controls are compatible with Agile methods by maintaining a sustainable development pace and tracking and defining and incorporating non-functional requirements in defining and incorporating critical features in development The department develops an environment that supports these processes. Within DHS, program management offices are responsible for planning and executing individual programs and implementing applicable Agile methodologies. In addition, the DHS Office of the Chief Information Officer (OCIO) is responsible for setting policies and procedures to ensure that programs leverage Agile development best practices to meet the department’s goals and are within acquisition policy. The DHS OCIO is also responsible for providing guidance for and reviewing the adoption and execution of Agile development.

Staff are appropriately trained in Agile methods

Train all program staff in Agile methods The agency should provide a training program in Agile for staff and track and monitor the training. All members supporting the team, not only the software development team, should be trained in the specific Agile framework they will be using.

DHS required its acquisitions workforce to take training that incorporated Agile methods. DHS Instruction 102-01-006, Acquisition Program Management Staffing, established certifications for key acquisition career fields, which included training requirements. According to the Associate Director for Training from the Homeland Security Acquisitions Institute, the certification requirements included training that has been updated to incorporate Agile methods. Specifically, the department updated course content for AQN 101: DHS Fundamentals of Systems Acquisition to include Agile development concepts, such as small team management and Agile metrics, following the issuance of department policy governing Agile development. This course was required training for seven of the acquisition career fields, including program and project managers, systems engineers, and test and evaluation managers.

DHS tracked and monitored the completion of training requirements for the acquisitions workforce. According to DHS Directive 064-04, Acquisition Professional Career Information, component acquisition executives were responsible for ensuring that acquisition personnel met the mandatory training requirements. Officials from the Homeland Security Acquisitions Institute within the Office of the Chief Procurement Officer stated that DHS employees leveraged the Federal Acquisition Institute’s training application system to track their training and certifications. According to the catalog of product services of the institute, members of the DHS acquisition workforce were required to attach copies of their training certificates to request certification of completion of the required training.

Because the DHS acquisitions workforce may not cover all personnel staffed to Agile projects, some program staff may not be subject to training requirements that incorporate Agile methods. According to the Director of the Homeland Security Acquisition Institute, certain Agile team members, such as the product owner, were not necessarily classified as part of the acquisitions workforce. For example, according to the U.S. Immigration and Customs Enforcement (ICE) Student and Exchange Visitor Information System (SEVIS) program staffing plan, the product owner role was not part of the acquisitions workforce and did not require any certifications.

To help address the Agile training needs of all staff, including those who are not part of the acquisitions workforce, DHS also provided elective training in Agile methods. The department offered commercial training through the Homeland Security Acquisition Institute, such as acquisition of Agile services and Agile requirements for creating user stories. The DHS Agile instruction manual also identified training offered by the U.S. Citizenship and Immigration Services Office of Information and Technology as another resource on Agile concepts, such as user stories and automated testing.

In addition to elective training, the Agile Development and Delivery for Information Technology Instruction Manual (Agile instruction manual) encouraged program managers to seek out an Agile coach to help teams adopt Agile methods and supplement training. The instruction manual suggested that program managers should identify an Agile coach to serve as an embedded trainer, consultant, and team advisor. This Agile coach could help the team adapt Agile methods to their environment and work through challenges. An Agile coach could also help individual team members understand the responsibilities of their role on an Agile team.

Although DHS did not provide coaches for Agile teams, the department offered resources that could help programs select and obtain an Agile coach. First, the department established a blanket purchase agreement for programs to acquire Agile development support in the form of hands- on coaching services for the design and use of Agile methods. According to Homeland Security Acquisition Institute officials, this agreement would enable programs and projects to acquire Agile coaching. Among other things, this agreement defined the scope of Agile coaching services and their pricing so that programs would not need to develop these terms on their own. Second, the Agile instruction manual included considerations to help program managers select a qualified Agile coach. For example, the instruction manual encouraged program managers to collaborate with contracting officials to identify an Agile coach who had demonstrated successful past performance on projects implementing similar technology and Agile methodologies.

The U.S. Customs and Border Protection’s (CBP) Biometric Entry Exit (BEE) program’s Air Exit project provided informal training for new team members that included a discussion of Agile methods. According to the Air Exit project manager in the Office of Information and Technology, new team members received onboarding training that covered CBP’s approach to Agile methods. The project did not track attendance for this onboarding training, but an Air Exit project manager noted that team members were incentivized to attend the training in order to learn how to satisfy their responsibilities. The Scrum master for the Air Exit project stated that this training was also available to the team as a refresher course approximately every fiscal quarter.

The BEE program also relied on an Agile coach to support the Agile team. According to the Agile coach supporting the Air Exit project, this role included training for the Agile team on basic Agile topics and working with the team on their use of a project management software tool. According to a project manager within the Office of Field Operations Air Exit project management office, the Agile coach that supported the project was instrumental in designing the CBP Office of Information and Technology’s Agile development program beyond the BEE program.

Ensure Agile teams have the appropriate technical expertise needed to perform their roles The agency should have policy or guidance in place to help programs ensure Agile teams have the appropriate technical expertise. A program should also consider Agile-centric skills when forming teams. In addition, programs should define requirements for contractor proposals and evaluate contractor proposals for Agile services (e.g., source selection).

DHS guidance provided programs with considerations for forming teams with Agile-centric skills. The DHS Agile instruction manual stated that a development team with experience in Agile practices can mitigate risks to on-time delivery. This experience included Agile processes as well as technical skills, such as automated testing. In the context of the Agile Scrum methodology in particular, the Agile instruction manual stated that teams needed to be cross-functional and have all of the skills required to deliver a project from conception to delivery.

To enable teams to deliver a project from conception to delivery, the Agile instruction manual stated that program managers should seek team members with general skills. The manual advised that team members should contribute to routine development activities and possess cross- functional expertise that allows the team to achieve work without depending on individuals outside of the team. For example, in Agile development, testers are part of the development team and should therefore possess both testing and development skills. In addition, the instruction manual stated that, according to industry experts, program managers should seek some overlap in team member’s skillsets to mitigate risks associated with a key person becoming temporarily unavailable.

DHS guidance further provided programs with considerations for defining requirements in solicitations for contract proposals for Agile services. For example, DHS supplemental guidance for incorporating testing and evaluation into contract requirements noted that contracts should specify government test and evaluation staff, as well as contractors, on the development team in order to access the test data they need.

The DHS IT acquisition review process also helped to ensure that requirements were defined in solicitations for contractor proposals. According to the Information Technology Acquisition Review Essentials Guide, Agile subject matter experts in the department review proposed contracts to ensure that they will enable development activities that adhere to Agile best practices and DHS systems engineering life cycle (SELC) guidance. For example, Agile subject matter experts should assess whether contract requirements documents, such as the statement of work, are prepared in terms that will enable vendors to clearly understand the Agile requirements.

The department also provided guidance to assist programs in evaluating contractor proposals for Agile services. The Agile instruction manual noted that programs can consider certifications in various Agile methodologies and recommended that programs coordinate with contracting officials to review vendors’ past performance in implementing Agile methods.

In addition, the department established the Procurement Innovation Lab within the Office of the Chief Procurement Officer to help programs address challenges in procuring Agile services, such as validating contractor qualifications. According to a Procurement Innovation Lab team leader, the lab shares lessons learned from Agile services contracts via webinars, which are available to staff on an as-needed basis. Several of these webinars highlighted the value of using technical demonstrations to validate the qualifications of vendors.

The ICE SEVIS program provided training for all team members, including contractors, to ensure they had the necessary Agile-centric skills and expertise. A team process agreement for one development module showed that the technical lead, development team, test engineer, and Scrum master roles were filled by contractors, while other positions such as the project manager, product owner, and test automation subject matter expert roles were filled by government employees. According to the ICE SEVIS program manager and Scrum master, the program provided training for contractors that covered Agile processes as well as technical and project management support tools. In addition, some government employees took role-specific training. For example, the program’s test automation subject matter expert completed training in continuous integration and test automation.

To further ensure contractors on ICE SEVIS Agile teams had the necessary Agile-centric skills, the ICE SEVIS program defined the Agile methodology and necessary technical expertise for contractors in the contract requirements. For example, the performance work statement for one development module required contractors to use the program’s management software tool to track user stories. The performance work statement also required use of the program’s continuous integration and automated testing tools. The terms and conditions for this contract also identified the required experience for key personnel, such as proven experience working in an Agile environment.

The ICE SEVIS program also evaluated contractor qualifications to ensure they had the necessary technical expertise. According to the program manager, contractor qualifications were evaluated in two stages; first, by assessing the contractor’s proposal, and second, by conducting a technical challenge to ensure that contractors could demonstrate the technical skills in the proposal. According to the instructions included in the request for contractor proposals, this technical challenge required the contractor to leverage Agile best practices to design, develop, and demonstrate working software that addressed user stories provided by the program. Although the instructions stated that contractors were required to follow Agile methods, the ICE SEVIS program manager stated that the primary goal of the technical challenge was to assess development skills rather than knowledge of Agile.

Technical environments enable Agile development

Agency policy or guidance should call for technical and project tools to be available to support Agile development and for system design that will support iterative delivery.

Make technical and project support tools available Project management and technical support tools should be integrated into a program’s technical environment, where appropriate. The tools within this technical environment should be readily available to Agile teams.

DHS policy and guidance called for Agile projects’ technical environments to support Agile methods. The department published guidance for standing up technical environments specifically for Agile projects. For example, the DHS Agile instruction manual identified the benefits of using program support tools for tracking program progress, reporting on that progress as part of program governance, and automating tests within an Agile technical environment. The manual stated that a program or project manager is responsible for fostering an environment that enables the Agile team to succeed, including obtaining the appropriate tools.

To supplement this guidance, DHS offered a suite of tools that Agile programs could access. The suite of tools was referenced in a checklist of activities for program or project managers in the Agile instruction manual. According to an IT specialist from the Technical Architecture and Engineering division within the Office of the Chief Technology Officer (OCTO), the tools available included program management tools as well as technical tools. The specialist stated that OCTO provided programs with access to this suite of tools to build support for and familiarity with the tools, evaluating any requested plug-ins from programs and doing their best to accommodate them.

The ICE SEVIS program defined the technical environment to include technical tools for automated testing and continuous integration. The team process agreement for one of the program’s development modules identified technical tools that supported continuous integration and testing within the program’s technical environment. This included Jenkins for continuous integration as well as MUnit and Soap UI for continuous testing. In addition, the ICE SEVIS Modernization Test and Evaluation Master Plan discussed that tools for helping to ensure code quality, such as an automated code analytics tool, should be used to identify test coverage of code and cybersecurity code vulnerabilities.

The program also defined management support tools in the process agreement. Specifically, it identified support tools for tracking and knowledge management, such as JIRA and Confluence. The team process agreement stated that JIRA should be the main knowledge management tool and that all changes, discussion, and history should be tracked in each ticket. This process agreement also stated that JIRA should be the team’s tracking tool with Confluence used to provide transparency.

Design a system that supports iterative delivery The agency should adopt policy or guidance that allows project designs to develop modular system components and the program should establish a loosely coupled architecture that allows for modular development.

DHS guidance allowed project designs to develop modular system components through upfront architecture planning. The DHS Technical Review Guide advises stakeholders to discuss and approve the technical design of the system, including its top-level architecture, as part of the system definition review. This review should take place prior to development work.

For Agile programs, DHS suggested that programs may elect to switch the system definition review with a release planning review. The SELC Tailoring Examples for Selected Types of DHS Acquisition Programs specified that this design discussion should take place as a part of release planning. The department referred to this design as an “architectural runway”, a description that should enable the team to conceptualize how the user stories will be implemented. In exiting the release planning review, the Technical Review Guide noted that programs should answer whether or not an architecture exists, if the architecture enables the deployment of the release, if architecture collaboration is explained and understood for this development process, and if the appropriate resources are available.

In addition to transitioning to a release planning review, DHS guidance urged Agile programs to move away from traditional artifacts associated with a system definition review. In this shift from traditional artifacts, the department proposed that programs document software design within a system design document on a release-by-release basis. According to the Requirements Engineering Users Guide, in Agile methodologies detailed design occurs at the iteration level and, as such, the design is documented in an iterative fashion in the system design document. The guide further stated that the system design document allows the development team to communicate the design to others including customers, managers, and other developers and that industry best practice was to represent the design through a series of “design views.” Each software design stakeholder could have a distinct perspective on what are the essential aspects of a software design. Together, these views provide a comprehensive description of the design in a concise and usable form that simplifies information access and assimilation.

DHS guidance did not discuss the system design document as a delivered artifact until after the sprint review and demo and a release readiness review had been discussed. At the end of each iteration, DHS guidance stated that the system design document should represent the design of the feature, function, and/or system as it existed at that moment. To facilitate communication between Agile teams and to ensure the most up-to-date description of the design is available, guidance called for the system design document to be developed and maintained in an electronic form using any number of programs or web tools that are available. The Requirements Engineering Users Guide noted that the system design document is to be considered complete when each identified design concern is the topic of at least one design view, all design constraints have been applied, and sufficient detail exists to be an authoritative and primary “code-to” artifact.

The system design document should also provide traceability to the feature, epic, and operational requirements document “shall” statements. The SELC Tailoring Examples for Selected Types of DHS Acquisition Programs stated that, prior to releasing software to the production environment, a release readiness review should be conducted. As part of this guidance, the department stated that the intent of this release readiness review included ensuring that all elements of the release were complete, including a system design document.

DHS guidance also discussed designing a loosely coupled architecture, another important aspect of project design that facilitates modular development. A member of the contractor support staff for the DHS OCIO stated that the Enterprise Architecture Team was expected to consider modularity and loose coupling generally through consideration of technical complexity. According to DHS Enterprise Architecture principles, technical complexity is to be mitigated in part by the implementation of loose coupling. According to the principles, DHS will incorporate loose coupling into architecture and systems design to minimize the risk resulting from changes within one system necessitating changes within an interoperable system.

The BEE Air Exit project design document defined the planned design for the system and addressed design and architectural concerns that could affect the system’s operating environment.

As part of this design consideration, the project established a loosely coupled architecture. This loosely coupled architecture was illustrated within the project’s system design document. This system design document defined the Traveler Verification Services software as consisting of two distinct components: 1) traveler verification services core and 2) traveler verification services matcher. The functionality and responsibility of these two components were distinguished throughout the document. Moreover, the document detailed how the Traveler Verification Services software would be delivered as a system of applications, combining an integration layer, business layer, data access layer, and data layer.

Project planning controls are compatible with Agile development

Agency policy or guidance should call for teams to maintain a sustainable development pace and track and monitor that pace and for non-functional requirements and critical features to be defined and incorporated in development.

Maintain a sustainable development pace and track and monitor that development pace The agency should have policy or guidance that calls for Agile projects to establish a sustainable development pace. This guidance should be supplemented by tracking and monitoring the pace. The program should establish a sustainable pace for Agile projects and that pace should be tracked and monitored.

DHS guidance called for Agile projects to manage the pace of the software development. The Agile instruction manual stated that Agile projects should consider velocity and burndown rates to track the overall project status and update the project plan to reflect this status. In a separate appendix, the Agile instruction manual also identified metrics for project and program managers and executives to consider in order to monitor how a project was progressing, how Agile was optimizing the use of team members and resources, and where the project stood in terms of key Agile measures. In the list of Agile metrics, DHS highlighted burndown rate and velocity, and offered a description and method of calculation for each.

In addition to the Agile instruction manual, the department provided training that spoke to development team pace. For example, the curriculum for lesson six of course APM 350 on managing program execution included a section covering Agile development metrics. Among the metrics discussed were those associated with progress, including velocity and burndown charts. Progress metrics were also covered in other course offerings. However, DHS guidance and training materials did not cover the concept of ensuring a sustainable pace.

In order to track and monitor the development team’s pace, the department incorporated several related measures into the Agile core metrics. Among others, programs executing Agile were expected to report on the following pace-related metrics after each iteration: story points planned to be completed, number of production deployment per quarter, and average product deployment lead time.

These measures could provide programs and the department with an understanding of the development team’s pace and the extent to which it was or was not sustainable.

However, the department was not tracking and monitoring development team pace as intended. The Agile instruction required Agile programs to submit Agile core metrics within six months of the instruction’s publication. However, according to the Director of STM, programs were not consistently reporting these core metrics. According to the Director of STM, the department was still working with programs to ensure they consistently reported the core metrics to the Investment Evaluation, Submission, & Tracking system.

The SeaWatch project at the United States Coast Guard (USCG) demonstrated that it was monitoring development pace on a monthly basis. SeaWatch officials stated that they used TAIGA as a tool to manage the overall project and to auto-calculate pace. Additionally, SeaWatch officials stated that contractors delivered a monthly progress report, which contained the accomplishments of each team and a snapshot from the latest TAIGA report. For example, one monthly report for SeaWatch included a burndown chart for the SeaWatch project’s development backlog and the monthly output of user stories and associated story points by development effort that could be used to assess development pace over time.

SeaWatch officials stated that the teams used velocity to help plan for the next iteration. Officials added that they tracked the collective velocity of all four teams as they were all working together on the same ship build. In the future, officials stated that this tracking of velocity could also be used to track individual team velocities as necessary.

The project demonstrated that it was adapting in order to achieve a sustainable pace. According to the April 2018 monthly report, the team completed 55 user stories worth 500 story points. The following month, in the May 2018 monthly report, the number of user stories dropped from 55 user stories to 17, worth 130 story points. According to the June 2018 monthly report, the team completed a development effort of 31 user stories and 278 story points.

According to the SeaWatch acquisition manager, development pace fluctuated because not all sprints were of equal difficulty. The acquisition manager added that the number of completed story points per sprint could also be inconsistent due to inaccurate user story estimates, changes in staff availability from sprint to sprint, and other external factors such as weather.

Define and incorporate non-functional requirements in development The agency should have policy or guidance in place for incorporating non-functional requirements for Agile projects and the program should account for non-functional requirements, such as security and privacy, in the program strategy and throughout development.

DHS guidance addressed the incorporation of non-functional requirements for Agile projects. According to the Technical Review Guide, non-functional requirements could be governed via a system definition review. According to the guide, this review was required at the end of the requirements definition phase to focus on the completeness of the requirements engineering activities, including the gathering, analysis, and documentation of functional and non-functional requirements. This review assessed the traceability of these requirements to the operational requirements document and concept of operations.

In the case of Agile programs, DHS suggested replacing the system definition review with a release planning review. In place of traditional artifacts associated with a system definition review, DHS guidance stated that the capabilities and constraints document, backlogs, and the system design document, which are developed iteratively throughout the release, should document the requirements and provide traceability to the operational requirements document. These artifacts served the function and filled in for the functional requirements document and the system requirements document previously required for a system definition review.

The Technical Review Guide noted that, as the capabilities and constraints document matures, business and architectural epics should decompose to features or themes, and, ultimately, user stories that reflect the specific tasks that users will perform. The Technical Review Guide cited as exit criteria that a program or project should answer whether the capabilities and constraints document identified the specific features and non-functional requirements to be addressed in the release.

DHS requirements engineering guidance expanded on how Agile programs and projects could manage non-functional requirements. The guidance explained that there were various ways that the constraints or non-functional requirements such as security, Section 508 accessibility, privacy, or reliability could be translated down to the iteration level. It stated that some Agile teams may include these non-functional requirements in the backlog, while other teams may include them as part of acceptance criteria or in an artifact called the “definition of done”. According to officials from the Science and Technology Directorate Office of Systems Engineering, once defined, the day-to-day operations and testing for non-functional requirements were the responsibility of the operational test agent.

DHS maintained some governance over non-functional requirements. According to the DHS acquisition management instruction, the operational requirements document should be approved by the Acquisition Decision Authority after validation by the Joint Requirements Council. The operational requirements document should include both the functional and non-functional requirements. Officials from the Office of the Director of Test and Evaluation said that they do not usually provide feedback on the decomposed functional or technical requirements for software development projects, focusing only on the operating requirements, because that is what directly impacts operations.

The CBP BEE program’s functional requirements document outlined a series of non-functional requirements as the requirements used to define how the system is to behave as opposed to functional requirements that define what the system should do. The project included 10 non-functional requirements in the functional requirements document. For example, the biometric match service should have an overall availability of greater than or equal to 99%, which included both scheduled and unscheduled downtime. These ten non-functional requirements comprised five related to availability, three related to reliability, one related to scalability, and one related to security. All of these non-functional requirements were scheduled for release as part of the initial operating capability.

CBP officials noted that non-functional requirements were also captured within the operational requirements document as measures of effectiveness. According to project officials, measures of effectiveness and other security-related parameters translated into the key performance parameters for the project. Officials noted that these key performance parameters were tracked on a daily basis and that information was fed into a monthly report. The operational requirements document stated that the program’s suitability requirements conformed to the DHS and CBP enterprise architectures and all DHS and CBP infrastructure policies and guidelines. Moreover, it noted that National Institute for Standards and Technology guidance and DHS guidance factored into the development of security related non-functional requirements. For example, system security controls should be compliant with National Institute of Standards and Technology and DHS sensitive system guidelines based on its Federal Information Processing Standard 199 rating for availability, integrity, and confidentiality.

Define and incorporate critical features in development The agency should have policy or guidance in place for incorporating critical features for Agile projects. The program should ensure that its strategy considers all mission, architectural, and critical safety components, along with their dependencies, on a regular basis.

DHS policy and guidance addressed the incorporation of critical features for Agile projects. As discussed in the non-functional requirements section, programs were expected to document functional requirements via the systems design review or, as recommended for Agile programs, a release planning review. Artifacts associated with these reviews served to capture the functional requirements for the program and should be evaluated as part of the entrance and exit criteria defined in the technical review guide. Additional guidance elaborated on the process for decomposing requirements.

Unlike non-functional requirements, applicable exit criteria on critical features expanded into the solution engineering review. This criteria included questions devoted to critical features and how they tied back to performance measures (e.g. key performance parameters). According to the Director of STM, headquarters oversight of critical features was limited to the higher-level requirements defined in the operational requirements and concept of operations documents.

The ICE SEVIS program captured critical features in documents required by department acquisition management policy and guidance. The ICE SEVIS Modernization Concept of Operations listed specific functional capabilities associated with mission and mission support scenarios.

The ICE SEVIS Modernization Operational Requirements document expanded on these functional capabilities and identified the operational and program-level requirements. These requirements were necessary to achieve the performance goals and mission of the Student and Exchange Visitor Program and the Department of State, the primary sponsors for the program. In particular, the SEVIS Modernization Operational Requirements document identified business capabilities and key performance parameters that measured system capabilities.

The core capabilities are long-term initiatives intended to span multiple contracts and deliver the major components necessary for SEVIS modernization. The SEVIS Modernization Operational Requirements document stated that these capabilities must be present for the SEVIS modernization to be considered a success. These business capabilities represented the core SEVIS functions needed to close outstanding SEVIS vulnerabilities. According to the ICE SEVIS Modernization SELC Tailoring Plan, there were 79 sub-capabilities supporting the eight core capabilities. The sub-capabilities generally fulfilled one or more stakeholder needs and were delivered within a release or series of releases. The SEVIS Modernization Operational Requirements document confirmed that the program should prioritize and sequence the capabilities for delivery during the release planning and delivery processes.

The program provided a road map for one development module. This road map listed areas for development in the order they were intended to be developed and identified the associated business capabilities. The business capabilities identified in the road map aligned with the sub- capabilities listed in the SEVIS Modernization Operational Requirements document. Examples of business capabilities in the road map that were also sub-capabilities identified in the operational requirements document included: create nonimmigrant record (including supporting forms), align nonimmigrant eligibility information with unique nonimmigrant, update nonimmigrant biographical information, and add/update dependent information.

Appendix V: Leading Practices for Adopting Agile Development—Team Activities and Dynamics

This appendix describes in more detail our evaluation of the three leading practices for team activities and dynamics when adopting Agile development. It does not present new findings; rather, the information is intended to assist the Department of Homeland Security (DHS) in implementing the recommendations described in this report.

For teams to successfully transition from processes using traditional software development methods to Agile methods, leading practices for team activities and dynamics recommend that the composition of the team supports Agile methods by defining the role of a product owner work is prioritized to maximize value for the customer through creating user stories to define work prioritizing requirements in a backlog based on value estimating the relative complexity of user stories repeatable processes are in place by meeting daily to review progress and discuss impediments ensuring the quality of code being developed Within DHS, program management offices are responsible for planning and executing individual programs and implementing applicable Agile methodologies. According to Office of the Chief Technology Officer (OCTO) officials, DHS contracts for Agile services, including development, rather than performing development in-house. As a result, Agile teams may be predominantly contractors rather than federal employees. In addition, DHS Office of the Chief Information Officer (OCIO) is responsible for setting the policies and procedures to ensure that programs and, in turn, the teams that make up those programs, leverage Agile development best practices to meet the department’s goals and are within acquisition policy. DHS OCIO is also responsible for providing guidance for and reviewing the adoption and execution of Agile development.

Team composition supports Agile methods

Agency policy or guidance should require individual, self-organizing Agile teams for each segment or iteration and define the role and responsibilities of the product owner.

Agile teams should be self-organizing, meaning they are empowered to collectively control how to accomplish their work and the resulting product. An Agile team’s authority should include lower-level decision making and team formation and highlight the importance of team stability. The team’s composition should be cross-functional and consist of members who possess all the skills needed to produce working software, including, but not limited to, contract specialists, developers, and testers.

DHS provided guidance to Agile teams on self-governance. The Agile Development and Delivery for Information Technology instruction (Agile instruction) and the Agile Development and Delivery for Information Technology Instruction Manual (Agile instruction manual) both explain that collaborative, self-organizing, and cross-functional teams help achieve the flexibility needed for the iterative development that characterizes Agile development methods. The Agile instruction manual notes that most Agile methodologies assume the dedicated involvement of all stakeholder, developer, and integration staff throughout the project.

DHS guidance also discusses team formation. The Agile instruction manual recommends that the project team include the roles of the program or project manager, a product owner, a development team of approximately five to nine members, testers, and an Agile coach, and any additional expertise as needed. According to DHS guidance, a program or project manager is responsible for establishing the project team. The program or project manager is supported in this by the component acquisition executive and other component management.

At DHS, U.S. Immigration and Customs Enforcement’s (ICE) Student and Exchange Visitor Information System (SEVIS) program had self-organizing teams that defined their own processes for completing work. ICE Agile teams, including those supporting the SEVIS program, were expected to document their processes in a team process agreement, where a team had the authority to define its own operational strategy and make decisions about the product, including when to consider the product completed according to the program’s “definition of done.” According to the ICE Agile principles instruction, a program chooses a baseline set of practices that are documented in a team process agreement and are adjusted over time.

ICE SEVIS teams were self-managing and included the roles necessary to deliver what they committed to in a sprint. ICE’s Agile playbook suggested minimum levels of experience, knowledge, and certifications necessary for key personnel to support Agile methodologies. For instance, the playbook suggests that Scrum masters be certified and have a minimum of one year of experience. To help ensure that contractors have the requisite skills necessary, ICE SEVIS officials stated that vendors are required to demonstrate their ability to develop a small software application before a contract is awarded to them.

Define the role of a product owner A product owner should understand the business and strategic values of the agency and its alignment with the vision of the product team and support Agile methods. A product owner’s responsibilities include availability to the team, authority for making programmatic decisions, general responsibilities as a member of the team, and the need to possess subject matter expertise related to the business needs. A product owner is an authoritative user who manages the requirements prioritization, communicates operational concepts, and provides continual feedback to the team.

DHS provided guidance on the role and responsibilities of a product owner. According to the Agile instruction manual, the product owner is responsible for representing stakeholders. To do so, the product owner should be available to the development team throughout the iteration to answer questions and clarify requirements on behalf of the stakeholders. The manual stated that the product owner is also responsible for ensuring that the product meets user needs and delivers value. This includes, for example, prioritizing user stories in the backlog and serving as an acceptance authority for work completed by the team.

The department also provided elective training on the role of a product owner. For example, the U.S. Citizenship and Immigration Services Office of Information Technology offered an elective product owner training course. The USCIS product owner training covered concepts such as the importance of the product owner’s availability to the team and the product owner’s authority for making programmatic decisions.

ICE identified a product owner for SEVIS to represent two user communities. The program identified one product owner from ICE’s Student and Exchange Visitor Program and a second product owner from a stakeholder organization within the Department of State. Both product owners were identified in the ICE SEVIS staffing plan.

According to a team process agreement for one development module, a product owner is responsible for, among other things:

Prioritizing and deciding which user stories will be implemented in each iteration.

Making an acceptance decision for each user story based on the story’s acceptance criteria.

Ensuring that the intended value of the functionality is delivered.

According to program officials, product owners for the ICE SEVIS program prioritized user stories during planning sessions. The Student and Exchange Visitor Program Agile Overview slides stated that the team, including the product owner, attends sprint planning to review the prioritized product backlog and to ensure a common understanding of the product owner’s immediate priorities.

Product owners also exercised authority to validate acceptance criteria and subsequently close user stories. The program’s “definition of done” stated that the product owner must test and indicate acceptance of each user story in order for a user story to be considered complete. In a written response, ICE SEVIS officials stated that ICE SEVIS product owners indicated a user story had met the acceptance criteria and could be closed by changing the user story’s status to “closed” using the team’s program management software tool.

In addition, product owners were available to the development team to ensure timely input. According to a team process agreement for a development module, the product owner should work closely with the development team to communicate the details of requirements and answer questions about user stories. In an interview, the ICE SEVIS product owner representing the Student and Exchange Visitor Program stated that the role was a full-time position and did not have any competing responsibilities. To ensure availability, the ICE SEVIS product owner representing the Student and Exchange Visitor Program stated that there was a designated backup who had the same authority and responsibilities as the full-time product owner.

Work is prioritized to maximize value for the customer

Agency policy or guidance should call for Agile teams to create user stories to define the work; prioritize requirements in a backlog based on value, including tracking and monitoring the value of work accomplished; and estimate the relative complexity of user stories. Individual Agile teams within the respective programs and projects should implement these aspects of Agile development.

Create user stories to define work A user story is to reflect a small segment of work that can be completed in a single iteration. The agency should have policy or guidance in place for writing user stories for Agile projects. The product owner should determine the value of a user story in consultation with the development team, including the acceptance criteria and defining what “done” means. User story value should then be re-evaluated based on requirements to ensure the greatest return on investment.

DHS provided guidance that Agile programs and projects could leverage when writing a user story. The Agile instruction manual, Homeland Security Acquisition Institute Agile lessons, such as “Managing Program Execution” and the Requirements Engineering User’s Guide provided a basic format for how to craft a user story. These resources noted that a user story defines where a “role” wants some “goal/desire” accomplished to result in a “benefit”.

The Requirements Engineering User’s Guide also discusses the role of acceptance criteria and a definition of done in user story development. The guide highlighted that acceptance criteria defines the boundaries of a user story and confirms when a story has been completed and is working as intended. It specifies that acceptance criteria should be included in an Agile program or project’s capabilities and constraints document, a DHS artifact unique to Agile development and highlighted in the systems engineering life cycle (SELC) tailoring example for Agile. This guide added that the definition of done identifies all of the activities/artifacts besides working code that must be completed for a feature or sub-epic to be ready for deployment or release including testing, documentation, training material development, certifications, etc.

The Agile instruction manual places much of the responsibility for defining a user story under the purview of the product owner. The Agile instruction manual stated that the product owner is the individual tasked with providing requirements to the development team and is responsible for determining the features necessary for the product release. The manual also emphasized that the product owner is only responsible for clarifying the user story requirements that would meet his or her needs and not responsible for clarifying how user stories should be implemented to meet those needs.

The ICE SEVIS program developed user stories based on business capabilities and other requirements as determined by the product owner and the business stakeholders. The SEVIS Modernization Operational Requirements Document describes eight business capabilities that represent core SEVIS functions. According to ICE SEVIS officials, these business capabilities are addressed through user stories, so there is traceability in the backlog from user stories to epics to business capabilities/operating requirements. The team’s process agreement for one development module—Information Sharing—assigned responsibility for writing user stories to the product owner. This agreement also noted that acceptance criteria would be required for most stories.

User stories for the program were managed through a program management software tool. An output of the backlog from the program management software tool for one development module—Managing Nonimmigrant Information—contained 525 user stories. These user stories generally followed DHS and ICE guidance for capturing what a user needs and why. Most of these user stories also included acceptance criteria.

The program also developed a “definition of done” for all user stories in the team process agreement. According to the definition, a user story was “done” when the following steps were addressed:

All code to meet the story’s needs was written according to the system’s development standards.

Unit tests were written and run successfully.

All code was checked in and the build completed successfully.

All database changes (if required) were complete and checked in (a functional test could be run).

The software had been deployed to the system test environment and passed system tests.

The product owner agreed that the implementation met the acceptance criteria written in the story as appropriate.

All documentation required to support the story was completed (test cases, interface updates, etc.)

Prioritize requirements in a backlog based on value Agile teams should pull work from a prioritized backlog and provide frequent deliveries of software with immediate value to the user. The team should determine the value of the user stories, prioritize work in a product backlog, and provide an ongoing assessment of value expected versus value delivered. The value of the work accomplished by Agile projects should be tracked and monitored.

DHS guidance called for prioritizing user stories in a backlog. The department published an example of a SELC tailoring plan for Agile development that encouraged programs and projects to prioritize user stories in a backlog as part of each release. To ensure that programs or projects took these steps, the Technical Review Guide exit criteria for the release planning review asks if programs or projects will have a process in place for prioritizing user stories prior to the development of features for each release.

Planning sessions were one such process that programs and projects could use to prioritize user stories in the backlog. The DHS Agile instruction manual stated that, during sprint planning, the product owner meets with the development team in order to identify user stories from the backlog that should be prioritized for the upcoming sprint and that prioritization decisions should be made based on value to the users. In addition, the product owner should ensure that prioritization decisions maximize mission values. The Requirements Engineering User’s Guide also states that requirements should be prioritized based on continuous stakeholder input so that programs can prioritize what users need the most.

DHS guidance also discussed how to determine the value of individual user stories. While the Director of STM said that the product owner is responsible for interpreting the concept of value as it applies to a user story and the relative prioritization of the backlog, Agile Requirements and Road Mapping Guidance for DHS includes a discussion on how a program can sequence its road map for learning, risk, and economic value. In this section, DHS offers models to consider to assist in user story prioritization decisions and considerations for the product owner, such as seeking to balance between business value and cost. The Director added that there were venues, such as Agile “chat and chews,” where program staff could ask questions and receive informal guidance.

DHS modified acquisition procedures to allow for an ongoing assessment of progress, and indirectly the value of work accomplished, via the release road map. DHS guidance stated that the release road map is submitted to the Acquisition Review Board prior to acquisition decision event 2B, as required by the Agile instruction. The Technical Review Guide exit criteria for the release planning review and the release readiness review asked if the development team was following the release road map and making adjustments that supported the successful completion requirements defined at acquisition decision event 2B. Thereafter, programs submitted a road map to the Acquisition Review Board during regular program reviews.

In addition to tracking and monitoring the value of work accomplished against a release road map, regular Acquisition Review Board program reviews allowed for the assessment of value expected versus value delivered. The presentation template for Acquisition Review Board program reviews included a slide for programs to report their progress toward planned features. For each review, programs identified a percentage of each capability that they planned to complete by the next review. In addition, programs reported on the percentage of each capability that they had completed since the last review.

The U.S. Coast Guard (USCG) SeaWatch Agile teams prioritized requirements in a backlog based on the team’s ability to complete them within a sprint. According to the acquisition manager for the Command, Control, Communications, and Computers, Intelligence, Surveillance and Reconnaissance (C4ISR) program, the SeaWatch product owner for new development determined priorities for new requirements with stakeholders. The product owner then defined those requirements as an epic or as a user story. The C4ISR acquisition manager stated that the user stories were prioritized in the backlog during sprint planning primarily based on whether the Agile team could complete the work in the upcoming sprint rather than on the value assigned by the stakeholders.

According to SeaWatch officials, user stories that could not be completed during the current sprint were marked as a priority item for the next sprint.

Although the SeaWatch program assessed value to the user for some epics, this did affect how the epic or its associated user stories were prioritized in the backlog. The C4ISR acquisition manager stated that SeaWatch assigned a value (e.g. extra large, large, or medium) to an epic based on the epic’s value to the user. However, the acquisition manager noted that user stories were not typically prioritized by the value of the associated epic. User stories were instead prioritized based on the Agile team’s ability to complete the work within the current sprint.

The project reported on its accomplishments via a road map. In May 2018, SeaWatch reported on progress toward milestones in its road map during an annual briefing for the Non-Major Acquisition Oversight Council. The program reported that it had installed SeaWatch v3.0 on 65 out of 70 in-service cutters.

Estimate the relative complexity of user stories The agency should have policy or guidance in place for relative estimation practices for Agile projects. Teams should use relative estimation for sizing the effort of work required to satisfy a user story by estimating its complexity based on work of similar size and complexity. Relative estimation enables teams to maintain a sustainable software development pace and predict work commitments. The team should size user stories relative to one another, assess the complexity of the work, refine user stories and estimates over time, and use prior estimates to inform future estimates. The product owner and team should continually revisit the estimates as they learn more about the business priorities and as user stories rise in the order of priority.

DHS did not provide policy or guidance for relative estimation. Although the Agile instruction manual identified estimating user story size as an integral part to sprint planning, it did not describe the specific techniques or processes for estimating the relative complexity of user stories. Instead, the Agile instruction manual discussed how programs could successfully apply traditional earned value management and cost estimating principles to Agile projects. DHS guidance noted that programs had largely moved from measuring story points to feature points to help programs quantify incremental progress The U.S. Customs and Border Protection’s (CBP) Biometric Entry Exit (BEE) program defined practices and guidelines for how the program expected to estimate user stories. For example, the Traveler Verification Services process definition document identified a formula for calculating story point values on the basis that one story point would equate to approximately four working hours. Moreover, the process definition document noted that story points must be reconciled to better reflect the level of effort and task completion at the end of a given sprint.

However, it was not evident that the BEE program had implemented its own guidance on the estimation of story points. Although the process definition document outlined procedures for estimating user stories, only two of 358 user stories in the Air Exit project backlog were estimated using story points.

Repeatable processes are in place

Agency policy or guidance should call for Agile teams to meet daily to review progress and discuss impediments, and to observe end-iteration demonstrations and end-iteration retrospectives. In addition, agency policy or guidance should call for Agile projects to employ continuous integration and confirm mechanisms are in place to ensure the quality of code being developed. This includes setting expectations for automated testing and code quality and tracking and monitoring against these expectations. Responsibility for these aspects of Agile development should lie with the individual Agile teams.

Meet daily to review progress and discuss impediments The agency should have policy or guidance in place for holding the daily stand-up and teams should hold daily meetings in order to stay on track to meet the iteration goals for Agile projects and adjust as necessary.

DHS guidance defined the general procedure for holding a daily stand-up. The Agile instruction manual stated that teams should conduct a daily stand-up meeting for all team members. It can be conducted in person or via another method of communication (particularly for remote employees) for a brief, informal meeting every work day. According to the manual, all team members should discuss the work each has accomplished since the last daily stand-up, the work to be accomplished by the next daily stand- up, and highlight any impediments that are preventing the team members from completing their work. Additionally, the manual suggests that it is necessary to conduct the daily stand-up with strict discipline, so that the meetings stick to their allotted brief time and are consistently productive.

The Agile instruction manual also highlighted the importance of the daily stand-up meeting to an Agile process. It called this meeting an essential collaboration event during which all team members were expected to participate and discuss their work. The manual suggested that holding these meetings allowed the team to practice discipline that would assist them in their work and foster mentoring and partnering relationships within the team that were reinforced through the constant communication of meeting every single day. The manual added that this activity allowed the team to hold its members accountable and be made aware of issues that may be mitigated through collaboration.

The Traveler Verification Services team supporting the BEE program Air Exit project at CBP held daily stand-up meetings. According to project officials and supporting project artifacts, a daily stand-up meeting was held each day at 10:00 a.m. Project officials noted that the daily stand- ups included the entire 40-person team.

The agency should have policy or guidance in place for holding demonstrations or other interactions for acceptance of user stories in Agile projects. Teams should hold frequent demonstrations to showcase features that have been implemented and obtain feedback for acceptance of user stories in Agile projects.

DHS guidance defined the general procedure for holding an end-iteration demonstration or review. In the SELC Tailoring Plan example for Agile development, DHS recommended a sprint review and demo as one type of technical review at the end of each iteration. The purpose of the review was to demonstrate the working software to end users and other stakeholders and to obtain feedback that could result in additional items being added to the backlog. It stated that this review should also ensure that the software design was documented for inclusion in the system design document, a proposed DHS Agile-specific artifact. The tailoring example noted that this review should formally end the iteration’s work with no further development or testing occurring on any stories. The Agile instruction manual added that this demonstration should confirm the value of the incremental piece of software produced.

DHS guidance also encouraged the use of demonstrations. The Agile instruction manual states that a demonstration or review could be used to reach a consensus on whether the work associated with a user story met expectations or not. The manual also recommended that program and project managers ensure that the functional software developed during each iteration was demonstrated to the stakeholder at an iteration review meeting.

The ICE SEVIS program held end-iteration demonstrations. The ICE SEVIS Modernization Systems Engineering Lifecycle Tailoring Plan stated that sprint demonstrations were tailored into the program to replace other review activities, such as the preliminary design, critical design, and integration readiness review. The Test and Evaluation Master Plan for SEVIS Modernization stated that standard sprint testing results were to be reported at sprint reviews. According to program artifacts, the sprint demonstration was to be conducted at the completion of each sprint, every other Wednesday from 11:00 a.m. to 12:00 p.m.

The agency should have policy or guidance in place for holding a retrospective to adapt and continuously improve on Agile projects. Teams should hold a retrospective at the end of each iteration to identify areas for improvement to adapt and continuously improve Agile practices.

DHS guidance defined the general procedure for holding a retrospective. The program or project manager and team reviewed progress after each iteration and release. This included the use of a retrospective to discuss what went well, what didn’t go well, and to identify actions to correct problems. Guidance noted that the team should immediately incorporate feedback from the retrospective into future iterations.

The DHS Agile instruction manual highlighted the importance of the retrospective. The manual stated that the end-iteration retrospective is a key part of ensuring that teams following Agile methodologies are able to identify problems and adapt to continuously improve for future sprints. Additionally, the manual stated that end-iteration retrospectives are useful in satisfying governance needs. For example, the Agile instruction manual stated that programs could tailor standard-format SELC artifacts (as codified in the SELC Tailoring Plan) to instead rely on assessment and performance data addressed in end iteration retrospectives.

The Traveler Verification Services team supporting the BEE program’s Air Exit project held end-iteration retrospectives. According to the process definition for this team, a retrospective was to be held between the end- iteration review and the subsequent planning session for the upcoming sprint. The process definition defined the goal of the retrospective as obtaining an honest review of the process with a consensus on how to adapt it. In an interview, project officials noted that the team documented the results of retrospectives on a release-by-release basis in a project management software tool.

The agency should have policy or guidance that defines and emphasizes the use of automated testing and continuous integration. This guidance should be supplemented by defining expectations for automated testing and tracking and monitoring against these expectations. Agile teams should adopt practices for continuous integration and automated testing to ensure that software handoffs are repeatable and dependable. Automated testing should be tracked and monitored based on established expectations.

The DHS Agile instruction manual defined continuous integration as the practice where delivery teams frequently integrate their code into a shared master copy. It noted that these integrations are verified by an automated build process, which performs testing to detect any integration errors quickly and automatically. The manual stated that continuous integration in Agile projects should be planned and recorded on a release-by-release basis.

The Agile instruction manual also emphasized the importance of continuous integration and automated testing. With regard to automated testing, the manual set an expectation for program or project managers and stakeholders to consider both automated testing tools and infrastructure support for the Agile software build and test processes as part of general project planning efforts. Moreover, the manual identified continuous integration, automated acceptance testing, and automated unit testing as key practices program or project managers can use for continuously monitoring and reporting project health. These practices could also help to identify opportunities for improving project team performance.

DHS officials acknowledged that current DHS programs implemented testing and evaluation inconsistently and that the department’s existing guidance and policies did not effectively support modern best practices in automated testing and continuous integration. To address these gaps, DHS had an Agile action plan that set an expectation for updating DHS acquisition guidance, policy, and practices for testing and evaluation to enable modern best practices in automated testing and continuous integration.

In lieu of more explicit guidance, DHS incorporated training as part of a curriculum geared toward test and evaluation managers that discussed both continuous integration and automated testing. According to the Deputy Director of Policy and Workforce Development in the Test and Evaluation Division of the Science and Technology Directorate, an alternative course containing content addressing Agile and continuous integration and automated testing was recently merged with a required test and evaluation course, creating a new course. According to the Deputy Director, the new course was piloted during fiscal year 2019 and will be standard in fiscal year 2020 as the required course for level II test and evaluation certification.

In order to track and monitor automated testing, the department incorporated several measures into the Agile core metrics. Programs executing Agile were expected to report on the following testing-related metrics after each iteration:

Percentage of unit test coverage,

Percentage of automated tests, and

Percentage of regression testing coverage.

DHS had not established expectations for these Agile core metrics. The Agile core metrics included a target. For example, the department suggested a program strive for seventy percent of tests to be automated. However, the instructions accompanying the Agile core metrics stated that all targets were notional and not expected to be reached. According to the Director of STM, the initial core metrics were intended to assess the level of DHS team achievement without imposing artificial industry- based target measures for each. The Director stated that, on receiving the metrics for a period of time, the department would then adjust the core metrics and begin to include target measures based on the results achieved. According to the Director, this effort was underway and an updated set of core metrics would be distributed in early fiscal year 2020. Moreover, the department was not tracking and monitoring automated testing as intended.

The CBP BEE program Air Exit project stood up a technical environment that allowed for continuous integration. This technical environment was outlined within the process definition of the Traveler Verification Services team that was developing software. The Traveler Verification Services process definition identified three operating environments: the development, test, and production environments. All development activities during the sprint were conducted within the development environment. Similarly, all testing activities in preparation for the release were conducted in the test environment. The final approved software would then be deployed to the production environment.

CBP officials noted that the BEE program primarily used Jenkins to integrate code for both continuous builds and deployment. The Air Exit systems design document also mentioned the role of Jenkins in continuous integration and continuous deployment for the project.

The Traveler Verification Services team incorporated JaCoCo and FindBugs automated tests as part of the continuous delivery process and they were run automatically when the code was checked in. Moreover, the project’s system design document noted that the Traveler Verification Services team integrated JaCoCo with the Eclipse Integrated Development Environment as a code coverage inspection tool for unit testing. Officials also noted that Selenium was used for automating the testing within the technical environment.

Ensure the quality of the code being developed The agency should have policy or guidance for an Agile project on ensuring the quality of code being developed. This guidance should be supplemented by defining expectations for code quality and tracking and monitoring against these expectations. Agile teams should adopt practices for code quality, such as having a test-driven development, pair programming, and manual code reviews to supplement automated testing. Agile teams should incorporate refactoring into code quality practices and understand the importance of setting aside time for refactoring.

DHS guidance recognizes the importance of ensuring code quality as part of the development and testing process. The SELC Guidebook set an expectation that code review and testing should be part of the software development environment. The guide recommended setting up servers where developers could test code and check whether the developed application runs successfully with that code. The guide suggested another level of tests on application reliability to help ensure that the application did not fail on the production server. The guide stated that the program manager should ensure that the team takes corrective action for any hardware and software deficiencies.

In order to find deficiencies early, DHS guidance identified coding and testing practices that could help development teams. The Agile instruction manual cited pair programming as one practice where two programmers work simultaneously on a single task: one programmer observes and reviews each line of code as it is written. DHS guidance also identified test-driven development as a practice that could motivate developers to write effective code. The Supplemental Guidance for Test and Evaluation stated that this approach consists of writing test cases that define a desired improvement, then writing the code to meet the desired functionality, ensuring that the test passes, and refactoring the code as necessary.

Refactoring, or re-coding, without changing the way the application functions, is an Agile practice that DHS guidance recommends for correcting deficiencies in the code. The Agile instruction manual stated that refactoring aims to improve code readability and reduce the complexity of previously delivered increments of software. It noted that refactoring is important because development teams are focused on adding the desired functionality with each release and may proceed with making improvements to the code. Refactoring was cited as one way to address this accumulation of needed improvements to the code, which are known as technical debt.

The Agile instruction manual further emphasizes the importance of setting aside time for refactoring to address risks associated with technical debt. The manual states that refactoring a previously developed increment of software to improve code quality may force a change in the release schedule. However, if the team does not make these revisions in a timely manner, the effort required to correct them later tends to increase. The manual states that this increasing technical debt is a risk factor to be addressed as soon as feasible. If the technical debt is allowed to accumulate unchecked, or if the project team loses track of the scope of its technical debt, the project could suffer from schedule and performance problems.

In order to track and monitor the quality of code being developed, the department incorporated several code quality and testing measures into the Agile core metrics. Among others, programs executing Agile were expected to report on the following quality-related metrics after each iteration:

Number of critical or major defects fixed.

Number of critical or major defects in the backlog.

Number of technical debt issues completed.

Number of technical debt issues in the backlog.

However, the department was not tracking and monitoring code quality as intended. These measures could provide programs and the department with an understanding of the development team’s ability to address defects and technical debt. In addition to these metrics, programs are also expected to report quarterly on the number of outages requiring a rollback or patch after production deployment.

The ICE SEVIS program used manual testing to ensure code quality. The definition of done for the program stated that new code should be peer reviewed to identify risk to the existing code, assess compliance with coding best practices, and evaluate refactoring. According to ICE SEVIS officials, an independent specialist provides internal code reviews and offers feedback on areas for improvement.

The ICE SEVIS program also employed automated testing to ensure code quality. The definition of done required that unit tests cover a minimum of 85 percent of code. Program officials stated that vulnerabilities and bugs identified through this process were added to the backlog and classified as technical debt.

The program refactored code to address technical debt, but did not set aside time for refactoring each sprint. According to ICE SEVIS officials, the development team refactored code as necessary to improve overall quality but did not set aside time for refactoring unless they were addressing a consistent issue. ICE SEVIS officials stated that the development team could propose refactoring code during sprint planning if there was a specific technical debt they had identified. However, according to the Scrum master for the program, addressing technical debt was additional work for the team to take on beyond the user stories they planned to complete and this additional work incentivized the development team to prevent the accumulation of technical debt.

Although DHS allowed Agile programs to tailor the core metrics, ICE SEVIS submitted some of the code quality-related Agile metrics to the department. The program included Agile metrics in June 2018 presentation slides for the Acquisition Review Board. For this initial reporting period, the program reported no critical or major defects in the backlog and no technical debt issues in the backlog. It also provided a screenshot of the Agile core metrics reported to DHS via the Investment, Evaluation, Submission, & Tracking system in February 2019. This reporting period covered two iterations. The program reported that it fixed four critical or major defects during the first iteration and did not have any critical or major defects in the backlog for either iteration. The program also reported that it completed eight technical debt issues in the first iteration, out of 14 technical debt issues in the backlog. The program did not report on the number of outages after deployment as part of the Acquisition Review Board program review or as part of the metrics submitted via the Investment Evaluation, Submission, and Tracking system.

Appendix VI: Comments from the Department of Homeland Security

Appendix VII: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, the following staff made key contributions to this report: Michael Holland (assistant director), Mathew Bader (analyst in charge), Lamis Alabed, Jennifer Beddor, Christina Bixby, Hannah Brookhart, Chris Businsky, Alan Daigle, Aryn Ehlow, Nancy Glover, Gina Hoover, Anna Irvine, Hoyt Lacy, Jennifer Leotta, Alexis Olson, Zsaroq Powe, Martin Skorczynski, Natalie Smith, and Daniel Spence.