Information Technology Reform: Agencies Need to Improve Certification of Incremental Development
Released: 2017-11-07T00:00:00

Letter

Background

Investments in federal IT have the potential to make agencies more efficient in fulfilling their missions. However, as we have previously reported, these investments too often result in failed projects that incur cost overruns and schedule slippages, while contributing little to mission- related outcomes. For example:

The Farm Service Agency’s Modernize and Innovate the Delivery of Agricultural Systems program, which was to replace aging hardware and software applications that process benefits to farmers, was halted in July 2014 after investing about 10 years and at least $423 million, while only delivering about 20 percent of the functionality that was originally planned.

Defense’s Expeditionary Combat Support System was canceled in December 2012, after spending more than a billion dollars and failing to deploy within 5 years of initially obligating funds.

VA’s Financial and Logistics Integrated Technology Enterprise program was intended to be delivered by 2014 at a total estimated cost of $609 million, but was terminated in October 2011 due to challenges in managing the program.

OPM’s Retirement Systems Modernization program was canceled in February 2011, after spending approximately $231 million on the agency’s third attempt to automate the processing of federal employee retirement claims.

DHS’s Secure Border Initiative Network program was ended in January 2011, after the department obligated more than $1 billion to the program, because the program did not meet cost-effectiveness and viability standards.

The tri-agency (Defense, NASA, and the National Oceanic and Atmospheric Administration) National Polar-orbiting Operational Environmental Satellite System was a weather satellite program that was disbanded by the White House Office of Science and Technology Policy in February 2010 after the program spent 16 years and almost $5 billion.

The VA Scheduling Replacement Project was terminated in September 2009 after spending an estimated $127 million over 9 years.

One approach to reducing software development risks is to divide investments into smaller parts, or increments. While a traditional waterfall software development effort usually is broadly scoped, multiyear, and produces a product at the end of a sequence of phases, an incremental development approach delivers software products in smaller modules with shorter time frames. This development technique has been recognized in prior law since 1996 and in OMB guidance since 2000.

By following an incremental development approach, agencies have the potential to: deliver capabilities to their users more rapidly, giving them more flexibility to respond to changing agency priorities; increase the likelihood that each project will achieve its cost, schedule, and performance goals; obtain additional feedback from users, increasing the probability that each successive increment and project will meet user needs; more easily incorporate emerging technologies; and terminate a poorly performing investment, with fewer sunk costs.

Since 2000, OMB Circular A-130 has directed agencies to incorporate an incremental development approach into their policies and ensure that investments implement them. Further, since 2012, OMB has required that functionality be delivered at least every 6 months.

In addition, FITARA states that OMB is to require in its annual IT capital planning guidance that covered agency CIOs certify that IT investments are adequately implementing incremental development, as defined in capital planning guidance issued by OMB. Accordingly, in June 2015, OMB released two related sets of guidance on the implementation of FITARA that included instructions pertaining to CIO certification of adequate incremental development. In particular, agencies were to, among other things:

Develop policies and processes which ensure CIO certification.

OMB required agencies to define IT policies and processes which ensure that the CIO certifies that IT resources are adequately implementing incremental development. In the guidance, OMB defined adequate incremental development as the planned and actual delivery of new or modified technical functionality to users that occurs at least every 6 months for development of software or services.

Report the status of CIO certification. OMB’s guidance required agency CIOs to certify in each major IT investment’s business case whether the investment’s plan for the current year adequately implements incremental development.

OMB uses the major IT business cases to monitor major investments once they are funded. Performance information on each major investment, including the status of incremental delivery, is made publicly available on the web-based IT Dashboard. In using the IT Dashboard, OMB intends to provide transparency and oversight into these agencies’ investments. This public display of data is also intended to allow Congress and government oversight bodies, as well as the general public, to hold agencies accountable for the results and progress of the investments.

Further, OMB issued its fiscal year 2018 and fiscal year 2019 capital planning guidance in June 2016 and August 2017, respectively, which required agency CIOs to provide the certifications needed to demonstrate compliance with FITARA.

GAO Has Reported on Efforts to Improve IT Acquisitions Using Incremental Development

During the past several years, we have reported on a variety of challenges related to improving federal IT acquisitions through the use of incremental development. In 2011, we identified seven successful investment acquisitions and nine common factors critical to their success. Specifically, we reported that department officials had identified seven successful investments that best achieved their respective cost, schedule, scope, and performance goals. Notably, all of these were smaller increments, phases, or releases of larger projects. For example, the Defense investment in our sample was the seventh increment of an ongoing investment; Energy’s system was the first of two phases; the DHS investment was rolled out to two locations prior to deployment to 37 additional locations; and Transportation’s investment had been part of a prototype deployed to four airports.

Common factors critical to the success of three or more of the seven investments were: 1. Program officials were actively engaged with stakeholders. 2. Program staff had the necessary knowledge and skills. 3. Senior department and agency executives supported the programs. 4. End users and stakeholders were involved in the development of requirements. 5. End users participated in testing system functionality prior to formal end-user acceptance testing. 6. Government and contractor staff were stable and consistent. 7. Program staff prioritized requirements. 8. Program officials maintained regular communication with the prime contractor. 9. Programs received sufficient funding.

These critical factors help support OMB’s objective of improving the management of large-scale IT acquisitions across the federal government.

In May 2014, we reported on the status of incremental development at five agencies (Defense, DHS, HHS, Transportation, and VA). We noted that these agencies planned to deliver functionality for fewer than half of the investments in 12-month cycles and that only about one-fourth of these investments would deliver in 6-month increments, as required by OMB. Additionally, OMB staff reported to us that they did not expect that many investments would meet the 6-month requirement. Therefore, we questioned whether a 6-month delivery requirement was an appropriate government-wide goal and whether OMB should instead consider a 12- month time frame, as called for in its IT Reform Plan. Accordingly, we recommended that OMB require projects to deliver functionality at least every 12 months. OMB disagreed with our recommendation, asserting that changing the requirement from 6 to 12 months would reduce the emphasis on incremental development that it had been advocating and that 6 months was an appropriate goal. However, we noted in our report, agencies’ plans to deliver functionality every 6 months was low and it would not always be practical for certain types of investments to deliver functionality every 6 months. We therefore continue to believe our recommendation is appropriate.

We also recommended that OMB develop and issue clearer guidance on incremental development to ensure that it has the necessary information to oversee the extent to which projects and investments are implementing its guidance. OMB took action to address this recommendation and issued capital planning guidance in fiscal year 2016 that requires agencies to report on whether each of their projects has delivered a production release every 6 months and to provide a rationale if functionality is not being delivered. In addition, we recommended that the five selected agencies—Defense, DHS, HHS, Transportation, and VA— update and implement their associated policies. Most agencies agreed with our recommendation or had no comment. As of September 2017, Defense, DHS, Transportation, and VA have addressed our recommendation.

In February 2015, we added improving the management of IT acquisitions and operations to our high-risk list, citing a lack of disciplined and effective management and inconsistent application of best practices to the successful acquisition of IT projects throughout the federal government. In particular, we noted the critical importance of implementing incremental development in order to reduce investment risk and called on federal agencies to ensure that a minimum of 80 percent of the government’s major acquisitions deliver functionality at least every 12 months.

In August 2016, we reported on the status of incremental development and noted that, for fiscal year 2016, 22 agencies had reported on the IT Dashboard that 64 percent of their software development projects would deliver useable functionality every 6 months, as required by OMB. However, shortcomings in OMB’s guidance—the lack of clarity regarding the types of projects where incremental development would not apply, and how the status of these nonsoftware projects should be reported— affected the accuracy of the data on the IT Dashboard. We therefore recommended in August 2016 that OMB clarify its existing guidance regarding what IT investments were and were not subject to requirements on the use of incremental development and how CIOs should report the status of projects that were not subject to these requirements. OMB did not specifically agree or disagree with our recommendation, but stated that it generally agreed with our report. In April 2017, OMB staff reported that the agency had taken action and included language to address our recommendation in its fiscal year 2018 guidance; however, an analysis of that guidance showed that it still lacked direction on how CIOs are to report the status of nonsoftware projects.

In addition, for our August 2016 report, we reviewed seven departments’ guidance and found that only three departments (Commerce, DHS, and Transportation) had policies and processes to ensure that the CIO would certify that IT investments were adequately implementing incremental development in accordance with FITARA. We therefore made recommendations to the remaining four departments (Defense, Education, HHS, and Treasury) to establish a policy and process for the certification of major IT investments’ adequate use of incremental development, in accordance with OMB’s guidance on the implementation of FITARA. Two departments concurred with our recommendation, one department disagreed, and one department did not comment. As of August 2017, none of the four departments had taken action to address the recommendation; as discussed later in the report.

We issued an update to our high-risk report in February 2017 and noted that, while progress has been made in addressing this high-risk area, significant work remains to be completed. For example, as of December 2016, OMB and agencies had implemented 366 (or about 46 percent) of the 803 open recommendations that we had made from fiscal years 2010 through 2015 related to IT acquisitions and operations. We also noted that agencies needed to make demonstrated progress in delivering functionality every 12 months on major acquisitions.

Further, in April 2017, we reported on the results of a forum, convened by the Comptroller General on September 14, 2016, to explore challenges and opportunities for CIOs to improve federal IT acquisitions and operations—with the goal of better informing policymakers and government leadership. Thirteen current and former federal agency CIOs, members of Congress, and private sector IT executives noted the importance of federal agencies’ IT procurement offices and processes evolving to align with new technologies, as agencies are not always set up to take advantage of acquisitions using Agile development processes.

Agencies Reported That Most of Their Major Software Development Investments Were Certified as Having Adequate Incremental Development, but Continue to Face Challenges and Identify Benefits

Agencies reported to OMB through the IT Dashboard that more than half of their major software development investments were certified by the CIO as implementing adequate incremental development as of August 2016. For the remaining investments, the agencies offered various interpretations regarding what investments needed to be certified. For example, officials of several agencies reported that they were not utilizing incremental development for certain investments. In other instances, agencies did not provide a response to OMB regarding the question in the major IT business case about certification, or responded that they did not consider certification to be applicable for their investments. However, based on OMB’s guidance, a number of these “not applicable” responses were incorrectly reported, as these agencies had investments that included software development and were, therefore, required to report on the certification of adequate incremental development.

In addition, officials from a majority of the agencies reported that multiple challenges had impacted their ability to implement adequate incremental development. These challenges related to inefficient governance processes; procurement delays; the lack of stable, prioritized requirements; and organizational and cultural changes associated with the transition from a traditional software methodology to an incremental methodology. Nevertheless, officials from 21 agencies reported that the certification process was beneficial because they used the information obtained during the process to assist with management oversight of major IT investments, including identifying investments that could be using a more effective incremental approach and using lessons learned to improve the agency’s incremental processes.

CIOs Certified 62 Percent of Major IT Investments as Having Adequate Incremental Development

FITARA states that, in its annual IT capital planning guidance, OMB is to require CIOs to certify that IT investments are adequately implementing incremental development. In 2015, OMB defined adequate incremental development as the planned and actual delivery of new or modified technical functionality to users that occurs at least every 6 months for development of software or services. Further, OMB’s IT capital planning guidance for fiscal year 2017 required CIOs to certify whether their agencies’ major IT investments had adequately implemented incremental development for the current year. Specifically, agencies were to respond to a question in the major IT business case regarding whether the CIO certified adequate incremental development for each investment with a response of either yes, no, or not applicable. Agencies’ responses to this question are publicly reported by OMB on the IT Dashboard.

As of August 31, 2016, 21 of the 24 agencies in our review had reported on the IT Dashboard a total of 166 major software development investments that were planned to be primarily in development for fiscal year 2017. Of these 166 investments, the agencies reported that 62 percent (103 investments) were certified by the CIO as using adequate incremental development for fiscal year 2017, as shown in table 1 in alphabetical order by department and agency. (For additional details on the certification status of the 166 investments, see appendix II.)

For the remaining 63 investments, 8 agencies either reported in the major IT business case that the investment was not certified as adequately implementing incremental development or that certification was not applicable. Three other agencies did not provide a response to the question regarding certification in the major IT business case submitted to OMB. Figure 1 shows the breakdown of responses by agency regarding investments that were not certified as implementing adequate incremental development, as reported on the IT Dashboard.

Officials in the Office of the CIO at each of the 3 agencies provided a variety of reasons for why the 11 investments were not certified as implementing adequate incremental development. For example, HHS officials noted that certain investments are required to meet complex statutory requirements and, thus, a 6-month release schedule is not always appropriate for them. Interior officials stated that their investment had just been categorized as a major investment and, at the time of the submission of certification status, a baseline had not been approved. The officials stated, however, that the baseline has since been approved and the investment is expected to deliver functionality every 6 months. Further, SSA officials reported that 3 investments were not software development initiatives even though 2 of these investments had been inaccurately reported as such on the IT Dashboard.

Regarding the 33 investments for which the 3 agencies did not provide a response in the major IT business case for the investment, officials from each agency’s Office of the CIO attributed the lack of a response to either data entry errors or the agency not being required to publicly report this information for the investments. In particular, USDA and Treasury officials reported that the lack of certification data on the IT Dashboard was the result of a data entry error. Treasury officials also stated that the agency’s missing responses were due to a lack of administrative oversight in reviewing the data for accuracy and consistency. The officials noted that the Treasury CIO had certified all of the agency’s investments but some investments failed to select the proper response in the business case.

Defense officials reported that 16 investments were categorized as national security systems and, therefore, were exempt from public reporting on the IT Dashboard (though not exempt from acquisition policies regarding the use of incremental development). The officials said that they did not provide a response on the remaining 7 investments because 1 investment was not a software development effort and the other 6 investments were designated as major automated information systems and, therefore, the agency did not have to submit business cases to OMB with this information.

Lastly, officials from the Office of the CIO at 7 agencies reported a variety of reasons for why they had provided a response of “not applicable” for 19 investments. For example, Interior officials stated that, at the time of the certification submission, the investment did not have any approved development projects and, therefore, the agency had indicated not applicable in its response for the one investment. However, the officials stated that the investment’s projects have since been approved and the CIO has reviewed the investment and certified adequate incremental development.

For the remaining 18 investments at the other 6 agencies (Commerce, DHS, Education, Energy, HHS, and Transportation), officials from each agency’s Office of the CIO reported that the majority of the projects associated with their investments were not primarily related to software development, or that they were using either a non-incremental development methodology or a mixed non-incremental/incremental development methodology. As a result, the officials believed the certification of adequate incremental development was not applicable, even though at least one project within each of the investments involved software development.

However, based on OMB’s guidance, these “not applicable” responses for the 18 investments were incorrectly reported and the agencies should have provided either a “yes” or “no” response to the certification question because the investment included software development. Specifically, OMB’s fiscal year 2017 capital planning guidance states that certification of incremental development applies to any investment that is developing software or services, as noted in its definition of adequate incremental development. In addition, staff in OMB’s Office of E-Government and Information Technology stated that a “not applicable” response to the question was only acceptable in cases where software development was not occurring, such as an investment related to infrastructure or technology refreshment of equipment.

Staff in the Office of E-Government and Information Technology acknowledged the need for more meaningful oversight of agencies’ use of incremental development and stated that, beginning in fiscal year 2018, OMB will no longer require agencies to report CIO certification information in their investments’ major IT business cases or on the IT Dashboard. Rather, OMB staff stated that agencies would be required to separately provide the certifications needed to demonstrate compliance with FITARA. OMB’s revised approach and agencies’ implementation of OMB’s guidance are further discussed later in this report.

Regardless of the reporting requirements in place, it remains critical that federal agencies report accurate incremental development information to OMB because of OMB’s plans to use this information for investment management and oversight. However, our September 2016 work has highlighted the poor quality of data related to incremental development at the project level, including whether a project is delivering a release every 6 months. Specifically, we reviewed seven agencies’ major IT software projects and found inconsistencies that affected the accuracy of the reported rates of delivery for all agencies—and at least a 10 percentage point difference in the reported rate on the IT Dashboard for five of these agencies. We therefore made recommendations to the seven agencies to improve their reporting of incremental development data on the IT Dashboard.

Having accurate data on agency investments’ use of incremental development is critical for providing oversight and management of these investments and to ensure that OMB and lawmakers can hold CIOs accountable for the investments’ performance. We have previously made recommendations to Commerce, Defense, DHS, Education, HHS, Transportation, and Treasury to improve the accuracy of reporting on the IT Dashboard and continue to believe these recommendations are appropriate. In addition, until Energy, SSA, and USDA improve their reporting of incremental development data on the IT Dashboard, their efforts to improve the use of incremental development may not be successful. As a result, the agencies increase the risk that the potential impact of utilizing incremental development to more quickly deliver useful functionality to users and improve the likelihood that these multimillion dollar projects will meet their stated goals, may not be realized.

Multiple Challenges Were Commonly Identified by Agencies as Impacting the Delivery of Incremental Functionality

The majority of the 24 agencies in our review reported that multiple challenges had impacted their ability to adequately implement incremental development for their major IT software development investments. In particular, when presented with a list of challenges identified by our past work on incremental development, 21 of the agencies selected seven common challenges to developing investments incrementally. Each of these seven challenges was selected by 5 or more agencies. For example:

14 agencies identified problems with program staff over-utilization and the lack of skills and experience as their top challenge;

6 agencies reported that development work was slowed by inefficient governance and oversight processes;

5 agencies reported that development schedules were impeded by procurement delays; and

5 agencies identified the lack of stable, prioritized requirements as a challenge.

In addition, 3 agencies identified a new challenge which had not been described in our prior work. Specifically, they reported that organizational and cultural changes associated with the transition from a traditional waterfall software methodology to an incremental methodology required more time and resources to implement than anticipated. Table 2 summarizes the common challenges identified by agencies and the number of agencies that reported each challenge, ranked by number of agencies reporting the challenge. Examples of the challenges—and actions taken to overcome them—are discussed following the table.

Project staff were over-utilized or lacked the necessary skills and experience. Officials from the Office of the CIO at 14 agencies (DHS, Education, EPA, GSA, Justice, NASA, NRC, OPM, SBA, SSA, State, Treasury, USAID, and VA) reported challenges in implementing incremental development practices associated with project staff, such as a lack of staff with the necessary skills and experience in utilizing incremental approaches, inadequate training on these approaches, overutilization of business or subject matter experts, and the lack of engagement between product owners and subject matter experts. To address these challenges, agency officials reported implementing new approaches, such as training programs focused on incremental development, coaching strategies to assist project managers in managing acquisitions, and hiring practices. For example, among these agencies:

DHS officials reported that project staffs’ lack of necessary skills and experience in understanding the requirements for managing major IT acquisitions is an ongoing issue, not only related to incremental development, but also to IT program and project management. The officials stated that they had developed an acquisition coaching and assistance strategy that was intended to establish an experienced team of acquisition coaches who were up-to-date on the latest acquisition, contracting, and development techniques to assist project managers in managing the acquisitions. The officials stated that they hoped to present lessons learned and recommendations on this strategy to the agency’s Agile working group in summer 2017.

Treasury officials reported a significant need for specialized engineers, architects, and developers with skills in older programming languages to maintain its many legacy systems. For example, officials noted that the agency is modernizing its core taxpayer account processing applications, which utilized antiquated programming languages, to more modernized platforms. Treasury officials noted that they have been shifting staff to meet immediate needs; augmenting teams with contractors, where possible; and hiring new staff to fill critical open positions. Nevertheless, the officials said they have had to slow work on four key projects and delay the launch of other projects. In addition, the officials stated that they are relying on contractors more to meet the agency’s staffing needs.

EPA officials noted that, as the agency transitions from using waterfall software development approaches to Agile-based approaches, it needs more skilled staff with experience in Agile development. These officials stated that the agency’s CIO had taken several actions to address this challenge, including creating an Office of Digital Services and Technical Architecture to promote Agile and user-centered design, establishing a fellowship program to bring outside Agile experts into the project teams, and creating a blanket purchase agreement to allow agency project teams to purchase Agile programming and consulting services.

According to NRC officials, one of the greatest incremental delivery challenges has been the difficulty of engaging sufficient business area product owners and subject matter experts. For example, the officials explained that, despite product owners’ enthusiasm for increased engagement with developers, the demands of the agency’s core mission work presents challenges for these owners in being available for meetings related to Agile development activities. NRC officials informed us that the agency had addressed the challenge by working to establish a predictable, recurring schedule for product owner and subject matter expert engagement on development projects, where expectations are communicated to management about time commitments.

Further, agency officials from a number of the 14 agencies that experienced this challenge reported varying approaches to implementing new incremental development training. For example, Treasury officials stated that the agency has developed in-house training for existing developers to meet the needs of its modernized programs. Education officials noted that the agency identified a select team of IT professionals within the agency to receive formal training in incremental development practices. Further, VA officials told us that its Enterprise Program Management Office is focused on training IT personnel on incremental development principles. Finally, SSA officials reported that the agency had launched a training program that had sent hundreds of developers through a 6-week boot camp program, which included courses in incremental development and modern coding languages.

Programs did not receive sufficient funding or received funding later than needed. Officials from the Office of the CIO at nine agencies (GSA, NASA, OPM, SSA, State, Treasury, USAID, USDA, and VA) reported challenges associated with programs not receiving sufficient funding or not receiving funding until late in the fiscal year. These challenges were a result of changing funding priorities, budget cuts, and continuing resolutions, which disrupted delivery schedules and required agencies to delay, reprioritize, or discontinue the rollout of particular investments or modernization activities. Agencies reported adopting various approaches to overcome the challenges in this area, such as delaying project schedules, developing alternate plans for delivering functionality, and using flexible contracting strategies. For example:

USDA officials reported that funding for a number of projects was not available until late in the fiscal year, which impacted project schedules. The officials stated that one component agency addressed the funding delay by adjusting schedule start dates for projects relative to the current fiscal year, which helped to improve schedule projections.

OPM officials told us that they had faced challenges in performing work on incremental projects due to a lack of available resources caused by delays in receiving funding. The officials stated that they addressed this challenge by developing alternate plans to delivering incremental functionality with a different scope or focus for the system.

VA officials reported that they faced challenges with funding IT efforts that span multiple years. The officials noted that administrative priorities often change over time, impacting the level of funding approved in subsequent years to undertake incremental development projects. To address this, officials noted that they used flexible contracting strategies—such as options that allow the government to continue the contract only when funding is assured, adjusting a contract’s time frames to match a delay, adjusting schedules, designing contracts so that a vendor is paid based on completion of measured functionality, and using the change request process to contribute funding to other projects.

Treasury officials stated that the lack of a dedicated funding commitment had led to difficulties in longer-term strategic planning for IT improvements. The officials stated that resources assigned to certain IT projects had to be leveraged for legislatively mandated investments, causing project delays and pauses for these projects. As a result, the officials reported that the agency had been reviewing core initiatives and infrastructure programs, such as infrastructure, hardware, and software refresh and process improvements, to determine if they can scale back scope or lengthen schedules. The officials said that at least one program has been formally paused.

Projects experienced management and organizational challenges that introduced delays. Officials from the Office of the CIO at seven agencies (Commerce, Interior, NASA, NRC, NSF, SBA, and Transportation) reported that management and organizational challenges had introduced delays in delivering functionality to users. These challenges included delays in testing and meeting delivery schedules due to dependencies on other systems or projects and a lack of approved software or appropriate equipment. Agency officials reported implementing various approaches to overcome these challenges, such as addressing external dependencies, tailoring development processes, and providing waivers for the acquisition of software and hardware. For example:

Commerce officials reported that they faced organizational challenges in meeting scheduled delivery time frames due to delays with another project that was not ready for testing. In particular, the officials reported that one of their systems was ready for testing but experienced delays because the system had an interface with another system that was not ready for testing. The officials said that the delay in Commerce’s ability to test its system resulted in missed delivery milestones. In order to continue development, the project team separately tested its system without including the interface functionality.

NRC officials reported that they had experienced delays in meeting their incremental projects’ delivery schedules due to dependencies on multiple complex projects. These officials told us that the agency addressed these delays by improving existing processes and implementing a change control board and an enterprise test development environment.

SBA officials reported that delays were introduced when the agency did not have necessary software and hardware available for development activities. Officials noted that these challenges were a result of the agency not maintaining an updated inventory of approved software and developers not having access to laptops needed for development activities. SBA officials stated that the agency addressed the lack of approved software and equipment needed for incremental development by processing a waiver to use software tools and procuring laptops for the developers.

Incremental development work was slowed by inefficient governance and oversight processes. Officials from the Office of the CIO at six agencies (DHS, HUD, NRC, State, USAID, and USDA) reported that they had experienced challenges in developing projects incrementally because they were required to follow agency processes that were lengthy, inefficient, or not easily adaptable to a more rapid incremental delivery release schedule. Agency officials also noted that a lack of understanding among project staff regarding the benefits of incremental development was a challenge. The officials reported implementing new guidance and management processes to overcome these challenges. For example:

DHS officials reported that inefficient governance and oversight processes had caused delays in obtaining necessary approvals for moving projects forward. Specifically, these officials reported that the agency’s acquisition lifecycle framework did not allow for tailoring any of its processes to accommodate Agile development. The officials noted that these challenges were addressed with the publication of updated lifecycle documents that incorporated incremental development guidance into the agency’s policies and procedures.

HUD officials reported that the agency’s internal approval process for the Privacy Act System of Record Notice did not accommodate incremental releases. Specifically, the agency’s incremental development process called for the release of functionality every 60 days, but the agency’s Privacy Office required 90 to 180 days to complete its approval process. HUD officials reported that the Office of the CIO is collaborating with the Privacy Office to expedite the existing approval process, and have proposed that a single system of record notice be prepared for each incremental development project, rather than one for each release.

USAID officials reported that the time needed for defining and incorporating changes in response to IT security and privacy standards, processes, and artifacts provided before the system is granted an Authority to Operate is a challenge. These officials stated that the Office of the CIO has acquired additional knowledgeable staff to support projects in the incorporation and execution of security and privacy requirements.

State officials reported that applying incremental development principles to projects has been a challenge because agency personnel have lacked a clear understanding of the benefits of incremental development and how to apply incremental concepts to unique project types. These officials reported that the agency was updating its guidance and processes to place greater emphasis on the importance of incremental development, and that the agency had established a review process to ensure projects plan for implementing incremental development.

Project characteristics made rapid delivery of functionality infeasible or impracticable. Officials from the Office of the CIO at six agencies (Interior, Justice, Labor, SSA, Transportation, and Treasury) reported that they believed rapid delivery of functionality was infeasible or impracticable for projects that addressed human health and safety concerns, had legislative mandates that established immovable delivery time frames, were primarily for infrastructure deployment, were updates to existing systems to address legal or other regulatory changes, or were updates to legacy systems that utilize old programming languages. However, none of the agencies identified solutions for these challenges that enabled them to deliver functionality in the 6-month time frames required by OMB. For example:

Transportation officials noted that Federal Aviation Administration projects, like those for its Next Generation Air Transportation System, are unique and complex due to safety concerns that impact the national airspace. As a result, these investments require years of design, development, and testing, which officials believe precludes using incremental approaches that must deliver usable functionality every 6 months.

Labor officials reported that certain projects, which are initiated in response to an executive order or other external mandate, come with required delivery time frames. This results in relatively short development schedules that do not lend themselves to using an incremental approach.

Justice officials reported that several of the agency’s investments primarily dealt with the deployment of secure telecommunications, data centers, and other network infrastructure, making it difficult to translate that delivery into meaningful increments. Justice officials stated that they did not deploy incremental development because the projects were infrastructure projects.

Treasury officials reported that the development and maintenance of some major investments, such as the agency’s legacy tax systems, are not conducive to a 6-month delivery schedule due to the number of modifications that must be made based on changes to the tax laws, legislative mandates, and other system updates. Treasury officials stated that the agency has established a mature governance process for rolling out changes to these tax systems so there is only one annual update to the systems.

SSA officials stated that using an incremental software development approach to modernize the agency’s legacy applications was challenging because the code for these applications was unstructured, overly complex, heavily interdependent, and utilized old programming languages. The officials stated that, in order to modernize these legacy applications, the project teams had to break programming changes into useful segments, streamline imbedded business process requirements, and rewrite the code using modern programming languages. As a result, the officials stated that these activities could not, at least initially, deliver functionality in smaller increments.

Incremental development schedules were impeded by procurement delays. Officials from the Office of the CIO at five agencies (Education, HUD, Interior, OPM, and USDA) reported that they had experienced challenges with meeting incremental development schedules due to delays in getting contracts awarded or getting contract modifications approved. To overcome this challenge, agency officials reported that they negotiated with vendors and worked with the offices of procurement within their agencies to reduce delays and ensure all paperwork was completed in the time frames required. For example:

Education officials reported that the agency uses contractors to perform most of its software development work. These officials stated that modifying existing contracts to require the use of incremental development approaches had caused delays in getting vendors to deliver functionality in 6-month increments. Education officials reported that they had negotiated with vendors to restructure delivery schedules in order to meet incremental delivery time frames.

HUD officials reported that they had faced challenges in meeting project schedules due to delays in getting paperwork approved by the agency’s procurement office, which was busy with end-of-year activities. To address this, HUD officials stated that they collaborated with the Office of the Chief Procurement Officer to ensure the project teams submitted the required documentation for approval in advance of the procurement office’s end-of-year activities.

OPM officials noted that they had faced challenges with adapting their procurement process to use incremental approaches. The officials stated that they worked with their Office of Procurement to incorporate incremental development procurement methodologies in order to reduce the time from contract initiation to award, as well as to reduce the amount of contract documentation and its complexity.

Programs did not have stable, prioritized requirements. Officials from the Office of the CIO at five agencies (DHS, Justice, NSF, Transportation, and VA) reported that maintaining stable requirements, including defining a set of initial requirements, handling ongoing changes, and managing stakeholder expectations regarding the scope of, and number of changes to requirements, were challenges. To overcome these challenges, agencies reported strengthening standards, implementing training and coaching, and exercising better requirements and business practices. For example:

DHS officials stated that managing stakeholder expectations related to requirements was challenging because product owners and business users expected project requirements not to change once they were developed, while development teams had planned for requirements to change and be reprioritized over the course of the project since the team was using an incremental approach. These officials reported that they issued new guidance and offered assistance and coaching for programs and projects to better identify and document needs and requirements, while encouraging business users to plan for and prioritize the backlog of items to be deployed incrementally.

Justice officials reported that it was a challenge to finalize the scope of work for various projects because disparate stakeholders had competing priorities which led to constant changes in the requirements. The officials noted that, for one of the agency’s projects, the project team is currently establishing a process to obtain consensus on stakeholder priorities in advance. For other projects, Justice teams have sought or received training from experienced, certified Agile experts in developing customer requirements.

NSF officials reported that, when first establishing its incremental development program, the agency had experienced challenges in defining a stable set of priority requirements for the initial increments. The officials told us that, to address this challenge, they elevated customers to fill the leadership roles of the working groups that provided the requirements to ensure the requirements of each increment were well defined and clearly prioritized.

VA officials reported that, while the agency has transitioned to Agile development methods over the past several months, it still works through challenges in developing detailed user stories with its business partners, and reported many instances when a project was undertaken without knowing the full scope of requirements. VA officials reported that they took several actions to help address this challenge, including introducing a new development methodology to promote incremental development principles, and establishing an account management office that works with business partners to ensure detailed business cases are prepared prior to approval. They also integrated more rapid prototyping into the planning stages as a way to gather requirements and test assumptions early and cheaply.

Organizational changes associated with the transition from a traditional software methodology to an incremental development methodology require time and resources. Officials from the Office of the CIO at three agencies (EPA, GSA, and Labor) independently reported challenges related to organizational changes, such as staff adapting to the culture shift from being business customers to taking on a more active role as product owners and project managers in the software development process. For example:

EPA officials stated that the agency had experienced challenges as staff transitioned from using waterfall development practices to Agile practices because there had been skepticism within the agency on whether an Agile approach could meet the requirements for agency systems. The officials stated that the CIO had established an office to provide support to project teams that needed assistance in adopting Agile approaches, created a community of practice group, and developed guides and other maturity models to provide guidance on the adoption of Agile methodologies.

GSA officials explained that implementing incremental delivery has required a culture shift for the agency’s business customers who were accustomed to having a different set of roles and responsibilities in the traditional software development process than what is used in the incremental development process. The officials stated that they have worked to train their customers to better capture the vision of what needs to be built and to be more active product owners and managers in communicating with the development team. As a result, the officials in the GSA Office of the CIO stated that they are enabling the business customers to serve as better product owners. The officials further stated that, by implementing this change, project staffs have (1) defined and prioritized clearer requirements; (2) selected the proper technical tools to support business needs; (3) worked with the contracting office to develop better-defined contracting documents and make contract awards; (4) identified dependencies associated with development efforts; and (5) provided transparency on what work has been completed, what work is planned, and the challenges associated with the investments.

Additionally, three agencies (Defense, Energy, and HHS) reported no challenges with implementing incremental development. However, officials from all three agencies discussed issues surrounding the use of incremental development, both as part of this review and as part of our prior work. In particular, Energy officials had told us that they had projects that failed to adequately employ incremental development practices, which required follow-up with program managers to identify corrective actions. Also, both Defense and HHS officials have reported facing management and organizational challenges, such as dependencies on integrating changes with other systems, which impacted the delivery of functionality every 6 months. Defense officials noted that many of the agency’s investments were complex and could not adhere to a 6-month delivery schedule.

Federal investments may continue to encounter increased cost and schedule risks if agencies cannot adequately implement incremental development approaches. The discussion of challenges identified in this report—and the range of actions taken by the agencies to address them— is a valuable resource that could have the potential to help agencies that face similar concerns.

Agencies Reported Using Information from the Incremental Certification Process to Improve Investment Management Oversight

Although a number of agencies identified challenges in utilizing incremental development, officials in the Office of the CIO at 21 of the 24 agencies also reported that the CIO certification process was beneficial to their agencies because it had assisted them in overseeing the management of agency investments. For example, officials from 13 agencies reported that they used the information derived from the certification process to identify challenged development projects that could be using a more effective incremental development approach and officials from 2 agencies stated that the information helped them determine whether an investment should undergo a TechStat review. Table 3 lists the four benefits reported by federal agencies in utilizing the CIO certification process and the number of agencies that reported each activity, ranked by number of agencies reporting the challenge. Examples of the benefits agency officials identified from these investment management oversight activities are discussed following the table.

More effective use of incremental development approaches. Officials from the Office of the CIO at 13 agencies (Defense, DHS, Education, Energy, EPA, GSA, Interior, NASA, NRC, SBA, SSA, Transportation, and USDA) stated that they review the information about the investment’s use of incremental development to identify projects that could be implementing a more effective incremental development approach. For example, Energy, GSA, and SBA officials stated that they review projects not using adequate incremental development in order to identify necessary corrective actions, such as: (1) breaking out projects into shorter duration activities; (2) implementing the use of investment reviews, whereby funds are released incrementally upon completion of clear success criteria; (3) developing major IT investment business cases that outline project plans for incremental development; and (4) monitoring new and existing investments to ensure delivery of capabilities within schedule and cost thresholds.

In addition, DHS, NASA, NRC, and SSA officials reported that the CIO uses the information to make corrections to projects that are not adequately implementing incremental development through such actions as the CIO’s office: (1) working with project team officials to convert project activities to an incremental approach; (2) requiring any deviations from approved releases of software development products to be approved by the CIO; (3) requiring projects that deviate from the use of adequate incremental development principles to be approved by the CIO; and (4) determining which investments must use incremental development, and requiring the projects to do so.

Provide oversight of IT investments. Officials from the Office of the CIO at seven agencies (Commerce, Interior, Labor, OPM, NSF, State, and VA) stated that they use the information to provide oversight of IT investments. In particular, Interior and NSF officials reported that their CIOs use the information obtained during the performance measurement baseline approval process to make decisions regarding the agency’s major IT investments. For Interior, officials stated that the types of decisions the CIO may make include, but are not limited to, accelerating delivery, reducing scope, or halting or terminating an IT project. For NSF, officials stated that the decisions could result in changes to program objectives or scope of individual projects under the program, redirection of resources, changes to planned levels of expenditure, or recommendations for corrective actions based on the evaluation.

In addition, Commerce officials stated that investment data are reviewed by the CIO on a monthly basis and, based on the status, can undergo further scrutiny at a review board meeting or other CIO review process. Labor officials noted that its capital planning team updates the CIO’s rating and explanation for each major IT investment in the agency’s capital planning and investment control system, and submits the rating information to the IT Dashboard each month.

Improve incremental development processes. Officials from the Office of the CIO at five agencies (DHS, EPA, HUD, Justice, and USDA) stated that they leveraged the information to improve their incremental development processes. For instance, USDA officials reported that they leveraged the results of the certification process to build an incremental development community of practice. DHS officials stated that they developed coaching and other assistance to help convert projects to an incremental process. Lastly, Justice officials stated that they utilized the results of the certification process to: (1) develop best practices and lessons learned on using incremental development, (2) establish additional training, and (3) establish mentoring programs or other familiarization with incremental techniques to support business improvement.

Determine if a TechStat is warranted. Officials from the Office of the CIO at two agencies (Labor and SBA) stated that they use the results of the certification process to determine whether an investment should undergo a TechStat review. In particular, Labor officials stated that if an investment is rated as high risk for 3 consecutive months during the review process, then a TechStat is initiated. In addition, SBA officials noted that, as part of their certification process, the Office of the CIO portfolio management team meets with the CIO to determine if any IT investments should have a Techstat review.

Given the significant size of the federal government’s annual investment in IT and the often disappointing results from IT development efforts, finding innovative ways to improve the quality and timeliness of agencies’ IT investments may help improve these development efforts. The discussion of benefits identified with using the certification process—and the range of management oversight activities taken by the agencies— may have the potential to help agencies improve their management and oversight of IT acquisitions.

Most Agencies Lack Detailed CIO Certification Policies and OMB Has Improved Related Reporting Guidance

Of the 24 agencies in our review, only 4 had clearly defined processes and policies to ensure that the CIO will certify that major IT investments are adequately implementing incremental development. The remaining 20 agencies either did not include details such as the role of the CIO in the certification process or how certification would be documented, or had not yet finalized a policy.

OMB’s fiscal year 2018 guidance was not clear regarding what actions agencies should take to demonstrate compliance with FITARA’s certification requirement. However, OMB issued its new fiscal year 2019 guidance in August 2017, which addressed the weaknesses we identified.

Only 4 of 24 Agencies Have Clearly Defined a Policy for CIO Certification of Incremental Development

A provision in FITARA, enacted in December 2014, states that, in its annual IT capital planning guidance, OMB is to require agency CIOs to certify that IT investments are adequately implementing incremental development. Subsequent OMB guidance on the law’s implementation, issued in June 2015, directed agency CIOs to define processes and policies for their agencies which ensure that they certify that IT resources are adequately implementing incremental development. As part of the guidance, OMB defined adequate incremental development as the development of software or services, with planned or actual delivery of new or modified technical functionality to users that occurs at least every 6 months.

OMB’s guidance allows agencies the flexibility to define the processes that CIOs use for ensuring the certification of adequate incremental development. For example, CIOs can rely on internal governance processes, such as investment and capital planning processes, to evaluate agency investments for adequate use of incremental development. In addition, agency CIOs are to use OMB’s definition of adequate incremental development when developing their certification processes and determining whether to certify that their investments met these criteria. While OMB’s guidance is not specific on what elements should be included in these certification policies and processes, GAO’s Information Technology Investment Management framework notes that policies and procedures should be clearly defined, including the role of appropriate stakeholders, and have appropriate artifacts to document decisions made.

Although OMB’s requirement has been in place since June 2015, only 4 of the 24 agencies we reviewed (Commerce, DHS, Energy, and Transportation) have clearly defined processes and policies intended to ensure that their CIOs certify that major IT investments are adequately implementing incremental development. Specifically, all 4 agencies’ policies contained all the elements that we evaluated in the agency guidance: descriptions of the role of the CIO in the process; how the CIO’s certification will be documented; and definitions of incremental development and time frames for delivering functionality consistent with OMB guidance.

However, the remaining 20 agencies did not have clearly defined processes and policies in place because their documentation either did not describe the CIOs’ role in the certification process or how certification would be documented, define incremental development and provide delivery time frames consistent with OMB guidance; or the policy had not yet been finalized. The results of our analysis of agencies’ policies is shown in figure 2, while additional details regarding the status of the 24 agencies’ incremental policies are provided in appendix III.

The four agencies that had clearly defined policies for certification took a variety of approaches to defining how the CIOs would conduct the review and certification of major IT investments, determining how certification would be documented, and ensuring OMB’s guidance regarding the definition of adequate incremental development and delivery time frames was followed. Specifically:

Commerce’s capital planning guidance requires bureau CIOs or other accountable officials to review project documentation regarding project deliverables and issue an e-mail or other time-stamped document that certifies the adequate implementation of incremental development. In addition, Commerce guidance adheres to OMB’s guidance requiring delivery time frames every 6 months or less and sets forth a definition of adequate incremental development that is consistent with OMB guidance.

DHS’s technical investment review guidance states that the CIO is to conduct a review of each investment using an investment review checklist that includes information provided by project managers as to whether the investments have used incremental development adequately. The CIO is to certify whether the project is implementing incremental delivery at least every 6 months and document this certification in the checklist. DHS guidance also includes a definition of adequate incremental development and time frames for delivering functionality that are consistent with OMB guidance.

Energy’s capital planning guidance states that the CIO is to review and certify each investment’s adequate use of incremental development as part of monthly investment review board meetings and during the monthly review of the IT Dashboard data. The status of this certification is documented in the agency’s monthly investment summary spreadsheet. In addition, Energy’s guidance adheres to OMB’s definition of adequate incremental development and its associated delivery time frames for its incremental development activities.

Transportation’s investment management guidance states that the CIO is to conduct a review of the investment as part of the investment review board process; this board is co-chaired by the agency CIO. The CIO is to certify adequate incremental development in the signed investment decision review document. In addition, Transportation’s guidance adheres to OMB’s definition of adequate incremental development and delivery time frames.

However, the remaining 20 agencies did not have clearly defined policies and processes in place to ensure CIOs are certifying each major IT investment’s adequate incremental development. In particular, while officials from the Office of the CIO at 11 agencies asserted that they had a policy for CIO certification, these policies lacked details, such as a description of the role of the CIO in the process, a description of how certification would be documented, and definitions of incremental development and delivery time frames consistent with OMB guidance. Table 4 details our evaluation of the certification policies provided to us by the 11 agencies.

Agency officials in the Office of the CIO at each of the 11 agencies provided a variety of reasons for why their policies lacked details regarding the role of the CIO in the process and how certification was documented, or did not include definitions for incremental development and delivery time frames. For example, State officials reported that updating their policies to comply with FITARA was not seen as a priority until Congress conducted its own evaluation of incremental development in May 2016. They stated that their new policy is currently in the process of being finalized but no time frames for finalization were provided. However, we could not determine whether the guidance is expected to address the issues we identified because State provided us excerpts of its new draft policy and the new proposed guidance that did not include any details in the areas we identified.

In addition, GSA officials stated that they had used existing governance bodies and processes to determine whether the investment would be certified. The officials stated that they did not see a reason to create a separate policy for CIO certification, since the agency always looks at using incremental development for new projects and the agency certifies the investment in the major IT business case.

Further, OPM officials stated that their agency had been on a path to address the FITARA requirements, but progress was slowed due to the lack of a budget for fiscal year 2017. The officials stated that they intend to update the agency’s policies, but had no firm plans for doing so pending the availability of budgetary resources.

Lastly, NSF officials stated that they have not seen the need to have a policy on CIO certification for a number of reasons. NSF reported that it is a small agency with few large IT investments, and many of those are legacy systems in operations and maintenance, rather than development. Therefore, according to the officials, the agency has not had many occasions for the CIO to need to certify adequate incremental development for major IT investments. Second, the officials stated that the NSF CIO is actively involved in the investment review process and did not feel a policy was needed to describe these activities. Third, NSF officials stated that it is their belief that policies are generally only required to correct something which is not working. Lastly, NSF officials stated that the agency’s definition of an Agile sprint was its definition of incremental development. However, sprints are not released directly to users, and therefore, the definition is not consistent with OMB guidance. However, the officials said they might reconsider developing a policy, but did not provide a time frame for doing so.

Finally, 9 agencies had not yet finalized a CIO certification policy. Office of the CIO officials in each of these agencies reported that they had relied on existing IT governance processes and budget mechanisms, or created new targeted IT reviews to determine the CIO certification for fiscal year 2017 that was reported on the IT Dashboard. For example, HHS officials reported that the agency used existing project and investment milestone reviews as part of its enterprise performance lifecycle to determine whether the investment would be certified as having adequate incremental development.

SBA officials told us that the agency’s portfolio management team met with investment managers during the monthly update process for the IT Dashboard, while USAID officials noted that the agency’s CIO reviews the incremental development status of all major investment software development projects on a monthly basis. Further, Justice officials reported that the IT Investment Oversight Manager’s staff reviewed the major business cases and requested justification for software development investments that were not: (1) using an iterative or Agile methodology, (2) expected to have a production release containing usable functionality every 6 months, or (3) showing an actual or planned date for deployment production within a 6-month time frame.

In addition, while six of these agencies reported plans to finalize a policy for CIO certification by December 2017, one agency reported its policy would be finalized in 2018, and two agencies did not provide a time frame for finalizing a policy. Figure 3 below shows the agencies’ reported time frames for finalizing a policy on CIO certification of incremental development.

Officials from each agency’s Office of the CIO provided a variety of reasons for why they had not yet developed or finalized policies for CIO certification of adequate incremental development. For example, EPA officials stated that the agency has been focusing on standing up the programs and structures needed to support incremental development and, thus, had not prioritized developing a policy. In addition, EPA officials stated that they had not developed a definition of functionality or time frames, but that their guidance points to industry standards.

SBA officials stated that, since the majority of the agency’s investments were in operations and maintenance, they did not see the need to have policies or procedures for incremental development. In addition, HUD, NASA, and USAID officials reported that their agencies were in the process of finalizing policies, but had experienced delays due to the number of stakeholder comments or limited staff resources.

Lastly, Defense officials stated that they had included information in their fiscal year 2018 budget submission guidance for component CIOs to certify adequate incremental development and were working to incorporate this process into their Financial Management Regulations, which were to be finalized in the first quarter of fiscal year 2018. However, the officials stated that the agency’s process is driven by its efforts to comply with whatever process OMB requires in the annual capital planning guidance and, thus, they would not have a separate certification policy from the budget guidance.

Additionally, Defense officials reported that, for their agency’s investments, delivery every 12 to 18 months was more appropriate than the 6 months that OMB requires. Nevertheless, while Defense officials may believe that 12 to 18 month delivery cycles may be more appropriate for their work, OMB’s guidance requires agencies to deliver functionality at least every 6 months and does not allow for exceptions. We previously recommended that Defense establish a policy on the CIO certification of incremental development. Until this guidance is finalized, Defense may not be able to ensure incremental development practices are adequately implemented at the agency. We therefore continue to believe the recommendation is appropriate.

Annual CIO certification of incremental development is critical to ensuring that agency CIOs exercise the proper authority and oversight over their agencies’ major IT investments. Having appropriate authority and oversight helps to create IT systems that add value and are aligned with agencies’ missions, while reducing the risks associated with low-value and wasteful investments. In the absence of clearly defined policies, agencies continue to run the risk of failing to deliver major investments in a cost-effective and efficient manner.

We have previously made recommendations to Defense, Education, HHS, and Treasury to establish CIO certification policies, but as noted in this report, these agencies still have not yet finalized their guidance to clearly detail their agencies’ processes for certification. Therefore, we continue to believe these recommendations are appropriate.

Agencies that lacked finalized policies may not be able to meet their reported time frames for finalizing their certification policies, since agency officials have noted that their approval processes are quite lengthy, and in some cases, the proposed dates for completion have changed several times. In addition, several policies were still being developed. Therefore, we cannot be assured that these documents will fully address the areas we noted. Until the 20 agencies update or finalize processes and policies for CIO certification, including defining the role of the CIO in the process, describing how certification will be documented, and including definitions of incremental development and delivery time frames consistent with OMB guidance, they will not be able to fully ensure adequate implementation of, or benefit from, incremental development practices. As a result, the agencies increase the risk that federal government resources will not be used in the most effective and efficient manner.

OMB Has Improved Its IT Capital Planning Guidance to Ensure CIO Certification Reporting Clearly Specifies Agency Responsibilities

FITARA states that OMB is to require in an agency’s annual IT capital planning guidance that each covered agency CIO certify that IT investments are adequately implementing incremental development, as defined in capital planning guidance issued by OMB. However, since the law was enacted in December 2014, OMB has taken three different approaches to address this reporting requirement. Of the approaches, one did not clearly and consistently provide agencies with the direction needed to effectively implement this important provision and report the status of certification.

As previously noted, OMB’s fiscal year 2017 IT capital planning guidance (issued in June 2015) required each major IT investment to respond to a question in the associated major IT business case regarding whether the CIO certified the adequate implementation of incremental development with either a yes, no, or not applicable. This reporting approach required that agency CIOs provide an explicit statement regarding the certification of adequate implementation of incremental development for each major IT investment. Further, this approach allowed for the status of CIO certification of each investment to be publicly reported on the IT Dashboard via the investment’s major IT business case.

However, OMB’s capital planning guidance for fiscal year 2018 (issued in June 2016) lacked clarity regarding how agencies were to address the requirement certifying adequate incremental development. While the 2018 guidance states that agency CIOs are to provide the certifications needed to demonstrate compliance with FITARA, there is no specific reference to the provision requiring CIO certification of adequate incremental development. As a result of this change, OMB placed the burden on agencies to know and understand how to demonstrate compliance with FITARA’s incremental development provision. Further, because of the lack of clarity in the guidance as to what agencies were to provide, OMB could not demonstrate how the fiscal year 2018 guidance ensured that agencies provided the certifications specifically called for in the law.

OMB staff explained that the changes to the fiscal year 2018 capital planning guidance were made with the intent to rely on agencies’ reported responses on the IT Dashboard regarding the use of incremental development by an investment’s projects, rather than relying on an agency’s response to the yes, no, or not applicable question about the status of an investment’s certification of incremental development.

Providing a clear and consistent approach for agencies to follow in reporting the status of certification is critical to ensure that agencies are able to comply with this key FITARA provision and to ensure that CIOs are held accountable for the performance of their major IT investments. OMB staff from the Office of E-Government and Information Technology stated that the fiscal year 2019 guidance would be responsive to the issues we raised.

Accordingly, in August 2017, OMB issued its fiscal year 2019 guidance, which addressed the weaknesses we identified in the previous fiscal year’s guidance. Specifically, the revised guidance requires agency CIOs to make an explicit statement regarding the extent to which the CIO is able to certify the use of incremental development, and to include a copy of that statement in the agency’s public congressional budget justification materials. As part of the statement, an agency CIO must also identify which specific bureaus or offices are using incremental development on all of their investments.

Agency CIO certification of the use of adequate incremental development for major IT investments is critical to ensuring that agencies are making the best effort possible to create IT systems that add value while reducing the risks associated with low-value and wasteful investments. These changes to OMB’s fiscal year 2019 guidance provide a key improvement for ensuring that agency CIOs have a consistent approach to follow in providing the certifications specifically called for in the law.

Conclusions

One of the aims of FITARA was to encourage the use of incremental development throughout the federal government and, as of August 2016, more than half of the 24 agencies’ IT investments had been certified as adequately implementing incremental development, as required by FITARA and defined in OMB guidance. However, a number of responses for agency investments were incorrectly reported and it will be critical that agencies continue to improve the accuracy of investment data reported on the IT Dashboard. While we have previously made recommendations to numerous agencies to improve the accuracy of reporting on the IT Dashboard, issues with reporting remain, reinforcing the need for agencies to ensure that accurate data are made available for the oversight and management of their investments.

In addition, while OMB issued guidance in June 2015, requiring agency CIOs to define policies and processes for CIO certification, as of August 2017, only 4 of 24 agencies had established policies that clearly define these processes. At this point, over 2 years since the law’s enactment, it is critical that agencies take action to put in place appropriate incremental certification polices to ensure CIOs exercise the proper authority and oversight over major IT investments, as required by law. Otherwise, agencies run the risk of not realizing the benefits of incremental development, as well as not implementing FITARA’s requirement for incremental development. While we previously made recommendations to Defense, Education, HHS, and Treasury to establish CIO certification policies, these agencies have still not yet finalized their guidance, and therefore, we continue to believe these recommendations are appropriate.

Further, OMB has taken three different approaches to addressing FITARA’s reporting requirement for CIO certification and one did not clearly and consistently provide agencies with the direction needed to effectively implement this important provision and report the status of certification. OMB’s fiscal year 2017 capital planning guidance was helpful to agencies, in that it clearly directed agencies on how to publicly report their certifications. This also helped Congress in its oversight of agencies’ FITARA compliance. In contrast, OMB’s fiscal year 2018 capital planning guidance was a step backward, and OMB could not demonstrate how the guidance ensured that agencies provided the certifications specifically called for in the law. Going forward, the changes in guidance that OMB has implemented for fiscal year 2019 recognize the importance of providing clear direction to CIOs and how critical it is for agencies to create IT systems that add value while reducing the risks associated with low-value and wasteful investments.

Recommendations for Executive Action

We are making a total of 19 recommendations to 17 departments and agencies in our review. Specifically: The Secretary of Energy should ensure that the CIO of Energy reports major IT investment information related to incremental development accurately in accordance with OMB guidance. (Recommendation 1)

The Secretary of Agriculture should ensure that the CIO of USDA reports major IT investment information related to incremental development accurately in accordance with OMB guidance. (Recommendation 2)

The Commissioner of the Social Security Administration should ensure that the CIO of SSA reports major IT investment information related to incremental development accurately in accordance with OMB guidance. (Recommendation 3)

The Secretary of Housing and Urban Development should ensure that the CIO of HUD establishes an agency-wide policy and process for the CIO’s certification of major IT investments’ adequate use of incremental development, in accordance with OMB’s guidance on the implementation of FITARA, and confirm that it includes: a description of the CIO’s role in the certification process; a description of how CIO certification will be documented; and a definition of incremental development and time frames for delivering functionality, consistent with OMB guidance. (Recommendation 4)

The Secretary of the Interior should ensure that the CIO of Interior updates the agency’s policy and process for the CIO’s certification of major IT investments’ adequate use of incremental development, in accordance with OMB’s guidance on the implementation of FITARA, and confirm that it includes: a description of the CIO’s role in the certification process; a description of how CIO certification will be documented; and a definition of incremental development, consistent with OMB guidance. (Recommendation 5)

The Attorney General of the United States should ensure that the CIO of Justice establishes an agency-wide policy and process for the CIO’s certification of major IT investments’ adequate use of incremental development, in accordance with OMB’s guidance on the implementation of FITARA, and confirm that it includes: a description of the CIO’s role in the certification process; a description of how CIO certification will be documented; and a definition of incremental development and time frames for delivering functionality, consistent with OMB guidance. (Recommendation 6)

The Secretary of Labor should ensure that the CIO of Labor updates the agency’s policy and process for the CIO’s certification of major IT investments’ adequate use of incremental development, in accordance with OMB’s guidance on the implementation of FITARA, and confirm that it includes a description of the CIO’s role in the certification process and a description of how CIO certification will be documented. (Recommendation 7)

The Secretary of State should ensure that the CIO of State updates the agency’s policy and process for the CIO’s certification of major IT investments’ adequate use of incremental development, in accordance with OMB’s guidance on the implementation of FITARA, and confirm that it includes: a description of the CIO’s role in the certification process; a description of how CIO certification will be documented; and a definition of incremental development and time frames for delivering functionality, consistent with OMB guidance. (Recommendation 8)

The Secretary of Agriculture should ensure that the CIO of USDA establishes an agency-wide policy and process for the CIO’s certification of major IT investments’ adequate use of incremental development, in accordance with OMB’s guidance on the implementation of FITARA, and confirm that it includes: a description of the CIO’s role in the certification process; a description of how CIO certification will be documented; and a definition of incremental development and time frames for delivering functionality, consistent with OMB guidance. (Recommendation 9)

The Secretary of Veterans Affairs should ensure that the CIO of VA updates the agency’s policy and process for the CIO’s certification of major IT investments’ adequate use of incremental development, in accordance with OMB’s guidance on the implementation of FITARA, and confirm that it includes a description of the CIO’s role in the certification process and a description of how CIO certification will be documented. (Recommendation 10)

The Administrator of EPA should ensure that the CIO of EPA establishes an agency-wide policy and process for the CIO’s certification of major IT investments’ adequate use of incremental development, in accordance with OMB’s guidance on the implementation of FITARA, and confirm that it includes: a description of the CIO’s role in the certification process; a description of how CIO certification will be documented; and a definition of incremental development and time frames for delivering functionality, consistent with OMB guidance. (Recommendation 11)

The Administrator of GSA should ensure that the CIO of GSA updates the agency’s policy and process for the CIO’s certification of major IT investments’ adequate use of incremental development, in accordance with OMB’s guidance on the implementation of FITARA, and confirm that it includes a description of the CIO’s role in the certification process and a description of how CIO certification will be documented. (Recommendation 12)

The Administrator of NASA should ensure that the CIO of NASA establishes an agency-wide policy and process for the CIO’s certification of major IT investments’ adequate use of incremental development, in accordance with OMB’s guidance on the implementation of FITARA, and confirm that it includes: a description of the CIO’s role in the certification process; a description of how CIO certification will be documented; and a definition of incremental development and time frames for delivering functionality, consistent with OMB guidance. (Recommendation 13)

The Director of the NSF should ensure that the CIO of NSF updates the agency’s policy and process for the CIO’s certification of major IT investments’ adequate use of incremental development, in accordance with OMB’s guidance on the implementation of FITARA, and confirm that it includes: a description of the CIO’s role in the certification process; a description of how CIO certification will be documented; and a definition of incremental development and time frames for delivering functionality, consistent with OMB guidance. (Recommendation 14)

The Chairman of NRC should ensure that the CIO of NRC establishes an agency-wide policy and process for the CIO’s certification of major IT investments’ adequate use of incremental development, in accordance with OMB’s guidance on the implementation of FITARA, and confirm that it includes a description of the CIO’s role in the certification process and a description of how CIO certification will be documented. (Recommendation 15)

The Director of OPM should ensure that the CIO of OPM updates the agency’s policy and process for the CIO’s certification of major IT investments’ adequate use of incremental development, in accordance with OMB’s guidance on the implementation of FITARA, and confirm that it includes a description of the CIO’s role in the certification process and a description of how CIO certification will be documented. (Recommendation 16)

The Administrator of SBA should ensure that the CIO of SBA establishes an agency-wide policy and process for the CIO’s certification of major IT investments’ adequate use of incremental development, in accordance with OMB’s guidance on the implementation of FITARA, and confirm that it includes: a description of the CIO’s role in the certification process; a description of how CIO certification will be documented; and a definition of incremental development and time frames for delivering functionality, consistent with OMB guidance. (Recommendation 17)

The Commissioner of the Social Security Administration should ensure that the CIO of SSA updates the agency’s policy and process for the CIO’s certification of major IT investments’ adequate use of incremental development, in accordance with OMB’s guidance on the implementation of FITARA, and confirm that it includes a description of the CIO’s role in the certification process and a description of how CIO certification will be documented. (Recommendation 18)

The Administrator of USAID should ensure that the CIO of USAID establishes an agency-wide policy and process for the CIO’s certification of major IT investments’ adequate use of incremental development, in accordance with OMB’s guidance on the implementation of FITARA, and confirm that it includes: a description of the CIO’s role in the certification process; a description of how CIO certification will be documented; and a definition of incremental development and time frames for delivering functionality, consistent with OMB guidance. (Recommendation 19)

Agency Comments and Our Evaluation

We received comments on a draft of this report from OMB and the 24 agencies that we reviewed. Of the 17 agencies to which we made recommendations, 11 agencies agreed with our recommendations, 1 agency partially agreed, and 5 agencies did not state whether they agreed or disagreed with the recommendations.

In addition, of the 7 agencies and OMB to which we did not make recommendations, 2 agencies agreed with the report and 5 agencies stated that they had no comments on the report. OMB did not agree with certain findings in the report. In addition, OMB and multiple agencies provided technical comments on the report, which we incorporated as appropriate.

The following discusses the comments received from each agency to which we made a recommendation.

In written comments, Energy concurred with our recommendation to ensure that the CIO reports major IT investment information related to incremental development accurately in accordance with OMB guidance, and described actions it has taken to address the recommendation. Specifically, the agency stated that its Office of the CIO reviews the accuracy of Energy’s major IT investment project reporting related to incremental development as part of monthly IT Dashboard and Investment Review Board meetings. By taking these actions, the agency considered the recommendation closed.

As noted earlier in our report, we identified issues with the accuracy of Energy’s reported data related to the certification of incremental development. If Energy consistently and effectively implements its reviews of IT Dashboard data, as described, these actions should help to improve the accuracy of reported incremental development data on the IT Dashboard. We plan to continue to monitor the agency’s reporting of its incremental data on the IT Dashboard and accordingly, consider our recommendation to currently remain open. Energy’s comments are reprinted in appendix IV.

In written comments, HUD concurred with our recommendation to establish an agency-wide policy and process for CIO certification of adequate incremental development and stated that it would provide more definitive information and timelines on how it plans to address the recommendation once our final report is issued. HUD’s comments are reprinted in appendix V.

In written comments, Interior stated that the agency concurred with our recommendation to update the agency’s policy and process for CIO certification of adequate incremental development and described planned actions to implement it. Specifically, the agency reported that it is committed to updating its existing policy to include a description of the CIO’s role in the incremental development certification process, a description of how the CIO’s certification is documented, and a definition of incremental development, consistent with OMB’s guidance. Interior’s comments are reprinted in appendix VI.

In an e-mail received on September 15, 2017, an audit liaison specialist in Justice’s Audit Liaison Group in the Internal Review and Evaluation Office stated that the agency agreed with our recommendation to establish an agency-wide policy and process for CIO certification of adequate incremental development and described planned actions to implement it. Specifically, the official stated that Justice will amend existing policy and processes to implement this recommendation. In addition, the official stated that Justice is fully supportive of incremental development and has drafted documentation, including guidance on an incremental system development life cycle.

In an e-mail received on September 5, 2017, an administrative officer in Labor’s Office of the Assistant Secretary for Administration and Management stated that the agency had no comments on the report.

In written comments, State did not say whether the agency agreed or disagreed with our recommendation to update the agency’s policy and process for CIO certification of adequate incremental development, but described ongoing actions to implement it. Specifically, the agency reported that it has developed an incremental development policy that addresses the recommendation we noted in our report. The agency added that the policy is currently in the process of being approved. State’s comments are reprinted in appendix VII.

In an e-mail received on September 1, 2017, a senior advisor in the USDA Office of the CIO’s Enterprise Management office stated that the agency concurred with our findings and recommendations to report major IT investment incremental development information accurately and to establish an agency-wide policy and process for CIO certification of adequate incremental development, and had no further comments.

In written comments, VA partially concurred with our recommendation to update the agency’s policy and process for CIO certification of adequate incremental development, stating that, while the agency does not currently have a policy in place outlining the CIO certification process, the agency CIO does direct that all investments utilize Agile and incremental delivery. The agency stated that it would take action to address our recommendation by drafting a policy that outlines the CIO’s role in the certification process and describes how certification will be documented. The agency added that the policy is targeted for completion by November 2017.

If implemented as planned, these actions should address the intent of our recommendation. VA’s comments are reprinted in appendix VIII.

In written comments, EPA stated that the agency generally agreed with our recommendation to establish an agency-wide policy and process for CIO certification of adequate incremental development, and presentation of facts in the report. The agency also noted that the policy developed in response to our recommendation is to address FITARA issues above and beyond the certification of incremental development.

In addition, the agency noted a technical correction to a sentence in our report related to EPA’s use of information from certification. We have incorporated changes to the draft, as appropriate, to address this comment. EPA’s comments are reprinted in appendix IX.

In written comments, GSA agreed with our recommendation to update the agency’s policy and process for CIO certification of adequate incremental development and reported that it would develop and implement a plan to fully address it. GSA’s comments are reprinted in appendix X.

In written comments, NASA concurred with the recommendation to establish an agency-wide policy and process for CIO certification of adequate incremental development and described ongoing actions to implement it. Specifically, the agency stated that it is currently updating its policies to address the incremental development requirement. In this regard, NASA Policy Directive 2800.1 is to include a responsibility for the Office of the CIO to certify that IT resources are adequately implementing incremental development. In addition, NASA Policy Directive 7120.7 is being updated to include a definition of incremental development and processes for ensuing that the CIO certifies incremental development. According to the agency, these policies are estimated to be completed by March 2018. NASA’s comments are reprinted in XI.

In an e-mail received on September 14, 2017, a senior advisor in NSF’s Office of the Director/Office of Integrative Activities stated that the agency had no comments on our report.

In written comments, NRC stated that it was in general agreement with the findings in our report. The agency did not state whether it agreed or disagreed with our recommendation to establish an agency- wide policy and process for CIO certification of adequate incremental development, but described the planned action to implement the recommendation. Specifically, the agency reported that it plans to establish agency-wide, formalized processes and procedures for the CIO to approve the incremental development of major IT investments by December 31, 2017. NRC’s comments are reprinted in appendix XII.

In written comments, OPM concurred with the recommendation to update the agency’s policy and process for CIO certification of adequate incremental development and described planned actions to implement it. Specifically, the agency reported that it intends to update its policies and processes to include a description of the CIO’s role in the certification process and a description of how certification will be documented. OPM’s comments are reprinted in appendix XIII.

In an e-mail received on September 11, 2017, a program manager in SBA’s Office of Congressional and Legislative Affairs stated that the agency concurred with our recommendation to establish an agency- wide policy and process for CIO certification of adequate incremental development, and had no further comments.

In written comments, SSA agreed with our two recommendations to report major IT investment incremental development information accurately and establish an agency-wide policy and process for CIO certification of adequate incremental development, and described planned actions being taken or planned to implement them. Specifically, the agency reported that it had implemented two new processes to support incremental development certification. According to the agency, each IT investment program manager is to answer a series of questions about the investment’s status and also certify whether their investment adequately implements incremental development. This information is to be used in the CIO’s ongoing investment evaluation process for reporting investment information on the IT Dashboard. SSA reported that these new processes are to be defined in an upcoming revision to the agency’s Capital Planning and Investment Control Guide. SSA’s comments are reprinted in appendix XIV.

In written comments, USAID did not state whether it agreed or disagreed with our recommendation to establish an agency-wide policy and process for CIO certification of adequate incremental development, but described ongoing actions to implement the recommendation. Specifically, the agency reported that it is in the process of establishing an agency-wide policy and process for the CIO’s certification of adequate incremental development. It estimates that this policy will be implemented by August 31, 2018. USAID’s comments are reprinted in appendix XV.

In addition to the aforementioned comments, the seven agencies and OMB to which we did not make recommendations provided the following responses.

In written comments, Commerce stated that the agency concurred with the report as written. Commerce’s comments are reprinted in appendix XVI.

In an e-mail received on September 7, 2017, a GAO Affairs staff member in Defense’s Executive Services Directorate stated that the agency had no formal comments on the report.

In an e-mail received on September 8, 2017, a staff member in Education’s Office of the Secretary/Executive Secretariat stated that the agency had no comments on the report.

In an e-mail received on September 11, 2017, an audit liaison in HHS’s Office of the Assistant Secretary for Legislation stated that the agency had no comments on the report.

In an e-mail received on September 11, 2017, a program analyst in DHS’s GAO-Office of Inspector General’s Liaison Office stated that the agency would not be sending a management response letter.

In an e-mail received on September 8, 2017, the Director of Audit Relations and Program Improvement in Transportation’s Office of the Secretary stated that the agency would not be providing a written management response.

In an e-mail received on September 15, 2017, a supervisory IT specialist/GAO-Office of Inspector General liaison in Treasury’s Office of the CIO stated that the agency generally agreed with the report. The agency also provided comments related to various challenges discussed in the report. Specifically, the official described Treasury’s efforts to address challenges noted in the report related to project staff lacking the necessary skills for implementing incremental development practices and programs not receiving sufficient funding. In this regard, the official stated that the agency continues to develop knowledge, skills, and abilities for project managers and IT specialists and continues to provide specialized programming training to its IT staff in order to move to more modern programming languages and IT tools as part of system modernization efforts. In addition, the official stated that, to address challenges related to programs receiving sufficient funding, Treasury continues to adjust planned and ongoing projects to align with the availability of funds and external mandates.

In an e-mail received on September 19, 2017, an OMB Assistant General Counsel stated that the agency generally disagreed with the tone, tenor, and conclusions of law reflected in aspects of our report. Among the concerns was that we had asserted that OMB’s prior year’s guidance to agencies on CIO certification of incremental development was not in compliance with OMB’s statutory obligations under FITARA.

As our report states, FITARA mandates OMB to include in its annual IT capital planning guidance, a requirement that CIOs certify that investments are adequately implementing incremental development as defined in the guidance. We reported that OMB had issued guidance for fiscal years 2017, 2018, and 2019. However, we noted that the fiscal year 2018 guidance differed from the guidance issued in the other two fiscal years in that it did not clearly establish how agency CIOs were to demonstrate compliance with FITARA’s certification of adequate incremental development provision. Instead, the fiscal year 2018 guidance placed the burden on agencies to know and understand how to implement the FITARA requirement.

Thus, while we concluded that OMB’s fiscal year 2018 guidance was not clear on how agencies were to certify adequate incremental development, we did not assert that this guidance failed to comply with FITARA. Accordingly, we did not make a conclusion of law regarding OMB’s guidance, as the e-mail stated. We continue to believe that our assessment of the fiscal year 2018 guidance is correct.

OMB also stated that it disagreed with our conclusion that OMB could not demonstrate compliance with FITARA.

However, our report did not make the conclusion that is stated in OMB’s response. As noted above, our report pointed out that OMB’s fiscal year 2018 guidance lacked clarity in terms of specifically stating what information agencies were to provide OMB in order to be compliant with FITARA’s requirement that agency CIOs certify incremental development. Therefore, we concluded that OMB could not demonstrate how the fiscal year 2018 guidance ensured that agencies provided the certifications specifically called for in the law. As such, we continue to believe that our conclusion is appropriate.

Further, OMB stated that our conclusion was predicated on OMB’s reluctance to share agency pre-decisional budget information.

It is up to OMB to demonstrate that its fiscal year 2018 guidance ensured agency compliance with FITARA. Though OMB asserted that our conclusion was based on OMB’s reluctance to share agency pre- decisional budget information, our conclusion was instead based on the fact that OMB provided no documentary evidence to establish how agencies complied with the FITARA certification requirement for fiscal year 2018. Consequently, we believe our assessment that OMB could not demonstrate how the fiscal year 2018 guidance ensured that agencies provided the certifications specifically called for in the law is accurate.

In a subsequent e-mail to us on October 4, 2017, the OMB Assistant General Counsel provided additional comments related to the disagreements described above. Specifically, OMB stated that our report’s “focus on the use of the term ‘certification’ was confusing in that appears to reference the term ‘certify’ [found in the FITARA provision on the adequate use of incremental development], and also seems to be a reference to the requirement that CIOs ‘approve’ and define development processes.”

In our report, we discuss FITARA’s requirement that OMB annually issue capital planning guidance requiring agency CIOs to certify that IT investments are adequately implementing incremental development. We analyzed the guidance that OMB has issued to meet this requirement over the past 3 years, and we evaluated agencies’ progress in implementing that guidance. In doing so, we noted that OMB had also issued supplementary FITARA implementation guidance in June 2015 that required agencies to define policies and processes to ensure that the CIO certifies that IT resources are adequately implementing incremental development. Throughout our discussion, we clearly delineate between the incremental development certification provided to OMB by an agency’s CIO and the agency’s policies and processes that support and inform that certification. As such, we believe we have used the term “certification” appropriately and consistently throughout our report.

We are sending copies of this report to interested congressional committees, the Director of the Office of Management and Budget, the Secretaries and agency heads of the departments and agencies in this report, and other interested parties. This report will also be available at no charge on our website at http://www.gao.gov.

If you or your staffs have any questions on matters discussed in this report, please contact me at (202) 512-9286 or pownerd@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix XVII.

Appendix I: Objectives, Scope, and Methodology

Our objectives for this engagement were to determine (1) the number of investments certified by agencies as implementing adequate incremental development and any reported challenges that impact the agencies’ incremental delivery of functionality; and (2) whether agencies are establishing policies and processes for chief information officer (CIO) certification of incremental development in accordance with the Federal Information Technology Acquisition Reform Act provisions (commonly referred to as FITARA) enacted as a part of the Carl Levin and Howard P. ‘Buck’ McKeon National Defense Authorization Act for Fiscal Year 2015.

For our first objective, we obtained and analyzed major information technology (IT) investment data reported by agencies on the IT Dashboard as of August 31, 2016, for fiscal year 2017, which was the first year that the Office of Management and Budget (OMB) required the 24 covered agencies to report the status of CIO certification of incremental development for each investment. We chose this date because it was the final day updated fiscal year 2017 data from the agencies would be publicly available until the release of the President’s fiscal year 2018 budget submission.

Initially, we analyzed the fiscal year 2017 data of major IT software development investments that were planning to allocate at least 50 percent of their funding to development, modernization, and enhancement activities. We then reviewed agency responses to the question regarding CIO certification of adequate incremental development and eliminated any investment where the agency’s rationale for choosing “not applicable” was due to the investment not undertaking software development activities. In doing so, we identified a total of 166 investments from 21 agencies. Three agencies (National Aeronautics and Space Administration, National Science Foundation, and U.S. Nuclear Regulatory Commission) out of the 24 in our review did not have any investments that met these criteria for fiscal year 2017. For the 21 agencies with major IT investments to review, we then determined the total number of investments that agencies reported were certified by the CIO for adequate incremental development. We also reviewed and summarized agency responses reported on the IT Dashboard for investments that did not have CIO certification.

To help determine the reliability of the reported agency CIO certification data on the IT Dashboard, we presented the results of our analysis of CIO certification responses to officials from each agency’s Office of the CIO that were involved in investment management and software development activities and solicited their input and explanations for the results. Two agencies each provided an update on one of their investments, which we have incorporated as appropriate. We determined that the data were sufficiently reliable for the purpose of this report.

In order to identify the challenges impacting the agencies’ incremental delivery of functionality, we developed a list of common challenges based on our prior work, in which eight agencies reported that the following eight challenges inhibited their delivery of functionality: 1. project staff were over-utilized or lacked the necessary skills and 2. programs did not receive sufficient funding or received funding later 3. projects experienced management and organizational challenges that introduced delays; 4. development work was slowed by inefficient governance and 5. project characteristics made rapid delivery of functionality infeasible or 6. development schedules were impeded by procurement delays; 7. programs did not have stable, prioritized requirements; and 8. incremental development was impeded by select technologies.

We sent the list of challenges to each of the 24 agencies and asked officials from the Office of the CIO at each agency involved with investment management and software development activities to identify their top three challenges from this list that impacted their ability to deliver incremental functionality for major IT investments. We also asked agency officials to identify any challenges that were not included in the list, but which were also among their top three challenges. Finally, we asked agencies to explain what actions were taken to address the reported challenges and describe the extent to which the challenges were overcome.

Because of the open-ended nature of the agencies’ responses to our questions, we conducted a content analysis of the information we received in order to identify common challenges that impact agencies’ ability to deliver incremental functionality. In doing so, team members individually reviewed the challenges reported by agencies and assigned them to various categories. Team members then compared categorization schemes, discussed the differences, and reached agreement on the final list of challenges by totaling the number of times each challenge was mentioned. For those challenges that were prompted by the list we provided to agencies, we reported challenges that were identified by five or more agencies. Three agencies also identified a new challenge that was not on our list, which we reported due to the number of agencies reporting it as a challenge. Three of the 24 agencies in our review (Departments of Defense, Energy, and Health and Human Services) reported that they had no challenges with implementing incremental development.

We also asked the agencies in our review how the CIO utilized the information obtained during the process of certifying investments’ adequate incremental development to make decisions regarding the agency’s major IT investments. Because of the wide variety of responses we received from agencies, we conducted a content analysis of the information in order to identify ways the CIOs used the information. In doing so, team members individually reviewed agencies’ responses and assigned them to various categories. Team members then compared their categorization schemes, discussed the differences, and reached agreement on the final characterization of ways in which agencies benefited from the certification process.

For our second objective, we analyzed the 24 agencies’ policies and processes governing the CIO certification of adequate incremental development to determine whether those policies and processes were consistent with FITARA. The provision states that OMB is to require in its annual IT capital planning guidance that agency CIOs covered by the law certify that IT investments are adequately implementing incremental development. To assess this, we reviewed guidance issued by OMB on the implementation of FITARA, and assessed agencies’ documentation of incremental development certification policies and processes against GAO’s IT investment management framework. This framework states that an organization’s policies and procedures should be clearly defined, in that they provide details regarding the role of appropriate stakeholders and the artifacts to document decisions made.

Because of the wide variety of responses and documents we received from agencies related to their incremental development certification processes, we conducted a content analysis of the information in order to determine compliance with OMB’s guidance. In doing so, team members individually reviewed agencies’ responses and documents and assigned them to various categories and sub-categories. Team members then compared their categorization schemes, discussed the differences, and reached agreement on the final characterization of compliance with OMB guidance. In cases where agencies provided multiple policies or documents, we followed up to clarify which portions were considered by the agency to support the CIO certification requirement.

In analyzing whether the agencies’ policies on CIO certification met FITARA, OMB, and GAO criteria, we assessed whether the policies clearly defined the role of the CIO in the certification of adequate incremental development, and described how CIO certification was documented. We also reviewed agencies’ incremental development policies and processes to identify the agencies’ definitions of incremental development and time frames for delivering functionality to determine whether they were consistent with OMB guidance. Agencies found to not have a policy where the CIO process was clearly defined were evaluated as such for one of two reasons: either the agency’s formal policy did not completely address our assessment criteria or the agency’s policy had not yet been finalized. For agencies that told us they had not yet finalized a policy for certification, we asked them to explain the process, if any, used by the agency to certify major IT investments for fiscal year 2017. In addition, we interviewed staff from OMB’s Office of E-Government and Information Technology regarding its guidance to agencies related to FITARA’s incremental development certification provision.

We conducted this performance audit from July 2016 to November 2017 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Federal Agency Major IT Investments’ Reported Chief Information Officer Certification of Incremental Development on the IT Dashboard for Fiscal Year 2017

Table 5 lists the 166 major information technology (IT) software development investments primarily in development, as reported on the IT Dashboard as of August 31, 2016, and the agency’s reported response to the question in the major IT business case regarding whether the agency’s Chief Information Officer certified the adequate use of incremental development for the investment for fiscal year 2017. All 166 investments reported in the major IT business case that the investment included software development.

Appendix III: Analysis of Federal Agency Chief Information Officer Incremental Development Certification Policies

Table 6 shows our analysis regarding whether the agency had policies and processes that clearly defined the Chief Information Officer (CIO) certification process for the adequate use of incremental development, including: (1) describing the CIO’s role in the certification process; (2) describing how CIO certification is to be documented; (3) having a definition of incremental development in the policy consistent with Office of Management and Budget (OMB) guidance; and (4) having time frames for delivering functionality in the policy consistent OMB guidance.

Appendix V: Comments from the Department of Housing and Urban Development

Appendix VI: Comments from the Department of the Interior

Appendix VII: Comments from the Department of State

Appendix VIII: Comments from the Department of Veterans Affairs

Appendix IX: Comments from the Environmental Protection Agency

Appendix X: Comments from the General Services Administration

Appendix XI: Comments from the National Aeronautics and Space Administration

The report number GAO-17-556 has been changed to GAO-18-148.

Appendix XII: Comments from the U.S. Nuclear Regulatory Commission

Appendix XIII: Comments from the Office of Personnel Management

Appendix XIV: Comments from the Social Security Administration

The report number GAO-17-556 has been changed to GAO-18-148.

Appendix XV: Comments from the U.S. Agency for International Development

The report number GAO-17-556 has been changed to GAO-18-148.

Appendix XVI: Comments from the Department of Commerce

Appendix XVII: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the individual named above, the following staff made key contributions to this report: Dave Hinchman (Assistant Director), Chris Businsky, Rebecca Eyler, Justin Fisher, Valerie Hopkins (Analyst in Charge), Sandra Kerr, James MacAulay, Jamelyn Payan, Priscilla Smith, and Andrew Stavisky.