Information Technology: Agencies Need to Establish and Implement Incremental Development Policies
Released: 2014-05-08T00:00:00

Letter

Background

If done correctly, investments in IT have the potential to make organizations more efficient in fulfilling their missions. For example, we recently reported that Defense officials stated that an IT system supporting military logistics has improved the organization’s performance by providing real-time information about road conditions, construction, incidents, and weather to facilitate rapid deployment of military assets.

However, as we have previously reported, investments in federal IT too frequently result in failed projects that incur cost overruns and schedule slippages while contributing little to mission-related outcomes. For example: In January 2011, the Secretary of Homeland Security ended the Secure Border Initiative Network program after obligating more than $1 billion for the program because it did not meet cost-effectiveness and viability standards. Since 2007, we have identified a range of issues and made several recommendations to improve this program. For example, in May 2010, we reported that the final acceptance of the first two deployments had slipped from November 2009 and March 2010 to September 2010 and November 2010, respectively, and that the cost-effectiveness of the system had not been justified. We concluded that DHS had not demonstrated that the considerable time and money being invested to acquire and deploy the program were a wise and prudent use of limited resources. As a result, we recommended that the department (1) limit near-term investment in the program, (2) economically justify any longer-term investment in it, and (3) improve key program management disciplines. This work contributed to the department’s decision to cancel the program.

In February 2011, the Office of Personnel Management canceled its Retirement Systems Modernization program after several years of trying to improve the implementation of this investment.the Office of Personnel Management, it spent approximately $231 million on this investment. We issued a series of reports on the According to agency’s efforts to modernize its retirement system and found that the Office of Personnel Management was hindered by weaknesses in several important management disciplines that are essential to successful IT modernization efforts. Accordingly, we made recommendations in areas such as project management, organizational change management, testing, and cost estimating. In May 2008, an Office of Personnel Management official cited the issues that we identified as justification for issuing a stop work order to the system contractor, and the agency subsequently terminated the contract.

In December 2012, Defense canceled the Air Force’s Expeditionary Combat Support System after having spent more than a billion dollars and missing multiple milestones, including failure to achieve deployment within 5 years of obligating funds. We issued several reports on this system and found that, among other things, the program was not fully following best practices for developing reliable schedules and cost estimates.

Agencies have reported that poor-performing projects have often used a “big bang” approach—that is, projects that are broadly scoped and aim to deliver functionality several years after initiation. For example, in 2009 the Defense Science Board reported that Defense’s acquisition process for IT systems—which was rooted in the “waterfall” development model—was too long, ineffective, and did not accommodate the rapid evolution of IT.The board reported that the average time to deliver an initial program capability for a major IT system acquisition at Defense was over 7 years. Also in 2009, VA’s former chief information officer (CIO) reported that many of its projects exceeded cost estimates by more than 50 percent and missed scheduled completion dates by more than a year. That official concluded that VA needed to make substantial changes to its acquisition process in order to eliminate project failures associated with the “big bang” approach.

One approach to reducing the risks from broadly scoped, multiyear projects is to divide investments into smaller parts—a technique long advocated by Congress and OMB. can potentially By following this approach, agencies deliver capabilities to their users more rapidly, giving them more flexibility to respond to changing agency priorities; increase the likelihood that each project will achieve its cost, schedule, and performance goals; obtain additional feedback from users, increasing the probability that each successive increment and project will meet user needs; more easily incorporate emerging technologies; and terminate poorly performing investments with fewer sunk costs.

See Clinger-Cohen Act of 1996, Pub. L. No. 104-106 § 5202, 110 Stat. 186, 690 (1996), codified at 41 U.S.C. § 2308; see also 48 C.F.R. § 39.103 (Federal Acquisition Regulation); OMB, Management of Federal Information Resources, Circular No. A-130 Revised. implement them. More recently, in its 2010 IT Reform Plan, OMB called for IT programs to deliver functionality at least every 12 months and complete initial deployment to end users no later than 18 months after the start of the program. In 2011, as part of its budget guidance, OMB first recommended that projects associated with major IT investments deliver functionality every 6 months. OMB’s latest guidance now makes this mandatory; specifically, in 2012, OMB began requiring that functionality be delivered at least every 6 months.

Roles and Responsibilities for Overseeing Federal IT Investments

Over the last three decades, Congress has enacted several laws to assist agencies and the federal government in managing IT investments. For example, the Paperwork Reduction Act of 1995 requires that OMB develop and oversee policies, principles, standards, and guidelines for federal agency IT functions, including periodic evaluations of major information systems. In addition, to assist agencies in managing their investments, Congress enacted the Clinger-Cohen Act of 1996. Among other things, the act requires agency heads to appoint CIOs and specifies many of their responsibilities. With regard to IT management, CIOs are responsible for implementing and enforcing applicable governmentwide and agency IT management principles, standards, and guidelines; assuming responsibility and accountability for IT investments; and monitoring the performance of IT programs and advising the agency head whether to continue, modify, or terminate such programs. Additionally, with regard to incremental development, Clinger-Cohen calls for provisions in the Federal Acquisition Regulation that encourage agencies to structure their IT contracts such that the capabilities are delivered in smaller increments. The Federal Acquisition Regulation provisions are also to provide, to the maximum extent practicable, that the increment should be delivered within 18 months of the contract solicitation.

As set out in these laws, OMB is to play a key role in helping federal agencies manage their investments by working with them to better plan, justify, and determine how much they need to spend on projects and how to manage approved projects. Within OMB, the Office of E-Government and Information Technology, headed by the Federal CIO, directs the policy and strategic planning of federal IT investments and is responsible for oversight of federal technology spending.

In carrying out its responsibilities, OMB uses several data collection mechanisms to oversee federal IT spending during the annual budget formulation process. Specifically, OMB requires federal departments and agencies to provide information related to their IT investments (called exhibit 53s) and capital asset plans and business cases (called exhibit 300s).

Exhibit 53. The purpose of the exhibit 53 is to identify all IT investments—both major and nonmajor—and their associated costs within a federal organization. Information included in agency exhibit 53s is designed, in part, to help OMB better understand agencies’ spending on IT investments.

Exhibit 300. The purpose of the exhibit 300 is to provide a business case for each major IT investment and to allow OMB to monitor IT investments once they are funded. An IT investment may include one or more projects that are to develop, modernize, enhance, or maintain a single IT asset or group of IT assets with related functionality. Agencies are required to provide information on each major investment’s projects, including cost, schedule, and performance information. For example, in order to measure compliance with its requirement that projects deliver functionality in 6-month cycles, OMB requires agencies to break their projects into activities, and describe when the activities are to deliver functionality.

OMB’s Recent Major Initiatives for Overseeing IT Investments

OMB has implemented a series of initiatives to improve the oversight of underperforming investments and more effectively manage IT. These efforts include the following: IT Dashboard. In June 2009, to further improve the transparency into and oversight of agencies’ IT investments, OMB publicly deployed the IT Dashboard. As part of this effort, OMB issued guidance directing federal agencies to report, via the Dashboard, the performance of their IT investments. Currently, the Dashboard publicly displays information on the cost, schedule, and performance of over 700 major federal IT investments at 26 federal agencies. Further, the public display of these data is intended to allow OMB, other oversight bodies, and the general public to hold the government agencies accountable for results and progress.

TechStat reviews. In January 2010, the Federal CIO began leading TechStats sessions—face-to-face meetings to terminate or turn around IT investments that are failing or are not producing results. These meetings involve OMB and agency leadership and are intended to increase accountability and transparency and improve performance. For example, the Federal CIO testified in June 2013 that he holds TechStat meetings on large investments that are not being acquired incrementally. More recently, OMB empowered agency CIOs to hold their own TechStat sessions within their respective agencies. In doing so, OMB has called for agencies to use their TechStat processes to identify investments that are not being acquired incrementally and undertake corrective actions.

IT Reform Plan. In December 2010, OMB released its 25-point plan to reform federal IT. This document established an ambitious plan for achieving operational efficiencies and effectively managing large- scale IT programs. In particular, as part of its effort to effectively manage IT acquisitions, the plan calls for federal IT programs to deploy functionality in release cycles no longer than 12 months, and ideally, less than 6 months. The plan also identifies key actions that can help agencies implement this incremental development guidance, such as working with Congress to develop IT budget models that align with incremental development, and issuing contracting guidance and templates to support incremental development.

GAO Has Reported on Efforts to Improve IT Acquisitions

In April 2012, we reported on OMB’s efforts to implement the actions called for in its IT Reform Plan and found that it had partially completed work on two key action items relating to incremental development— issuing contracting guidance and templates to support incremental development and working with Congress to create IT budget models that align with incremental development. With respect to the contracting guidance and templates, we found that, although OMB worked with the IT and acquisition community to develop guidance, it had not yet issued this guidance or the templates. Regarding the IT budget models, we found that, although OMB worked to promote ideas for IT budget flexibility (such as multiyear budgets or revolving funds) with congressional committees, there has not yet been any new legislation to create budget models, and OMB has not identified options to increase transparency for programs that would fall under these budgetary flexibilities. We recommended that the Director of OMB ensure that all action items called for in the IT Reform Plan are completed. OMB agreed with this recommendation. OMB has since issued contracting guidance for incremental development,Office of E-Government and Information Technology stated that activities to address the development of new IT budget models are still ongoing. but, as of January 2014, a staff member from the OMB Additionally, in 2011, we identified seven successful investment acquisitions and nine common factors critical to their success. Specifically, we reported that department officials identified seven successful investment acquisitions, in that they best achieved their respective cost, schedule, scope, and performance goals. Notably, all of these were smaller increments, phases, or releases of larger projects. For example, the Defense investment in our sample was the seventh increment of an ongoing investment; the Department of Energy system was the first of two phases; the DHS investment was rolled out to two locations prior to deployment to 37 additional locations; and the Transportation investment had been part of a prototype deployed to four airports. In addition, common factors critical to the success of three or more of the seven investments were: (1) program officials were actively engaged with stakeholders, (2) program staff had the necessary knowledge and skills, (3) senior department and agency executives supported the programs, (4) end users and stakeholders were involved in the development of requirements, (5) end users participated in testing of system functionality prior to formal end user acceptance testing, (6) government and contractor staff were stable and consistent, (7) program staff prioritized requirements, (8) program officials maintained regular communication with the prime contractor, and (9) programs received sufficient funding. These critical factors support OMB’s objective of improving the management of large-scale IT acquisitions across the federal government, and wide dissemination of these factors could complement OMB’s efforts.

Further, in 2012, we identified 32 practices and approaches as effective for applying Agile software development methods to IT projects.Officials from five agencies who had used Agile methods on federal projects cited beneficial practices, such as obtaining stakeholder and customer feedback frequently, managing requirements, and ensuring staff had the proper knowledge and experience. We also identified 14 challenges with adapting and applying Agile in the federal environment, including agencies having difficulty with committing staff to projects, procurement practices that did not support Agile projects, and compliance reviews that were difficult to execute within an iterative time frame. We noted that the effective practices and approaches identified in the report, as well as input from others with broad Agile experience, could help agencies in the initial stages of adopting Agile.

Majority of Selected Agencies’ Incremental Development Policies Have Significant Weaknesses

Since 2000, OMB Circular A-130 has required agencies to (1) develop policies that require their major investments to deliver functionality incrementally and (2) ensure that investments comply with their policies. In addition, as part of its recent budget guidance, OMB has defined how often investments must deliver functionality. Specifically, each project associated with major IT investments is to deliver functionality at least Further, through the President’s Budget, OMB once every 6 months.has provided additional guidance on how incremental development is to be enforced by requiring agencies to use their TechStat processes to identify investments that are not being acquired incrementally and undertake corrective actions.

Although OMB’s guidance requires agencies to develop incremental development policies, it does not specify what those policies are to include. Absent this detail and in reviewing the previously mentioned guidance and leading practices on institutionalizing processes throughout an organization, include in their policies in order to effectively carry out OMB’s incremental development guidance: we identified three components that agencies should require that all projects associated with major IT investments deliver functionality in cycles that are not more than 6-months long; define functionality—that is, what the projects are to deliver at the end of a 6-month cycle; and define a process for ensuring that major IT investments and their projects deliver functionality every 6 months. This should include identifying investments that are not being acquired incrementally through agency TechStat processes and undertaking corrective actions.

Although all five selected agencies developed policies that address incremental development, the majority of the agencies’ policies did not fully address all three components. Specifically, only VA fully addressed the three components; Defense partially addressed the majority of them; and HHS, DHS, and Transportation did not address the majority of components.

Table 1 provides a detailed assessment of each agency’s policies against the three key components of an incremental development policy. In addition, a discussion of each policy component follows the table.

SEI, CMMI-ACQ, Version 1.3 (November 2010). component. ●=Fully met—the agency provided evidence that addressed the component. ◐=Partially met—the agency provided evidence that addressed about half or a large portion of the ○=Not met—the agency did not provide evidence that addressed the component or provided evidence that minimally addressed the component.

Require delivery of functionality every 6 months. Only one of the five agencies—VA—fully addressed this policy component by clearly requiring that its projects be completed in increments that must not exceed 6 months. The other four agencies did not address this policy component. Three of these agencies—Defense, HHS, and DHS—all developed policies that promote the use of incremental development, but these policies do not require functionality to be delivered every 6 months. Specifically, with regard to Defense, although the department’s acquisition framework calls for investments to use incremental development, its policy on IT budget submissions encourages investments to deliver functionality every 12-18 months— not every 6 months. According to officials of the Defense Office of the CIO, 12-18 month incremental development is better aligned with the acquisition framework that many of its IT acquisitions have used.

For HHS, although its policy requires incremental development, its policy recommends—but does not require—that all projects deliver functionality every 3-6 months. According to an HHS Office of the CIO official, HHS does not require its projects to deliver functionality every 6 months because it wants to provide projects with flexibility. For DHS, in June 2012, the former DHS CIO issued a draft policy encouraging IT projects to move towards Agile development approaches. Additionally, with respect to financial systems modernization programs, DHS’s policy calls for providing financial capabilities to the customer in small increments of 6-12 months. According to DHS officials representing the Office of the CIO, the department is currently developing a departmentwide policy on incremental development; however, they said that the draft currently encourages investments to deliver the first release 18 months after program initiation and thereafter deploy functionality in cycles no longer than 12 months, but ideally less than 6 months. Lastly, Transportation also did not address the component because, although Transportation has a policy that calls for projects to deliver functionality every 6 months, officials from the Office of the CIO explained that this policy does not apply to the Federal Aviation Administration (FAA). These officials explained that how often FAA projects deliver functionality depends on their size, scope, risk, visibility, and interdependencies with other programs.

Define functionality. Only one of the five agencies—VA—fully addressed this policy component. VA has a policy that defines what it means to deliver functionality—both in terms of what constitutes an increment and what should be delivered at the end of an increment. For example, VA defines an increment as the segment of the project that produces, in a cycle of 6 months or less, a deliverable that can be used by customers in an operational environment. For the agency that partially addressed the component—Defense—although it has defined functionality for purposes of its acquisition framework, it has not defined the functionality that its IT budget submission policy encourages projects to deliver every 12-18 months. The department stated that it is working with OMB to define this term. Lastly, three agencies—HHS, DHS, and Transportation—had not defined functionality in terms of what they expected projects to deliver at the end of a development cycle. Officials representing these agencies’ respective Office of the CIO acknowledged that they have not defined functionality. These officials told us that they would update their policies to define the term, but officials from HHS and Transportation did not provide a time frame for doing so. DHS officials stated that, although they did not have a definitive timeframe, they hoped to finalize the policy in 2014. Until the agencies define this term, investments may create definitions that are inconsistent with the intent of OMB’s policy.

Define a process for enforcing compliance. Only one of the five agencies—VA—fully addressed this policy component by defining processes for ensuring that increments are structured to deliver functionality every 6 months or less and for reviewing projects that fall behind schedule. In particular, VA’s policy requires the agency to hold a TechStat session when any increment delivery date has been or will be missed. Two agencies partially addressed this component— Defense and HHS—because, although they established processes for ensuring that IT is acquired incrementally, these processes do not (1) require enforcement of incremental development within the specific time frames consistent with OMB guidance (12-18 months for Defense and 3-6 months for HHS) or (2) include using TechStat processes to identify investments that are not being acquired incrementally. Finally, two agencies—DHS and Transportation—have not established processes for enforcing compliance with their incremental development policies. Officials from their respective Office of the CIO told us that they are updating their policies to address this issue. Transportation officials representing the Office of the CIO stated that it would update its policy later this year; DHS Office of the CIO officials stated that, although they did not have a definitive timeframe, they hoped to finalize their policy in 2014.

Agencies cited several underlying reasons that contributed to these weaknesses: (1) they were not always aware of OMB guidance, (2) they did not believe that the guidance was realistic, and (3) they said the guidance was not always clear. Regarding agency awareness of the guidance, since the 2010 IT Reform Plan, OMB has communicated changes to incremental development requirements, such as the change from 12 to 6 months, through budget guidance. However, selected agency officials said they were not always aware of this guidance. For example, DHS Office of the CIO officials told us that did not know about OMB’s requirement to deliver functionality every 6 months. Additionally, Transportation officials representing the Office of the CIO were not aware that in 2012 OMB had changed its guidance from recommending to requiring that projects deliver in 6-month cycles.

With respect to whether OMB’s guidance is realistic, officials from Defense, HHS, DHS, and Transportation explained that they do not want to require all of their projects to deliver functionality every 6 months because it may not be reasonable for all investments to do so. Defense, DHS, and Transportation officials said that delivering every 12 months, as advocated in OMB’s IT Reform Plan, is more reasonable.Defense officials from the Office of the CIO, 12-18 month incremental development is better aligned with the acquisition framework that many of its IT investments have used. DHS and Transportation Office of the CIO officials stated that, depending on program size, scope, complexity, budget, schedule, and expertise, it may be more reasonable to deliver functionality every 12 months. As discussed later in this report, we agree that OMB’s requirement to deliver functionality every 6 months is unrealistic. OMB staff members from the Office of E-Government and Information Technology noted that it will take time for agencies to embrace delivering functionality every 6 months because of the perceived risk of adopting such approaches. Those staff members explained that agencies, such as Defense, have been using the waterfall development method for many years, and they perceive less risk in continuing with that method than with changing to a method that produces functionality more rapidly.

Lastly, two key components of OMB’s guidance are not clear. First, in revising Circular A-130 in 2000, OMB did not identify the minimum requirements of what the agencies’ policies are to include and did not specify when the policies are to be completed. Although OMB issued later guidance on incremental development, it has not yet specified what agencies’ incremental development policies are to include. Second, OMB’s guidance did not provide a complete definition of the functionality it expects to be delivered every 6 months. According to staff from the Office of E-Government, OMB intends for agencies to deliver functionality that can be tested by business users; nevertheless, they noted that they left this definition out of their guidance so that agencies could develop a definition that would be flexible enough to meet their needs. However, in the absence of further guidance from OMB, agencies may continue to not define this term or may create definitions that are inconsistent with the intent of OMB’s policy. For example, HHS officials from the Office of the CIO told us that the completion of requirements documents could meet OMB’s definition of delivering functionality. Additionally, an FAA Office of the CIO official explained that some investments have classified the delivery of requirements documentation as functionality. These two examples are not consistent with OMB’s intent since they do not deliver functionality that can be tested, but instead only plan to deliver project documentation.

Until OMB explicitly issues realistic and clear guidance and Defense, HHS, DHS, and Transportation address the identified weaknesses in their incremental development policies, it will be difficult to deliver project functionality more rapidly, measure how often projects are delivering functionality, and enforce compliance with the delivery time frames called for in their policies.

Most Selected Investments Do Not Plan to Deliver Functionality Every 6 or 12 Months

In its 2010 IT Reform Plan, OMB called for IT programs to deliver functionality at least every 12 months. Subsequently, OMB has made this requirement more stringent in that it now requires projects associated with major IT investments to deliver functionality every 6 months.

The majority of the selected investments we reviewed did not plan to deliver functionality every 6 months. Specifically, only 23 of the selected 89 investments had one or more projects that, when taken collectively, planned to deliver functionality every 6 months. To VA’s credit, all six of the department’s selected investments planned to deliver functionality every 6 months. The other agencies varied in the extent to which they met the standards established by OMB’s guidance. Table 2 shows how many of the selected investments at each agency planned on delivering functionality every 6 months during fiscal years 2013 and 2014.

The variety of life-cycle cost estimates for these investments shows that incremental development can be applied to a wide variety of investment scopes. Specifically, of the 23 investments that planned to deliver functionality in 6-month cycles, 9 had cost estimates that were less than $250 million, 4 had estimates between $250 and $575 million, 5 had estimates between $575 million and $2 billion, and 5 had estimates greater than $2 billion.

Twenty-seven of the 89 investments in our review (30 percent) reported using an Agile development methodology for one or more of their projects. Of those 27 investments, 14 (52 percent) planned to deliver functionality every 6 months. The other 13 investments using Agile did not plan on delivering functionality as frequently as OMB guidance requires. We have previously found that Agile projects typically produce working functionality every 1 to 8 weeks; as such, it appears that these investments may not be properly implementing an Agile development methodology.

Agency officials cited three types of investments for which it may not always be practical or necessary to expect functionality to be delivered in 6-month cycles: (1) investments in life-cycle phases other than acquisition (i.e., planning and budgeting, management in-use, and disposition); (2) investments intended to develop IT infrastructure; and (3) research and development investments.

Life-cycle phases other than acquisition. Officials from Defense, HHS, DHS, and Transportation stated that it is not reasonable to expect investments to deliver functionality every 6 months when their investments’ projects are not in the acquisition phase. Specifically, 24 investments did not have projects in the acquisition stage. Of those 24 investments, 22 did not plan to deliver functionality in 6-month cycles (10 from Defense, 3 from HHS, 1 from DHS, and 8 from Transportation), and 2 investments did plan to do so (2 from HHS). For the 2 investments that planned to deliver functionality every 6 months, both had at least one project in the management in-use phase, meaning that at least one of the investments’ projects were beyond the planning and development stages and were being used to support agency operations.

Infrastructure investments. Officials from Defense, DHS, and Transportation explained that not all infrastructure investments can be expected to deliver functionality every 6 months. Specifically, 21 investments provide infrastructure, such as IT security, office automation, and telecommunications. For example, officials representing two of the DHS investments explained that, prior to deploying functionality, they need to acquire real estate, conduct environmental assessments, and perform construction work, such as digging trenches, burying cables, and building facilities. Of the 21 investments, 20 did not plan to deliver functionality every 6 months (17 from Defense, 1 from HHS, and 2 from DHS); however, 1 investment did plan to do so (1 from Defense).

Research and development investments. Officials from FAA’s Office of the CIO explained that FAA’s research and development investments are not intended to deliver functionality. Those officials stated that, before the agency approves a technology for further development, it performs research and development to ensure that the technology meets safety standards. If those standards are met, FAA creates a new investment aimed at deploying that technology. Consistent with this, none of FAA’s six research and development investments planned to deliver any functionality during fiscal years 2013 and 2014.

These concerns have merit. For example, with respect to investments in life-cycle phases other than acquisition, at the outset of a new investment, an agency may need more than 6 months to, among other things, define high-level requirements and find a contractor to help develop the system. In addition, it may not be necessary for an investment to continue delivering new functionality when all planned functionality has been fully deployed.

Regarding infrastructure investments, it may not be practical or cost- effective for an investment to refresh fully functioning hardware every 6 months. Additionally, it may not be feasible for an agency to build a physical facility (e.g., data center) within 6 months. Further, for research and development investments, industry practices and our work on best practices support FAA’s efforts to thoroughly validate the feasibility and cost-effectiveness of new technology prior to making significant investments.

Although OMB requires all investments to deliver functionality every 6 months, an OMB staff member from the Office of E-Government explained that not all investments will be able to meet this goal. Instead, that staff member said that about half of the federal government’s major IT investments will deliver functionality in 6 months or less, and the other half will have longer development cycles. However, OMB’s guidance does not make this distinction. As a result, agencies may be confused about whether OMB’s incremental development guidance applies to all investments.

If these three types of investments, which account for 40 of the selected 29 of the remaining 49 investments 89 investments, are not considered,did not plan to deliver functionality in 6-month cycles. Table 3 shows, after removing the three types of investments discussed above, how many of the selected investments at each agency planned to deliver functionality every 6 months during fiscal years 2013 and 2014.

Considering agencies’ concerns about delivering functionality every 6 months for the three types of investments discussed above and OMB’s own expectations that many investments will not meet this goal, it is unclear whether this is the most appropriate governmentwide goal, and it raises the question of whether OMB should consider a longer time frame, such as 12 months, as called for in OMB’s IT Reform Plan. However, even using the time frame of 12 months as the target, less than half of the selected investments planned to deliver functionality in 12-month cycles. Specifically, 41 of the 89 selected investments planned to deliver functionality every 12 months during fiscal years 2013 and 2014 and 48 did not. Most notably, the preponderance of Defense and Transportation investments (70 percent for Defense and 65 percent for Transportation) did not plan to deliver functionality every 12 months. Table 4 shows how many of the selected investments at each agency planned on delivering functionality every 12 months during fiscal years 2013 and 2014.

The previously discussed weaknesses in agency policies have permitted the inconsistent implementation of incremental development approaches. OMB staff members from the Office of E-Government acknowledged that inconsistent implementation of OMB’s guidance can be at least partially attributed to challenges in ensuring that agencies develop consistent definitions of and approaches to incremental development.

Although OMB has led the government’s recent effort to improve incremental development, it has not completed a key commitment aimed at improving incremental development—namely, OMB has rarely used the TechStat process to turn around or cancel investments that are not using incremental development. As previously mentioned, TechStat sessions are intended to terminate or turn around IT investments that are failing or are not producing results. Additionally, many failed IT investments have not used incremental development approaches. Therefore, OMB could use TechStat sessions as a powerful tool to turn around investments that are not being acquired incrementally. However, OMB staff members from the Office of E-Government said that OMB has only held one such TechStat.

OMB staff from the Office of E-Government explained that, in order to select investments in need of a TechStat session, agency exhibit 300 data are reviewed for evidence of poor performance. However, the usefulness of the exhibit 300 data for the purpose of identifying whether investments are using incremental approaches is limited. For example, OMB does not require agencies to explicitly identify whether their investments are using incremental approaches. Additionally, of the 89 selected investments, 34 had activities in their exhibit 300 submissions that were inaccurately classified as delivering functionality (9 from Defense, 8 from HHS, 5 from DHS, 9 from Transportation, and 3 from VA). For example, one Defense investment indicated that awarding a contract constituted a delivery of functionality. OMB staff from the Office of E-Government acknowledged that the exhibit 300 data are not as helpful as possible in addressing incremental development. Consequently, for its TechStat reviews, OMB’s insight into investments’ use of incremental development approaches is limited. Officials from Defense, HHS, DHS, and Transportation attributed the problem to a lack of guidance from OMB on what is to be delivered every 6 months. Nevertheless, officials from Defense, DHS, and VA stated that they would properly classify activities in future exhibit 300 submissions.without additional guidance from OMB, agencies may continue to improperly classify activities. Additionally, as previously mentioned, four of the five selected agencies—Defense, HHS, DHS, and Transportation— have not updated their TechStat policies to include identifying investments that are not being acquired incrementally.

Without better implementation of incremental development approaches and identification of investments using incremental development, IT expenditures will likely continue to produce disappointing results— including large cost overruns, long schedule delays, and questionable mission-related achievements. Further, without useful information, including whether investments are following an incremental approach, on projects associated with major IT investments, OMB does not have the necessary information to oversee the extent to which projects and investments are implementing its guidance.

Multiple Factors Were Commonly Identified as Enabling and Inhibiting Incremental Development

Multiple factors were identified by the five agencies as enabling and inhibiting incremental development during a 6-month period. Specifically, eight factors were identified by three or more of the five agencies in our review as enabling incremental development of IT systems, and seven factors were identified by three or more agencies as inhibiting incremental development. The enabling factor identified by all of five of the agencies was active engagement of program officials with stakeholders. The inhibiting factors identified by all five agencies were (1) the lack of sufficient, timely funding; (2) program characteristics that made rapid incremental development infeasible; and (3) the lack of stable, prioritized requirements.

Eight Factors Were Commonly Identified as Enabling Incremental Development during a 6- month Period

Eight factors were identified by three or more of the five agencies in our review as contributing to the successful development of functionality in 6- month cycles. All five of the agencies in our review cited active program engagement with stakeholders. Table 5 shows the distribution of the eight factors among the agencies, and examples of how the agencies implemented them are discussed following the table.

Program officials actively engaged with stakeholders. Officials from all five agencies explained that active engagement with program stakeholders—individuals or groups with an interest in the success of the investment—was a factor that enabled the development of functionality in 6-month cycles. For example, officials from one of the HHS investments that we reviewed stated that having strong communication between the business program, the IT program, and contractors enabled projects to move forward on schedule in a cohesive manner with clear goals and objectives.

Programs used an Agile development methodology. Officials from four of the five agencies indicated that the use of an Agile development methodology helped them to deliver functionality every 6 months. For example, Defense officials explained that the use of Agile development processes allowed software to be broken into smaller releases that were easily achieved, tested, and fielded. Additionally, VA officials stated that the agency has embraced Agile development, which has helped the department deliver functionality more quickly. Those officials also noted that merely forcing investments to use 6- month waterfall iterations would not have resulted in the same success.

Programs successfully prioritized, managed, and tested requirements.

Officials from four of the five agencies identified implementation of requirements management practices—including requirements prioritization, management, and testing—as a factor that enabled the agencies to deliver functionality every 6 months. For example, Transportation officials explained that the primary factor that enabled incremental development has been obtaining clear requirements. Further, HHS officials stated that the use of a prioritized product backlogrequirements should be allocated to future releases. has helped teams make decisions about which

Staff had the necessary skills and experience. Officials from four of the five agencies stated that consistent and stable program staff with experience and expertise allowed them to deliver functionality frequently. For example, DHS officials representing one of the selected investments indicated that having skilled program managers has a large impact on the success of a program. In addition, VA officials reported that their ability to retain key personnel with both functional and technical expertise has enabled them to deliver high- quality functionality rapidly.

Programs successfully implemented cost and schedule estimating best practices. Officials from four of the five agencies cited the implementation of cost and schedule estimating practices as enabling the frequent delivery of functionality. For example, Defense officials told us that development of realistic cost estimates and comprehensive program schedules helped programs to deliver functionality while meeting established performance goals. Further, Transportation officials stated that improved guidance and training on cost estimating and scheduling helped them to deliver functionality in 6-month cycles.

Officials used various contracting strategies to increase flexibility and improve contractor oversight. Officials from four of the five agencies cited the use of specific contracting strategies. For example, DHS officials indicated that they worked with contractors to modify their contracts from the traditional waterfall approach to a structure that allows for Agile development. In addition, officials from VA stated that the use of performance-based acquisitions assisted them in monitoring contractor progress towards achieving actual results against planned objectives.

Staff used key technologies to accelerate development work. Officials from three of the five agencies indicated that the use of key technologies enabled them to deliver functionality more quickly. For example, HHS officials explained that having an established cloud environment has enabled them to reduce deployment time for releases, while also providing the needed flexibility to meet their customers’ changing needs. Additionally, DHS officials explained that projects, especially those using an Agile development methodology, have saved time using automated testing tools.

Programs successfully implemented risk management best practices.

Officials from three agencies explained that the successful implementation of risk management practices helped them to deliver functionality more rapidly. For example, officials from one of the selected VA investments told us that the organization’s risk management process allowed teams to quickly escalate risks to senior leadership so that they could be managed before their impacts were fully realized.

Many of the factors that the five agencies cited are consistent with our 2011 work on factors critical to successful IT acquisitions and our 2012 work on effective practices in implementing Agile software development.In particular, our work in both areas discussed the importance of active engagement with program stakeholders, such as actively engaging with stakeholders to obtain customer feedback, and effectively managing requirements.

Additionally, the eight commonly identified factors that enable the delivery of functionality in 6 months are consistent with OMB’s IT Reform Plan. In particular, as previously mentioned, one high-level objective of the plan—effectively managing large-scale IT programs—aims to improve areas that impact the success rates of large IT programs by, among other things, enabling programs to use incremental development approaches. As part of this high-level objective, the plan addresses the importance of actively engaging with stakeholders, ensuring that program management professionals have proper skills and experience, and developing new contracting strategies that more effectively support incremental development.

Seven Factors Were Commonly Identified as Inhibiting Incremental Development during a 6- month Period

Seven factors were identified by three or more of the five agencies as inhibiting the development of IT functionality every 6 months. The factors most commonly cited include (1) programs did not receive sufficient funding or receiving funding later than needed, (2) program characteristics made rapid delivery of functionality infeasible or impractical, (3) programs did not have stable and prioritized requirements. Table 6 shows the distribution of the seven factors among the agencies, and examples of how the factors impacted the agencies are discussed following the table.

Programs did not receive sufficient funding or received funding later than needed. Officials from all five departments cited insufficient funding, such as reductions caused by the fiscal year 2013 sequester, or receiving funding later than needed because of continuing resolutions. For example, Defense officials representing one of the investments we reviewed stated that furloughs brought on by the 2013 sequester significantly impacted program schedules, both in terms of lost work days and the inability to coordinate integration between software providers, resulting in overall inefficiencies. In addition, several FAA officials explained that the delivery of planned functionality was adversely affected by the uncertainty brought about by the 2013 sequester, and that future funding instability has impacted the agency’s ability to plan when functionality is to be delivered.

Program characteristics made rapid delivery of functionality infeasible or impractical. Officials from all five agencies indicated that some of their programs have certain characteristics—such as the deployment of new physical infrastructure, human health and safety concerns, or external dependencies—that make the delivery of functionality in 6- month time frames infeasible or impractical. For example, DHS officials explained that their infrastructure projects cannot deliver functionality until all key activities (e.g., land acquisition, environmental assessments, site preparation, and construction) have been completed and the new infrastructure has been fully deployed, tested, and accepted. Additionally, Transportation officials reported that air traffic control systems require years of development and testing in order to ensure that the systems do not compromise airspace security.

Programs did not have stable, prioritized requirements. Officials from all five agencies stated that not having complete requirements at the beginning of their investments’ development or having changes made to requirements and their relative priorities during development was a factor that negatively affected their programs’ delivery of incremental functionality in 6-month periods. For example, HHS officials representing one of the selected investments explained that that final rules detailing eligibility for a medical program were not completed until late in the development cycle; consequently, the program could not define the corresponding business requirements in time to meet the development schedule. Further, VA officials stated that schedules are disrupted when stakeholders identify new program requirements that must be delivered before others.

Development work was slowed by inefficient governance and oversight processes. Officials from three of the five agencies cited inefficient governance and oversight processes. For example, DHS officials representing the Office of the CIO stated that the current DHS governance model was not conducive to frequent delivery of functionality. To illustrate, those officials noted that it can take up to 2 months to schedule a meeting with DHS review boards prior to releasing functionality. However, a DHS official from Program Accountability and Risk Management disagreed with this statement, explaining that DHS’s acquisition review boards perform reviews very quickly, and that any delays in completing these reviews are attributable to investments being unprepared. Further, DHS Office of the CIO officials suggested that governance over programs using an Agile development methodology should be performed at the lowest practicable level of the organization. In addition, VA officials explained that, although the periodic approvals required of various department groups are useful, these approvals can slow the testing and release processes and could use further streamlining.

Development schedules were impeded by procurement delays.

Officials from three of the five agencies explained that procurement delays—such as delays in getting contracts awarded or approving contract modifications—contributed to difficulties in delivering incremental functionality. For example, DHS officials explained the process of planning for an acquisition, developing solicitations, selecting contractors, and, in some cases, addressing protests, are not conducive to delivering functionality in 6-month cycles.

Program staff were overutilized or lacked the necessary skills and experience. Officials from three of the five agencies indicated that they did not have enough program staff with the expertise and experience necessary to deliver functionality every 6 months. For example, officials representing one of the HHS investments stated that the loss of key personnel and the lack of staff with knowledge and skill in key disciplines, such as incremental development, have negatively impacted the agency’s ability to deliver functionality.

Incremental development was impeded by select technologies. Officials from three of the five agencies in our review explained that the software and tools they selected for their programs either introduced delays in development or were ultimately not usable by the program. Transportation officials representing one of the selected investments reported that their chosen technology added a new level of complexity to storage and data processing that required all development and testing to be completed before the implementation could occur. As a result, those officials told us that all functionality had to be deployed in one release, instead of being spread out in stages.

Many of the factors that the five agencies identified as inhibiting delivery of functionality in 6-month increments were consistent with our work on the challenges we identified in the application of Agile software development methods.deploying software developed using an Agile development method is that traditional governance and oversight activities were difficult to execute For example, we noted that one challenge in within an iterative time frame. In particular, one agency official stated that completing the necessary reviews within the short, fixed time frame of an Agile iteration was difficult because reviewers followed a slower waterfall schedule with reviews that could take months to perform after the completion of an iteration. This caused delays for iterations that needed such reviews within the few weeks of the iteration. We also noted that procurement practices sometimes delay Agile development.

Further, OMB’s IT Reform Plan also identified some of the same problems that inhibit incremental development and described solutions to In particular, as previously mentioned, one item of the address them.plan was to improve funding of programs by working with Congress to create IT budget models that promote IT budget flexibility (such as multiyear budgets or revolving funds). In addition, the plan addresses the importance of streamlining IT governance by strengthening the quality and timing of oversight activities.

The factors identified in this report as enabling and inhibiting incremental development could help agencies address the challenges they face in incrementally acquiring IT.

Conclusions

Given the enormous size of the federal government’s investment in IT and the often disappointing results from IT development efforts—many of which are attributable to the use of a “big bang” approach—finding ways to improve the quality and timeliness of agencies’ investments is important. Congress, OMB, and our work support the use of incremental development practices. However, although the selected agencies have developed policies that address incremental development, most of the agencies’ policies have significant weaknesses. With the exception of VA, these policies do not fully define functionality, and do not have a complete process for ensuring that the agencies’ investments are developed incrementally, including the use of TechStat sessions to enforce compliance with incremental development policies. In the absence of such policies, agencies continue to run the risk of failing to deliver major investments in a cost-effective and efficient manner.

Regarding implementation of incremental development, slightly more than one-fourth of selected investments planned to deliver functionality every 6 months—and less than one-half planned to do so every 12 months. Thus, delivering functionality every 6 months is not an appropriate requirement for all agencies given current performance. Requiring the delivery of functionality every 12 months, consistent with OMB’s IT Reform Plan, would be an appropriate starting point and be a substantial improvement. Further, since there are three types of investments for which it may not always be practical or necessary to expect functionality to be delivered in 6-month cycles, this raises questions about whether OMB’s requirement for shorter delivery cycles should be applied to all investments or whether investments should be allowed the latitude to determine a more appropriate time frame.

The lack of progress in updating policies and implementing OMB’s guidance was enabled by weaknesses in OMB’s guidance. In the absence of agency and OMB use of TechStat sessions to ensure compliance with incremental development policy, investments will continue to be at risk of not delivering promised capabilities on time and within budget. Until OMB clearly and explicitly disseminates guidance with realistic goals and clear expectations, and agencies update their policies to reflect this guidance, agencies may not consistently adopt incremental development approaches, and IT expenditures will continue to produce disappointing results—including sizable cost overruns and schedule slippages, and questionable progress in meeting mission goals and outcomes. Additionally, without useful information on how often projects are delivering functionality, it will be difficult for OMB to identify investments that are not implementing its guidance. Further, dissemination of the factors identified in this report as enabling and inhibiting incremental development may help federal agencies address the challenges they face in acquiring IT investments faster and more efficiently.

Recommendations for Executive Action

We recommend that the Director of the Office of Management and Budget direct the Federal Chief Information Officer to take the following two actions.

Update, and clearly and explicitly issue incremental development guidance that addresses the following three components: requires projects associated with major IT investments to deliver incremental functionality at least every 12 months, with the exception of the three types of investments identified in this report; specifies how agencies are to define the project functionality that is to be delivered; and requires agencies to define a process for enforcing compliance with incremental functionality delivery, such as the use of TechStat sessions.

Require agencies to clearly identify on exhibit 300 submissions whether, for each project, functionality will be delivered within the time frames called for by this incremental development guidance, and to provide justification for projects that do not plan to do so.

We further recommend that the Secretaries of Defense, Health and Human Services, Homeland Security, and Transportation take the following two actions: modify, finalize, and implement their agencies’ policies governing incremental development to ensure that those policies comply with OMB’s guidance, once that guidance is made available; and when updating their policies, consider the factors identified in this report as enabling and inhibiting incremental development.

We also recommend that the Secretary of Veterans Affairs consider incorporating the factors identified in this report as enabling and inhibiting incremental development in the department’s related policy.

Agency Comments and Our Evaluation

We received comments on a draft of this report from OMB and the five agencies in our review. OMB agreed with one recommendation and partially disagreed with the other; Defense generally concurred with the report; HHS neither agreed nor disagreed with the report’s recommendations; DHS agreed with our recommendations; Transportation did not agree with the recommendations in that it did not believe the department should be dependent on OMB first taking action; and VA generally agreed with the report’s conclusions and concurred with our recommendation. Each agency’s comments that we received are discussed in more detail below.

In comments provided via e-mail on April 15, 2014, staff in OMB’s Office of General Counsel, on behalf of OMB, stated that the agency agreed with one of our recommendations and partially disagreed with the other. Specifically, OMB agreed with our recommendation to require agencies to clearly identify on exhibit 300 submissions whether functionality will be delivered within the time frames called for by this incremental development guidance.

OMB stated that it agreed with our recommendation to update and issue incremental development guidance, but did not agree that this guidance should require major IT investments to deliver incremental functionality at least every 12 months. OMB explained that changing the requirement from 6 to 12 months would reduce the emphasis on incremental development that it has been advocating. OMB also noted that it believes requiring investments to deliver functionality every 6 months is an appropriate governmentwide goal and said that updating and clarifying its guidance on incremental development will make it easier for agencies to meet this target. However, as we state in this report, slightly more than one-fourth of selected investments planned to deliver functionality every 6 months—and less than one- half planned to do so every 12 months. Additionally, there are three types of investments for which it may not always be practical or necessary to expect functionality to be delivered in 6-month cycles. Thus, we continue to believe that delivering functionality every 6 months is not an appropriate requirement for all agencies and that requiring the delivery of functionality every 12 months, consistent with OMB’s IT Reform Plan, is a more appropriate starting point. We therefore maintain that OMB should require projects associated with major IT investments to deliver functionality at least every 12 months.

In written comments, Defense stated that it generally concurred with the report and outlined planned actions to address the recommendations. However, Defense explained that many of its IT investments are consistent with the three types of investments for which it may not always be practical or necessary to expect functionality to be delivered in 6-month cycles or have other exceptional circumstances. Given these issues, Defense stated that requiring the delivery of functionality every 12 months is not a useful management constraint. However, as we state in our report, many failed projects have been broadly scoped in that they aim to deliver their capabilities several years after initiation. Additionally, the Defense Science Board reported that Defense’s acquisition process for IT systems was too long, was ineffective, and did not accommodate the rapid evolution of IT. In order to resolve these issues, we continue to believe that investments and their projects should deliver functionality more frequently. In addition, the existence of many investments that are consistent with the three types of investments identified in this report for which it may not be practical to deliver functionality quickly does not excuse Defense from the requirement of delivering functionality every 12 months. Consistent with our recommendation to OMB, investments and projects that meet these three exceptions should be exempt from the requirement, but all other investments and projects should aim to meet this requirement. Further, we expect that Defense’s actions to implement our recommendation to consider the factors identified in this report that enable and inhibit incremental development will help its investments and projects deliver functionality more rapidly. As such, we continue to believe that requiring the delivery of functionality every 12 months is an appropriate goal for all federal agencies, including Defense. Defense’s comments are reprinted in appendix III. The department also provided technical comments, which we have incorporated in the report as appropriate.

In comments provided via e-mail on April 15, 2014, an official from HHS’s Office of the Assistant Secretary for Legislation, on behalf of HHS, stated that the department had no comments.

In written comments, DHS stated that it concurred with our recommendations and outlined planned actions to address the recommendations. DHS’s comments are reprinted in appendix IV. The department also provided technical comments, which we have incorporated in the report as appropriate.

In comments provided via e-mail on April 15, 2014, Transportation’s Deputy Director of Audit Relations, on behalf of Transportation, stated that the department would prefer to have specific recommendations and deliverables so that it can achieve success in closing them. Specifically, the department explained that relying on another agency to concur with one of our recommendations before Transportation can take action leaves the department with the potential challenge of a recommendation that cannot be implemented. However, as previously stated, OMB agrees with our recommendation to update and issue incremental guidance, meaning that OMB has committed to taking the actions necessary to enable Transportation to begin addressing our recommendation. Additionally, our recommendation to consider the factors identified in this report as enabling and inhibiting incremental development when updating incremental development policies does not require another agency to take action before Transportation can implement it. Accordingly, we continue to believe that our recommendations are warranted and can be implemented.

In written comments, VA stated that it generally agreed with the report’s conclusions, that it concurred with our recommendation to the department, and that it will review its existing policy for incremental development and consider incorporating the factors identified as enabling and inhibiting incremental development. VA’s comments are reprinted in appendix V. The department also provided technical comments, which we have incorporated in the report as appropriate.

As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies of this report to interested congressional committees; the Director of the Office of Management and Budget; and the Secretaries of the Departments of Defense, Health and Human Services, Homeland Security, Transportation, and Veterans Affairs. In addition, the report will also be available at no charge on our website at http://www.gao.gov/.

If you or your staffs have any questions on matters discussed in this report, please contact me at (202) 512-9286 or pownerd@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix VI.

Appendix I: Objectives, Scope, and Methodology

Our objectives for this review were to (1) assess whether selected agencies have established policies for incremental information technology (IT) development, (2) determine whether selected agencies are using incremental development approaches to manage their IT investments, and (3) identify the key factors that enabled and inhibited the selected agencies’ abilities to effectively use incremental development approaches to manage their IT investments.

In conducting our review, we selected five agencies and a total of 89 investments from those agencies. To choose the agencies, we identified the five agencies with the largest IT budgets for development, modernization, and enhancement on major IT investments in fiscal years 2013 and 2014 as reported in the Office of Management and Budget’s (OMB) fiscal year 2014 exhibit 53. Those agencies are the Departments of Defense (Defense), Health and Human Services (HHS), Homeland Security (DHS), Transportation (Transportation), and Veterans Affairs (VA). To choose the agencies’ investments, we identified the 98 major IT investments for which the selected agencies planned to spend more than 50 percent of the investments’ fiscal year 2013 and 2014 budgets on development, modernization, and enhancement as reported in OMB’s fiscal year 2014 exhibit 53. We removed 9 investments because, after reporting their planned budgets for OMB’s fiscal year 2014 exhibit 53, these investments changed their budgets so that they no longer planned to spend more than 50 percent of the investments’ fiscal year 2013 and The 2014 budgets on development, modernization, and enhancement.final 89 investments are identified in appendix II. The investments selected for the review account for about 58 percent of the development, modernization, and enhancement spending on all federal agencies’ major IT investments for fiscal years 2013 and 2014 reported in OMB’s exhibit 53 for fiscal year 2014.

To address our first objective, we reviewed OMB guidance related to the use of incremental development, as well as industry guidance, and identified three key components of incremental development that agencies should include in their policies. In order to identify these components, we reviewed OMB’s guidance and leading industry guidance on institutionalizing processes throughout an organization.analysis identified three key policy components that will help agencies effectively implement OMB’s requirement for incremental development.

Require delivery of functionality every 6 months. According to the Software Engineering Institute’s (SEI) Capability Maturity Model® Integration for Acquisition (CMMI-ACQ), as part of institutionalizing a process, organizations should document their processes, to include defining standards and requirements. According to OMB budget guidance, projects associated with major IT investments must deliver functionality every 6 months.

Define functionality. As previously stated, according to SEI’s CMMI- ACQ, as part of institutionalizing a process, organizations should document that process, to include defining standards and requirements.

Define a process for enforcing compliance. According to SEI’s CMMI- ACQ, as part of institutionalizing a process, organizations should document that process, including management review activities (e.g., taking corrective action when requirements and objectives are not being satisfied). Additionally, OMB Circular A-130 requires agencies to ensure that investments comply with their policies. Further, according to the President’s Fiscal Year 2014 Budget, agencies are to use their TechStat processes to identify investments that are not being acquired incrementally and undertake corrective actions.

At each selected agency, we then analyzed agency policies for incremental development and compared these policies to the three components identified above.

For each agency, each policy component was assessed as either being not met—the agency did not provide evidence that addressed the component or provided evidence that minimally addressed the component; partially met—the agency provided evidence that addressed about half or a large portion of the component; or fully met—the agency provided evidence that addressed the component. We also interviewed officials from OMB and the five selected agencies to obtain information about their current and future incremental development policies.

To address our second objective, we administered a data collection instrument to each of the selected investments about how often each investment planned to deliver functionality during fiscal years 2013 and 2014. We then analyzed information obtained from data collection instruments describing how often the selected investments planned to deliver functionality. We prepopulated these instruments with data obtained from OMB’s fiscal year 2014 exhibit 53, as well as each investment’s exhibit 300 data, which describe the investments’ projects and how often each project plans to deliver functionality. We asked officials from each investment to verify the accuracy and completeness of the data and to make corrections where needed. Because the exhibit 300 data did not always describe the investments’ plans for delivering functionality in fiscal year 2014, we asked the officials to indicate for each project whether they planned to deliver functionality in the first half of fiscal year 2014 (i.e., from October 1, 2013, to March 31, 2014) and in the second half of fiscal year 2014 (i.e., from April 1, 2014, to September 30, 2014). We also asked the investments to provide their life-cycle cost estimates and, for each project, the development methodology used and phase of the acquisition life cycle.

Using the information obtained through the data collection instruments, we determined the extent to which the selected investments and their projects planned to meet OMB’s guidance on incremental development. To assess whether investments had planned to deliver functionality every 6 months, we determined whether the selected investments planned to deliver functionality in each of the following four time frames: (1) the first half of fiscal year 2013 (i.e., from October 1, 2012, to March 31, 2013), (2) the second half of fiscal year 2013 (i.e., from April 1, 2013, to September 30, 2013), (3) the first half of fiscal year 2014, and (4) the second half of fiscal year 2014. To determine whether investments had planned to deliver functionality every 12 months, we analyzed whether the selected investments planned to deliver functionality in each of the following two time frames: (1) fiscal year 2013 and (2) fiscal year 2014. We presented our results to the five selected agencies and OMB and solicited their input and explanations for the results.

To determine the reliability of the exhibit 300 data, we performed three steps. First, as previously mentioned, we asked officials from each investment to verify the accuracy and completeness of these data and provide the correct information where needed. Second, we removed projects that were not intended to deliver functionality and activities that were inaccurately classified as resulting in the delivery of functionality. To do so, we compared the descriptions of the projects and activities with a definition of functionality that we developed. To develop this definition, we reviewed OMB and agency guidance, as well as leading practices.We defined functionality as follows: the implementation of IT requirements that is intended to either (1) ultimately be used by one or more customers in production (actual deployment may occur at a later date than when the functionality is reported as being delivered) or (2) be a delivery of a prototype or pilot. We then assessed the descriptions of the activities and projects agencies reported in their exhibit 300 submissions against our definition. We presented the activities and projects that did not meet our definition to the selected agencies and solicited their input and explanations. Third, where there was a conflict between the exhibit 300 data and agencies’ answers to our questions regarding plans for delivering functionality in fiscal year 2014, we presented the conflict to the agencies and obtained clarification. We determined that the data were sufficiently reliable for the purpose of this report, which is to determine the extent to which the selected investments planned to deliver functionality every 6 and 12 months, during fiscal years 2013 and 2014.

GAO, Information Technology: Critical Factors Underlying Successful Major Acquisitions, GAO-12-7 (Washington, D.C.: Oct. 21, 2011). methods. Additionally, we compared the factors to OMB’s 25 Point Implementation Plan to Reform Federal Information Technology Management.

Further, because Defense guidance encourages investments to deliver functionality every 12-18 months where possible, we asked the selected investments and officials from Defense’s Office of the CIO to identify the key factors that have both enabled and inhibited their efforts to deliver functionality for their major IT investments every 12-18 months. We compared the information we received to the eight factors the five selected agencies commonly identified as enabling incremental development during a 6-month period and the seven factors commonly identified by the five agencies as inhibiting incremental development during a 6-month period. We also performed a content analysis of the information we received in order to identify additional factors.

We conducted this performance audit from May 2013 to May 2014, in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Selected Investments’ Plans for Delivering Functionality

Below is the list of investments that are included in this review, as well as whether each planned to deliver functionality every 6 and 12 months for fiscal years 2013 and 2014.

Appendix III: Comments from the Department of Defense

Appendix IV: Comments from the Department of Homeland Security

Appendix V: Comments from the Department of Veterans Affairs

Appendix VI: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact name above, individuals making contributions to this report included Dave Hinchman (Assistant Director), Deborah A. Davis (Assistant Director), Rebecca Eyler, Kaelin Kuhn, Jamelyn Payan, Meredith Raymond, Kevin Smith, Andrew Stavisky, and Kevin Walsh.