Information Technology: Agencies Need to Establish and Implement Incremental Development Policies
Published: 2014-05-01T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Federal agencies plan to spend at least $82 billion on IT in fiscal year 2014. However, prior IT expenditures have often produced disappointing results. Thus, OMB has called for agencies to deliver investments in smaller parts or increments. In 2010, it called for IT investments to deliver capabilities every 12 months and now requires investments to deliver capabilities every 6 months. GAO was asked to review agencies' incremental development approaches. Among other things, this report (1) assesses whether selected agencies have established policies for incremental IT development; and (2) determines whether selected agencies are using incremental development approaches to manage their IT investments. To do so, GAO selected five agencies—Defense, HHS, DHS, Transportation, and VA—and 89 total investments at these agencies. GAO then reviewed the agencies' incremental development policies and plans.
section_title: What GAO Found
paragraphs: All five agencies in GAO's review—the Departments of Defense (Defense), Health and Human Services (HHS), Homeland Security (DHS), Transportation (Transportation), and Veterans Affairs (VA)—have established policies that address incremental development; however, the policies usually did not fully address three key components for implementing the Office of Management and Budget's (OMB) guidance (see table). Specifically, only VA fully addressed the three components. Among other things, agencies cited the following reasons that contributed to these weaknesses: (1) the guidance was not feasible because not all types of investments should deliver functionality in 6 months, and (2) the guidance did not identify what agencies' policies are to include or time frames for completion. GAO agrees these concerns have merit. Until OMB issues realistic and clear guidance and agencies address the weaknesses in their incremental development policies, it will be difficult to deliver project capability more rapidly.
Key =Fully met =Partially met =Not met
Source: GAO analysis of agency documentation.
The weaknesses in agency policies have enabled inconsistent implementation of incremental development approaches: almost three-quarters of the selected investments did not plan to deliver functionality every 6 months, and less than half planned to deliver functionality in 12-month cycles (see table). Without consistent use of incremental development approaches, information technology (IT) expenditures are more likely to continue producing disappointing results.
Source: GAO analysis of agency data.
section_title: What GAO Recommends
paragraphs: Among other things, GAO recommends that OMB develop and issue realistic and clear guidance on incremental development and that the selected agencies update and implement their incremental development policies to reflect OMB's guidance. OMB partially disagreed, believing its guidance is realistic. Four agencies generally agreed with the report or had no comments, and one agency did not agree that its recommendations should be dependent on OMB first taking action. GAO continues to believe that its recommendations are valid, as discussed in this report.

section_title: Letter
paragraphs: Information systems are critical to the health, economy, and security of  the nation. To support agency missions, the federal government plans to  spend at least $82 billion on information technology (IT) investments in  fiscal year 2014. However, as we have previously reported, prior IT  expenditures too often have produced failed projects—that is, projects  with multimillion dollar cost overruns and schedule delays measured in  years, with questionable mission-related achievements. Many of these  investments have been broadly scoped projects that aim to deliver their  capabilities several years after initiation.
To help resolve these issues, Congress and the Office of Management  and Budget (OMB) have called for agencies to deliver investments in  smaller parts, or increments, in order to reduce investment risk, deliver  capabilities more quickly, and facilitate the adoption of emerging   In 2010, OMB placed a renewed emphasis on incremental  technologies. IT development by calling for major IT investments to deliver functionality  at least every 12 months. Subsequently, OMB has made this guidance  more stringent, and annual budget guidance now states that each project  associated with major IT investments is to deliver functionality every 6  months.
This report responds to your request that we review agencies’  incremental IT development. Specifically, we (1) assessed whether  selected agencies have established policies for incremental IT  development, (2) determined whether selected agencies are using  incremental development approaches to manage their IT investments,  and (3) identified the key factors that enabled and inhibited the selected  agencies’ abilities to effectively use incremental development approaches  to manage their IT investments.
In conducting our review, we selected five agencies and 89 investments  to review. We selected the five agencies with the largest IT budgets for  development, modernization, and enhancement on major IT investmentsin fiscal years 2013 and 2014: the Departments of Defense (Defense),  Health and Human Services (HHS), Homeland Security (DHS),  Transportation (Transportation), and Veterans Affairs (VA). We then  selected the major IT investments at these agencies that planned to  spend more than 50 percent of the investments’ fiscal year 2013 and  2014 budgets on development, modernization, and enhancement. These  89 investments are identified in appendix II.
OMB, Management of Federal Information Resources, Circular No. A-130 Revised. processes throughout an organization.agencies’ incremental development policies to the components we  identified.
We then compared the selected  To address our second objective, we analyzed information obtained from  data collection instruments describing how often the selected investments  planned to deliver functionality. Specifically, we administered a data  collection instrument to officials responsible for each of the selected  investments that asked how often they planned to deliver functionality  during fiscal years 2013 and 2014. Using these data, we determined the  extent to which the selected investments and their projects are meeting  OMB’s guidance on incremental development.
To address our third objective, officials from the five selected agencies  and investments identified the key factors that have both enabled and  inhibited their efforts to deliver functionality for their major IT investments,  consistent with OMB guidance. We then determined the key factors that  were commonly identified by the selected agencies and investments.
We conducted this performance audit from May 2013 to May 2014, in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives. Details of our objectives,  scope, and methodology are contained in appendix I.
subsections: section_title: Background
paragraphs: If done correctly, investments in IT have the potential to make  organizations more efficient in fulfilling their missions. For example, we  recently reported that Defense officials stated that an IT system  supporting military logistics has improved the organization’s performance  by providing real-time information about road conditions, construction,  incidents, and weather to facilitate rapid deployment of military assets.
However, as we have previously reported, investments in federal IT too  frequently result in failed projects that incur cost overruns and schedule  slippages while contributing little to mission-related outcomes. For  example:  In January 2011, the Secretary of Homeland Security ended the  Secure Border Initiative Network program after obligating more than  $1 billion for the program because it did not meet cost-effectiveness  and viability standards. Since 2007, we have identified a range of  issues and made several recommendations to improve this program.  For example, in May 2010, we reported that the final acceptance of  the first two deployments had slipped from November 2009 and  March 2010 to September 2010 and November 2010, respectively,  and that the cost-effectiveness of the system had not been justified.  We concluded that DHS had not demonstrated that the considerable  time and money being invested to acquire and deploy the program  were a wise and prudent use of limited resources. As a result, we  recommended that the department (1) limit near-term investment in  the program, (2) economically justify any longer-term investment in it,  and (3) improve key program management disciplines. This work  contributed to the department’s decision to cancel the program.
In February 2011, the Office of Personnel Management canceled its  Retirement Systems Modernization program after several years of  trying to improve the implementation of this investment.the Office of Personnel Management, it spent approximately $231  million on this investment. We issued a series of reports on the   According to  agency’s efforts to modernize its retirement system and found that the  Office of Personnel Management was hindered by weaknesses in  several important management disciplines that are essential to  successful IT modernization efforts. Accordingly, we made  recommendations in areas such as project management,  organizational change management, testing, and cost estimating. In  May 2008, an Office of Personnel Management official cited the  issues that we identified as justification for issuing a stop work order  to the system contractor, and the agency subsequently terminated the  contract.
In December 2012, Defense canceled the Air Force’s Expeditionary  Combat Support System after having spent more than a billion dollars  and missing multiple milestones, including failure to achieve  deployment within 5 years of obligating funds. We issued several  reports on this system and found that, among other things, the  program was not fully following best practices for developing reliable  schedules and cost estimates.
Agencies have reported that poor-performing projects have often used a  “big bang” approach—that is, projects that are broadly scoped and aim to  deliver functionality several years after initiation. For example, in 2009 the  Defense Science Board reported that Defense’s acquisition process for IT  systems—which was rooted in the “waterfall” development model—was  too long, ineffective, and did not accommodate the rapid evolution of IT.The board reported that the average time to deliver an initial program  capability for a major IT system acquisition at Defense was over 7 years.  Also in 2009, VA’s former chief information officer (CIO) reported that  many of its projects exceeded cost estimates by more than 50 percent  and missed scheduled completion dates by more than a year. That official  concluded that VA needed to make substantial changes to its acquisition  process in order to eliminate project failures associated with the “big  bang” approach.
One approach to reducing the risks from broadly scoped, multiyear  projects is to divide investments into smaller parts—a technique long  advocated by Congress and OMB. can potentially   By following this approach, agencies deliver capabilities to their users more rapidly, giving them more  flexibility to respond to changing agency priorities; increase the likelihood that each project will achieve its cost,  schedule, and performance goals; obtain additional feedback from users, increasing the probability that  each successive increment and project will meet user needs; more easily incorporate emerging technologies; and  terminate poorly performing investments with fewer sunk costs.
See Clinger-Cohen Act of 1996, Pub. L. No. 104-106 § 5202, 110 Stat. 186, 690 (1996),  codified at 41 U.S.C. § 2308; see also 48 C.F.R. § 39.103 (Federal Acquisition  Regulation); OMB, Management of Federal Information Resources, Circular No. A-130  Revised. implement them. More recently, in its 2010 IT Reform Plan, OMB called  for IT programs to deliver functionality at least every 12 months and  complete initial deployment to end users no later than 18 months after the  start of the program. In 2011, as part of its budget guidance, OMB first  recommended that projects associated with major IT investments deliver  functionality every 6 months. OMB’s latest guidance now makes this  mandatory; specifically, in 2012, OMB began requiring that functionality  be delivered at least every 6 months.
subsections: section_title: Roles and Responsibilities for Overseeing Federal IT Investments
paragraphs: Over the last three decades, Congress has enacted several laws to assist  agencies and the federal government in managing IT investments. For  example, the Paperwork Reduction Act of 1995 requires that OMB  develop and oversee policies, principles, standards, and guidelines for  federal agency IT functions, including periodic evaluations of major  information systems. In addition, to assist agencies in managing their  investments, Congress enacted the Clinger-Cohen Act of 1996. Among  other things, the act requires agency heads to appoint CIOs and specifies  many of their responsibilities. With regard to IT management, CIOs are  responsible for implementing and enforcing applicable governmentwide  and agency IT management principles, standards, and guidelines;  assuming responsibility and accountability for IT investments; and  monitoring the performance of IT programs and advising the agency head  whether to continue, modify, or terminate such programs. Additionally,  with regard to incremental development, Clinger-Cohen calls for  provisions in the Federal Acquisition Regulation that encourage agencies  to structure their IT contracts such that the capabilities are delivered in  smaller increments. The Federal Acquisition Regulation provisions are  also to provide, to the maximum extent practicable, that the increment  should be delivered within 18 months of the contract solicitation.
As set out in these laws, OMB is to play a key role in helping federal  agencies manage their investments by working with them to better plan,  justify, and determine how much they need to spend on projects and how  to manage approved projects. Within OMB, the Office of E-Government  and Information Technology, headed by the Federal CIO, directs the policy and strategic planning of federal IT investments and is responsible  for oversight of federal technology spending.
In carrying out its responsibilities, OMB uses several data collection  mechanisms to oversee federal IT spending during the annual budget  formulation process. Specifically, OMB requires federal departments and  agencies to provide information related to their IT investments (called  exhibit 53s) and capital asset plans and business cases (called exhibit  300s).
Exhibit 53. The purpose of the exhibit 53 is to identify all IT  investments—both major and nonmajor—and their associated costs  within a federal organization. Information included in agency exhibit  53s is designed, in part, to help OMB better understand agencies’  spending on IT investments.
Exhibit 300. The purpose of the exhibit 300 is to provide a business  case for each major IT investment and to allow OMB to monitor IT  investments once they are funded. An IT investment may include one  or more projects that are to develop, modernize, enhance, or maintain  a single IT asset or group of IT assets with related functionality.  Agencies are required to provide information on each major  investment’s projects, including cost, schedule, and performance  information. For example, in order to measure compliance with its  requirement that projects deliver functionality in 6-month cycles, OMB  requires agencies to break their projects into activities, and describe  when the activities are to deliver functionality.
section_title: OMB’s Recent Major Initiatives for Overseeing IT Investments
paragraphs: OMB has implemented a series of initiatives to improve the oversight of  underperforming investments and more effectively manage IT. These  efforts include the following:  IT Dashboard. In June 2009, to further improve the transparency into  and oversight of agencies’ IT investments, OMB publicly deployed the  IT Dashboard. As part of this effort, OMB issued guidance directing  federal agencies to report, via the Dashboard, the performance of  their IT investments. Currently, the Dashboard publicly displays  information on the cost, schedule, and performance of over 700 major  federal IT investments at 26 federal agencies. Further, the public  display of these data is intended to allow OMB, other oversight  bodies, and the general public to hold the government agencies  accountable for results and progress.
TechStat reviews. In January 2010, the Federal CIO began leading  TechStats sessions—face-to-face meetings to terminate or turn  around IT investments that are failing or are not producing results.  These meetings involve OMB and agency leadership and are  intended to increase accountability and transparency and improve  performance. For example, the Federal CIO testified in June 2013 that  he holds TechStat meetings on large investments that are not being  acquired incrementally. More recently, OMB empowered agency  CIOs to hold their own TechStat sessions within their respective  agencies. In doing so, OMB has called for agencies to use their  TechStat processes to identify investments that are not being  acquired incrementally and undertake corrective actions.
IT Reform Plan. In December 2010, OMB released its 25-point plan to  reform federal IT. This document established an ambitious plan for  achieving operational efficiencies and effectively managing large- scale IT programs. In particular, as part of its effort to effectively  manage IT acquisitions, the plan calls for federal IT programs to  deploy functionality in release cycles no longer than 12 months, and  ideally, less than 6 months. The plan also identifies key actions that  can help agencies implement this incremental development guidance,  such as working with Congress to develop IT budget models that align  with incremental development, and issuing contracting guidance and  templates to support incremental development.
section_title: GAO Has Reported on Efforts to Improve IT Acquisitions
paragraphs: In April 2012, we reported on OMB’s efforts to implement the actions  called for in its IT Reform Plan and found that it had partially completed  work on two key action items relating to incremental development— issuing contracting guidance and templates to support incremental  development and working with Congress to create IT budget models that  align with incremental development. With respect to the contracting  guidance and templates, we found that, although OMB worked with the IT  and acquisition community to develop guidance, it had not yet issued this  guidance or the templates. Regarding the IT budget models, we found  that, although OMB worked to promote ideas for IT budget flexibility (such  as multiyear budgets or revolving funds) with congressional committees,  there has not yet been any new legislation to create budget models, and  OMB has not identified options to increase transparency for programs  that would fall under these budgetary flexibilities. We recommended that  the Director of OMB ensure that all action items called for in the IT  Reform Plan are completed. OMB agreed with this recommendation.  OMB has since issued contracting guidance for incremental  development,Office of E-Government and Information Technology stated that activities  to address the development of new IT budget models are still ongoing.   but, as of January 2014, a staff member from the OMB  Additionally, in 2011, we identified seven successful investment  acquisitions and nine common factors critical to their success.   Specifically, we reported that department officials identified seven  successful investment acquisitions, in that they best achieved their  respective cost, schedule, scope, and performance goals. Notably, all of  these were smaller increments, phases, or releases of larger projects. For  example, the Defense investment in our sample was the seventh  increment of an ongoing investment; the Department of Energy system  was the first of two phases; the DHS investment was rolled out to two  locations prior to deployment to 37 additional locations; and the  Transportation investment had been part of a prototype deployed to four  airports. In addition, common factors critical to the success of three or  more of the seven investments were: (1) program officials were actively  engaged with stakeholders, (2) program staff had the necessary  knowledge and skills, (3) senior department and agency executives  supported the programs, (4) end users and stakeholders were involved in  the development of requirements, (5) end users participated in testing of  system functionality prior to formal end user acceptance testing, (6)  government and contractor staff were stable and consistent, (7) program  staff prioritized requirements, (8) program officials maintained regular  communication with the prime contractor, and (9) programs received  sufficient funding. These critical factors support OMB’s objective of  improving the management of large-scale IT acquisitions across the  federal government, and wide dissemination of these factors could  complement OMB’s efforts.
Further, in 2012, we identified 32 practices and approaches as effective for applying Agile software development methods to IT projects.Officials from five agencies who had used Agile methods on federal  projects cited beneficial practices, such as obtaining stakeholder and  customer feedback frequently, managing requirements, and ensuring staff  had the proper knowledge and experience. We also identified 14  challenges with adapting and applying Agile in the federal environment,  including agencies having difficulty with committing staff to projects,  procurement practices that did not support Agile projects, and compliance  reviews that were difficult to execute within an iterative time frame. We  noted that the effective practices and approaches identified in the report,  as well as input from others with broad Agile experience, could help  agencies in the initial stages of adopting Agile.
section_title: Majority of Selected Agencies’ Incremental Development Policies Have Significant Weaknesses
paragraphs: Since 2000, OMB Circular A-130 has required agencies to (1) develop  policies that require their major investments to deliver functionality  incrementally and (2) ensure that investments comply with their policies.  In addition, as part of its recent budget guidance, OMB has defined how  often investments must deliver functionality. Specifically, each project  associated with major IT investments is to deliver functionality at least   Further, through the President’s Budget, OMB  once every 6 months.has provided additional guidance on how incremental development is to  be enforced by requiring agencies to use their TechStat processes to  identify investments that are not being acquired incrementally and  undertake corrective actions.
Although OMB’s guidance requires agencies to develop incremental  development policies, it does not specify what those policies are to  include. Absent this detail and in reviewing the previously mentioned  guidance and leading practices on institutionalizing processes throughout  an organization, include in their policies in order to effectively carry out OMB’s incremental  development guidance:   we identified three components that agencies should  require that all projects associated with major IT investments deliver  functionality in cycles that are not more than 6-months long; define functionality—that is, what the projects are to deliver at the end  of a 6-month cycle; and define a process for ensuring that major IT investments and their  projects deliver functionality every 6 months. This should include  identifying investments that are not being acquired incrementally  through agency TechStat processes and undertaking corrective  actions.
Although all five selected agencies developed policies that address  incremental development, the majority of the agencies’ policies did not  fully address all three components. Specifically, only VA fully addressed  the three components; Defense partially addressed the majority of them;  and HHS, DHS, and Transportation did not address the majority of  components.
Table 1 provides a detailed assessment of each agency’s policies against  the three key components of an incremental development policy. In  addition, a discussion of each policy component follows the table.
SEI, CMMI-ACQ, Version 1.3 (November 2010). component.  ●=Fully met—the agency provided evidence that addressed the component.  ◐=Partially met—the agency provided evidence that addressed about half or a large portion of the  ○=Not met—the agency did not provide evidence that addressed the component or provided  evidence that minimally addressed the component.
Require delivery of functionality every 6 months. Only one of the five  agencies—VA—fully addressed this policy component by clearly  requiring that its projects be completed in increments that must not  exceed 6 months. The other four agencies did not address this  policy component. Three of these agencies—Defense, HHS, and  DHS—all developed policies that promote the use of incremental  development, but these policies do not require functionality to be  delivered every 6 months. Specifically, with regard to Defense,  although the department’s acquisition framework calls for investments  to use incremental development, its policy on IT budget submissions  encourages investments to deliver functionality every 12-18 months— not every 6 months. According to officials of the Defense Office of  the CIO, 12-18 month incremental development is better aligned with  the acquisition framework that many of its IT acquisitions have used.
For HHS, although its policy requires incremental development, its  policy recommends—but does not require—that all projects deliver  functionality every 3-6 months. According to an HHS Office of the CIO  official, HHS does not require its projects to deliver functionality every  6 months because it wants to provide projects with flexibility. For  DHS, in June 2012, the former DHS CIO issued a draft policy  encouraging IT projects to move towards Agile development  approaches. Additionally, with respect to financial systems  modernization programs, DHS’s policy calls for providing financial  capabilities to the customer in small increments of 6-12 months.  According to DHS officials representing the Office of the CIO, the  department is currently developing a departmentwide policy on  incremental development; however, they said that the draft currently  encourages investments to deliver the first release 18 months after  program initiation and thereafter deploy functionality in cycles no  longer than 12 months, but ideally less than 6 months. Lastly,  Transportation also did not address the component because, although  Transportation has a policy that calls for projects to deliver  functionality every 6 months, officials from the Office of the CIO  explained that this policy does not apply to the Federal Aviation  Administration (FAA). These officials explained that how often FAA  projects deliver functionality depends on their size, scope, risk,  visibility, and interdependencies with other programs.
Define functionality. Only one of the five agencies—VA—fully  addressed this policy component. VA has a policy that defines what it  means to deliver functionality—both in terms of what constitutes an  increment and what should be delivered at the end of an increment.  For example, VA defines an increment as the segment of the project  that produces, in a cycle of 6 months or less, a deliverable that can be  used by customers in an operational environment. For the agency  that partially addressed the component—Defense—although it has  defined functionality for purposes of its acquisition framework, it has  not defined the functionality that its IT budget submission policy  encourages projects to deliver every 12-18 months. The department  stated that it is working with OMB to define this term. Lastly, three  agencies—HHS, DHS, and Transportation—had not defined  functionality in terms of what they expected projects to deliver at the  end of a development cycle. Officials representing these agencies’  respective Office of the CIO acknowledged that they have not defined  functionality. These officials told us that they would update their  policies to define the term, but officials from HHS and Transportation  did not provide a time frame for doing so. DHS officials stated that,  although they did not have a definitive timeframe, they hoped to  finalize the policy in 2014. Until the agencies define this term,  investments may create definitions that are inconsistent with the intent  of OMB’s policy.
Define a process for enforcing compliance. Only one of the five  agencies—VA—fully addressed this policy component by defining  processes for ensuring that increments are structured to deliver  functionality every 6 months or less and for reviewing projects that fall  behind schedule. In particular, VA’s policy requires the agency to hold  a TechStat session when any increment delivery date has been or will  be missed. Two agencies partially addressed this component— Defense and HHS—because, although they established processes for  ensuring that IT is acquired incrementally, these processes do not (1)  require enforcement of incremental development within the specific  time frames consistent with OMB guidance (12-18 months for  Defense and 3-6 months for HHS) or (2) include using TechStat  processes to identify investments that are not being acquired  incrementally. Finally, two agencies—DHS and Transportation—have  not established processes for enforcing compliance with their  incremental development policies. Officials from their respective  Office of the CIO told us that they are updating their policies to  address this issue. Transportation officials representing the Office of  the CIO stated that it would update its policy later this year; DHS  Office of the CIO officials stated that, although they did not have a  definitive timeframe, they hoped to finalize their policy in 2014.
Agencies cited several underlying reasons that contributed to these  weaknesses: (1) they were not always aware of OMB guidance, (2) they  did not believe that the guidance was realistic, and (3) they said the  guidance was not always clear. Regarding agency awareness of the  guidance, since the 2010 IT Reform Plan, OMB has communicated  changes to incremental development requirements, such as the change  from 12 to 6 months, through budget guidance. However, selected  agency officials said they were not always aware of this guidance. For  example, DHS Office of the CIO officials told us that did not know about  OMB’s requirement to deliver functionality every 6 months. Additionally,  Transportation officials representing the Office of the CIO were not aware  that in 2012 OMB had changed its guidance from recommending to  requiring that projects deliver in 6-month cycles.
With respect to whether OMB’s guidance is realistic, officials from  Defense, HHS, DHS, and Transportation explained that they do not want  to require all of their projects to deliver functionality every 6 months  because it may not be reasonable for all investments to do so. Defense,  DHS, and Transportation officials said that delivering every 12 months, as  advocated in OMB’s IT Reform Plan, is more reasonable.Defense officials from the Office of the CIO, 12-18 month incremental  development is better aligned with the acquisition framework that many of  its IT investments have used. DHS and Transportation Office of the CIO  officials stated that, depending on program size, scope, complexity,  budget, schedule, and expertise, it may be more reasonable to deliver  functionality every 12 months. As discussed later in this report, we agree  that OMB’s requirement to deliver functionality every 6 months is  unrealistic. OMB staff members from the Office of E-Government and  Information Technology noted that it will take time for agencies to  embrace delivering functionality every 6 months because of the perceived  risk of adopting such approaches. Those staff members explained that  agencies, such as Defense, have been using the waterfall development  method for many years, and they perceive less risk in continuing with that  method than with changing to a method that produces functionality more  rapidly.
Lastly, two key components of OMB’s guidance are not clear. First, in  revising Circular A-130 in 2000, OMB did not identify the minimum  requirements of what the agencies’ policies are to include and did not  specify when the policies are to be completed. Although OMB issued later  guidance on incremental development, it has not yet specified what  agencies’ incremental development policies are to include. Second,  OMB’s guidance did not provide a complete definition of the functionality  it expects to be delivered every 6 months. According to staff from the  Office of E-Government, OMB intends for agencies to deliver functionality  that can be tested by business users; nevertheless, they noted that they  left this definition out of their guidance so that agencies could develop a  definition that would be flexible enough to meet their needs. However, in  the absence of further guidance from OMB, agencies may continue to not  define this term or may create definitions that are inconsistent with the  intent of OMB’s policy. For example, HHS officials from the Office of the  CIO told us that the completion of requirements documents could meet  OMB’s definition of delivering functionality. Additionally, an FAA Office of  the CIO official explained that some investments have classified the  delivery of requirements documentation as functionality. These two  examples are not consistent with OMB’s intent since they do not deliver  functionality that can be tested, but instead only plan to deliver project  documentation.
Until OMB explicitly issues realistic and clear guidance and Defense,  HHS, DHS, and Transportation address the identified weaknesses in their  incremental development policies, it will be difficult to deliver project  functionality more rapidly, measure how often projects are delivering  functionality, and enforce compliance with the delivery time frames called  for in their policies.
section_title: Most Selected Investments Do Not Plan to Deliver Functionality Every 6 or 12 Months
paragraphs: In its 2010 IT Reform Plan, OMB called for IT programs to deliver  functionality at least every 12 months. Subsequently, OMB has made  this requirement more stringent in that it now requires projects associated  with major IT investments to deliver functionality every 6 months.
The majority of the selected investments we reviewed did not plan to  deliver functionality every 6 months. Specifically, only 23 of the selected  89 investments had one or more projects that, when taken collectively,  planned to deliver functionality every 6 months. To VA’s credit, all six of  the department’s selected investments planned to deliver functionality  every 6 months. The other agencies varied in the extent to which they  met the standards established by OMB’s guidance. Table 2 shows how  many of the selected investments at each agency planned on delivering  functionality every 6 months during fiscal years 2013 and 2014.
The variety of life-cycle cost estimates for these investments shows that  incremental development can be applied to a wide variety of investment  scopes. Specifically, of the 23 investments that planned to deliver  functionality in 6-month cycles, 9 had cost estimates that were less than  $250 million, 4 had estimates between $250 and $575 million, 5 had  estimates between $575 million and $2 billion, and 5 had estimates  greater than $2 billion.
Twenty-seven of the 89 investments in our review (30 percent) reported  using an Agile development methodology for one or more of their  projects. Of those 27 investments, 14 (52 percent) planned to deliver  functionality every 6 months. The other 13 investments using Agile did  not plan on delivering functionality as frequently as OMB guidance  requires. We have previously found that Agile projects typically produce  working functionality every 1 to 8 weeks; as such, it appears that these  investments may not be properly implementing an Agile development  methodology.
Agency officials cited three types of investments for which it may not  always be practical or necessary to expect functionality to be delivered in  6-month cycles: (1) investments in life-cycle phases other than acquisition  (i.e., planning and budgeting, management in-use, and disposition); (2) investments intended to develop IT infrastructure; and (3) research and  development investments.
Life-cycle phases other than acquisition. Officials from Defense, HHS,  DHS, and Transportation stated that it is not reasonable to expect  investments to deliver functionality every 6 months when their  investments’ projects are not in the acquisition phase. Specifically,  24 investments did not have projects in the acquisition stage. Of those  24 investments, 22 did not plan to deliver functionality in 6-month  cycles (10 from Defense, 3 from HHS, 1 from DHS, and 8 from  Transportation), and 2 investments did plan to do so (2 from HHS).  For the 2 investments that planned to deliver functionality every 6  months, both had at least one project in the management in-use  phase, meaning that at least one of the investments’ projects were  beyond the planning and development stages and were being used to  support agency operations.
Infrastructure investments. Officials from Defense, DHS, and  Transportation explained that not all infrastructure investments can be  expected to deliver functionality every 6 months. Specifically, 21  investments provide infrastructure, such as IT security, office  automation, and telecommunications. For example, officials  representing two of the DHS investments explained that, prior to  deploying functionality, they need to acquire real estate, conduct  environmental assessments, and perform construction work, such as  digging trenches, burying cables, and building facilities. Of the 21  investments, 20 did not plan to deliver functionality every 6 months  (17 from Defense, 1 from HHS, and 2 from DHS); however, 1  investment did plan to do so (1 from Defense).
Research and development investments. Officials from FAA’s Office  of the CIO explained that FAA’s research and development  investments are not intended to deliver functionality. Those officials  stated that, before the agency approves a technology for further  development, it performs research and development to ensure that  the technology meets safety standards. If those standards are met,  FAA creates a new investment aimed at deploying that technology.  Consistent with this, none of FAA’s six research and development  investments planned to deliver any functionality during fiscal years  2013 and 2014.
These concerns have merit. For example, with respect to investments in  life-cycle phases other than acquisition, at the outset of a new investment,  an agency may need more than 6 months to, among other things, define  high-level requirements and find a contractor to help develop the system.  In addition, it may not be necessary for an investment to continue  delivering new functionality when all planned functionality has been fully  deployed.
Regarding infrastructure investments, it may not be practical or cost- effective for an investment to refresh fully functioning hardware every 6  months. Additionally, it may not be feasible for an agency to build a  physical facility (e.g., data center) within 6 months. Further, for research  and development investments, industry practices and our work on best  practices support FAA’s efforts to thoroughly validate the feasibility and  cost-effectiveness of new technology prior to making significant  investments.
Although OMB requires all investments to deliver functionality every 6  months, an OMB staff member from the Office of E-Government  explained that not all investments will be able to meet this goal. Instead,  that staff member said that about half of the federal government’s major  IT investments will deliver functionality in 6 months or less, and the other  half will have longer development cycles. However, OMB’s guidance does  not make this distinction. As a result, agencies may be confused about  whether OMB’s incremental development guidance applies to all  investments.
If these three types of investments, which account for 40 of the selected   29 of the remaining 49 investments  89 investments, are not considered,did not plan to deliver functionality in 6-month cycles. Table 3 shows, after  removing the three types of investments discussed above, how many of  the selected investments at each agency planned to deliver functionality  every 6 months during fiscal years 2013 and 2014.
Considering agencies’ concerns about delivering functionality every 6  months for the three types of investments discussed above and OMB’s  own expectations that many investments will not meet this goal, it is  unclear whether this is the most appropriate governmentwide goal, and it  raises the question of whether OMB should consider a longer time frame,  such as 12 months, as called for in OMB’s IT Reform Plan. However,  even using the time frame of 12 months as the target, less than half of the  selected investments planned to deliver functionality in 12-month cycles.  Specifically, 41 of the 89 selected investments planned to deliver  functionality every 12 months during fiscal years 2013 and 2014 and 48  did not. Most notably, the preponderance of Defense and Transportation  investments (70 percent for Defense and 65 percent for Transportation)  did not plan to deliver functionality every 12 months. Table 4 shows how  many of the selected investments at each agency planned on delivering  functionality every 12 months during fiscal years 2013 and 2014.
The previously discussed weaknesses in agency policies have permitted  the inconsistent implementation of incremental development approaches.  OMB staff members from the Office of E-Government acknowledged that  inconsistent implementation of OMB’s guidance can be at least partially  attributed to challenges in ensuring that agencies develop consistent  definitions of and approaches to incremental development.
Although OMB has led the government’s recent effort to improve  incremental development, it has not completed a key commitment aimed  at improving incremental development—namely, OMB has rarely used  the TechStat process to turn around or cancel investments that are not  using incremental development. As previously mentioned, TechStat  sessions are intended to terminate or turn around IT investments that are  failing or are not producing results. Additionally, many failed IT  investments have not used incremental development approaches.  Therefore, OMB could use TechStat sessions as a powerful tool to turn  around investments that are not being acquired incrementally. However,  OMB staff members from the Office of E-Government said that OMB has  only held one such TechStat.
OMB staff from the Office of E-Government explained that, in order to  select investments in need of a TechStat session, agency exhibit 300  data are reviewed for evidence of poor performance. However, the  usefulness of the exhibit 300 data for the purpose of identifying whether  investments are using incremental approaches is limited. For example,  OMB does not require agencies to explicitly identify whether their  investments are using incremental approaches. Additionally, of the 89  selected investments, 34 had activities in their exhibit 300 submissions  that were inaccurately classified as delivering functionality (9 from  Defense, 8 from HHS, 5 from DHS, 9 from Transportation, and 3 from  VA). For example, one Defense investment indicated that awarding a  contract constituted a delivery of functionality. OMB staff from the Office  of E-Government acknowledged that the exhibit 300 data are not as  helpful as possible in addressing incremental development.  Consequently, for its TechStat reviews, OMB’s insight into investments’  use of incremental development approaches is limited. Officials from  Defense, HHS, DHS, and Transportation attributed the problem to a lack  of guidance from OMB on what is to be delivered every 6 months.  Nevertheless, officials from Defense, DHS, and VA stated that they would  properly classify activities in future exhibit 300 submissions.without additional guidance from OMB, agencies may continue to  improperly classify activities. Additionally, as previously mentioned, four  of the five selected agencies—Defense, HHS, DHS, and Transportation— have not updated their TechStat policies to include identifying  investments that are not being acquired incrementally.
Without better implementation of incremental development approaches  and identification of investments using incremental development, IT  expenditures will likely continue to produce disappointing results— including large cost overruns, long schedule delays, and questionable  mission-related achievements. Further, without useful information,  including whether investments are following an incremental approach, on  projects associated with major IT investments, OMB does not have the  necessary information to oversee the extent to which projects and  investments are implementing its guidance.
section_title: Multiple Factors Were Commonly Identified as Enabling and Inhibiting Incremental Development
paragraphs: Multiple factors were identified by the five agencies as enabling and  inhibiting incremental development during a 6-month period. Specifically,  eight factors were identified by three or more of the five agencies in our  review as enabling incremental development of IT systems, and seven  factors were identified by three or more agencies as inhibiting incremental  development. The enabling factor identified by all of five of the agencies  was active engagement of program officials with stakeholders. The  inhibiting factors identified by all five agencies were (1) the lack of  sufficient, timely funding; (2) program characteristics that made rapid  incremental development infeasible; and (3) the lack of stable, prioritized  requirements.
subsections: section_title: Eight Factors Were Commonly Identified as Enabling Incremental Development during a 6- month Period
paragraphs: Eight factors were identified by three or more of the five agencies in our  review as contributing to the successful development of functionality in 6-  month cycles. All five of the agencies in our review cited active program  engagement with stakeholders. Table 5 shows the distribution of the eight  factors among the agencies, and examples of how the agencies  implemented them are discussed following the table.
Program officials actively engaged with stakeholders. Officials from all  five agencies explained that active engagement with program  stakeholders—individuals or groups with an interest in the success of  the investment—was a factor that enabled the development of  functionality in 6-month cycles. For example, officials from one of the  HHS investments that we reviewed stated that having strong  communication between the business program, the IT program, and  contractors enabled projects to move forward on schedule in a  cohesive manner with clear goals and objectives.
Programs used an Agile development methodology. Officials from four  of the five agencies indicated that the use of an Agile development  methodology helped them to deliver functionality every 6 months. For  example, Defense officials explained that the use of Agile  development processes allowed software to be broken into smaller  releases that were easily achieved, tested, and fielded. Additionally,  VA officials stated that the agency has embraced Agile development,  which has helped the department deliver functionality more quickly.  Those officials also noted that merely forcing investments to use 6- month waterfall iterations would not have resulted in the same  success.
Programs successfully prioritized, managed, and tested requirements.
Officials from four of the five agencies identified implementation of  requirements management practices—including requirements  prioritization, management, and testing—as a factor that enabled the  agencies to deliver functionality every 6 months. For example,  Transportation officials explained that the primary factor that enabled  incremental development has been obtaining clear requirements.  Further, HHS officials stated that the use of a prioritized product  backlogrequirements should be allocated to future releases.  has helped teams make decisions about which
Staff had the necessary skills and experience. Officials from four of  the five agencies stated that consistent and stable program staff with  experience and expertise allowed them to deliver functionality  frequently. For example, DHS officials representing one of the  selected investments indicated that having skilled program managers  has a large impact on the success of a program. In addition, VA  officials reported that their ability to retain key personnel with both  functional and technical expertise has enabled them to deliver high- quality functionality rapidly.
Programs successfully implemented cost and schedule estimating  best practices. Officials from four of the five agencies cited the  implementation of cost and schedule estimating practices as enabling  the frequent delivery of functionality. For example, Defense officials  told us that development of realistic cost estimates and  comprehensive program schedules helped programs to deliver  functionality while meeting established performance goals. Further,  Transportation officials stated that improved guidance and training on  cost estimating and scheduling helped them to deliver functionality in  6-month cycles.
Officials used various contracting strategies to increase flexibility and  improve contractor oversight. Officials from four of the five agencies  cited the use of specific contracting strategies. For example, DHS  officials indicated that they worked with contractors to modify their  contracts from the traditional waterfall approach to a structure that  allows for Agile development. In addition, officials from VA stated that  the use of performance-based acquisitions assisted them in  monitoring contractor progress towards achieving actual results  against planned objectives.
Staff used key technologies to accelerate development work. Officials  from three of the five agencies indicated that the use of key  technologies enabled them to deliver functionality more quickly. For  example, HHS officials explained that having an established cloud  environment has enabled them to reduce deployment time for  releases, while also providing the needed flexibility to meet their  customers’ changing needs. Additionally, DHS officials explained that  projects, especially those using an Agile development methodology,  have saved time using automated testing tools.
Programs successfully implemented risk management best practices.
Officials from three agencies explained that the successful  implementation of risk management practices helped them to deliver  functionality more rapidly. For example, officials from one of the  selected VA investments told us that the organization’s risk  management process allowed teams to quickly escalate risks to  senior leadership so that they could be managed before their impacts  were fully realized.
Many of the factors that the five agencies cited are consistent with our  2011 work on factors critical to successful IT acquisitions and our 2012  work on effective practices in implementing Agile software development.In particular, our work in both areas discussed the importance of active  engagement with program stakeholders, such as actively engaging with  stakeholders to obtain customer feedback, and effectively managing  requirements.
Additionally, the eight commonly identified factors that enable the delivery  of functionality in 6 months are consistent with OMB’s IT Reform Plan.  In particular, as previously mentioned, one high-level objective of the  plan—effectively managing large-scale IT programs—aims to improve  areas that impact the success rates of large IT programs by, among other  things, enabling programs to use incremental development approaches.  As part of this high-level objective, the plan addresses the importance of  actively engaging with stakeholders, ensuring that program management  professionals have proper skills and experience, and developing new  contracting strategies that more effectively support incremental  development.
section_title: Seven Factors Were Commonly Identified as Inhibiting Incremental Development during a 6- month Period
paragraphs: Seven factors were identified by three or more of the five agencies as  inhibiting the development of IT functionality every 6 months. The factors  most commonly cited include (1) programs did not receive sufficient  funding or receiving funding later than needed, (2) program  characteristics made rapid delivery of functionality infeasible or  impractical, (3) programs did not have stable and prioritized requirements.  Table 6 shows the distribution of the seven factors among the agencies,  and examples of how the factors impacted the agencies are discussed  following the table.
Programs did not receive sufficient funding or received funding later  than needed. Officials from all five departments cited insufficient  funding, such as reductions caused by the fiscal year 2013 sequester,  or receiving funding later than needed because of continuing  resolutions. For example, Defense officials representing one of the  investments we reviewed stated that furloughs brought on by the 2013  sequester significantly impacted program schedules, both in terms of  lost work days and the inability to coordinate integration between  software providers, resulting in overall inefficiencies. In addition,  several FAA officials explained that the delivery of planned  functionality was adversely affected by the uncertainty brought about  by the 2013 sequester, and that future funding instability has impacted  the agency’s ability to plan when functionality is to be delivered.
Program characteristics made rapid delivery of functionality infeasible  or impractical. Officials from all five agencies indicated that some of  their programs have certain characteristics—such as the deployment  of new physical infrastructure, human health and safety concerns, or  external dependencies—that make the delivery of functionality in 6- month time frames infeasible or impractical. For example, DHS  officials explained that their infrastructure projects cannot deliver  functionality until all key activities (e.g., land acquisition,  environmental assessments, site preparation, and construction) have  been completed and the new infrastructure has been fully deployed,  tested, and accepted. Additionally, Transportation officials reported  that air traffic control systems require years of development and  testing in order to ensure that the systems do not compromise  airspace security.
Programs did not have stable, prioritized requirements. Officials from  all five agencies stated that not having complete requirements at the  beginning of their investments’ development or having changes made  to requirements and their relative priorities during development was a  factor that negatively affected their programs’ delivery of incremental  functionality in 6-month periods. For example, HHS officials  representing one of the selected investments explained that that final  rules detailing eligibility for a medical program were not completed  until late in the development cycle; consequently, the program could  not define the corresponding business requirements in time to meet  the development schedule. Further, VA officials stated that schedules  are disrupted when stakeholders identify new program requirements  that must be delivered before others.
Development work was slowed by inefficient governance and  oversight processes. Officials from three of the five agencies cited  inefficient governance and oversight processes. For example, DHS  officials representing the Office of the CIO stated that the current DHS  governance model was not conducive to frequent delivery of  functionality. To illustrate, those officials noted that it can take up to 2  months to schedule a meeting with DHS review boards prior to  releasing functionality. However, a DHS official from Program  Accountability and Risk Management disagreed with this statement,  explaining that DHS’s acquisition review boards perform reviews very  quickly, and that any delays in completing these reviews are  attributable to investments being unprepared. Further, DHS Office of  the CIO officials suggested that governance over programs using an  Agile development methodology should be performed at the lowest  practicable level of the organization. In addition, VA officials explained  that, although the periodic approvals required of various department  groups are useful, these approvals can slow the testing and release  processes and could use further streamlining.
Development schedules were impeded by procurement delays.
Officials from three of the five agencies explained that procurement  delays—such as delays in getting contracts awarded or approving  contract modifications—contributed to difficulties in delivering  incremental functionality. For example, DHS officials explained the  process of planning for an acquisition, developing solicitations,  selecting contractors, and, in some cases, addressing protests, are  not conducive to delivering functionality in 6-month cycles.
Program staff were overutilized or lacked the necessary skills and  experience. Officials from three of the five agencies indicated that  they did not have enough program staff with the expertise and  experience necessary to deliver functionality every 6 months. For  example, officials representing one of the HHS investments stated  that the loss of key personnel and the lack of staff with knowledge and  skill in key disciplines, such as incremental development, have  negatively impacted the agency’s ability to deliver functionality.
Incremental development was impeded by select technologies.  Officials from three of the five agencies in our review explained that  the software and tools they selected for their programs either  introduced delays in development or were ultimately not usable by the  program. Transportation officials representing one of the selected  investments reported that their chosen technology added a new level  of complexity to storage and data processing that required all  development and testing to be completed before the implementation  could occur. As a result, those officials told us that all functionality had  to be deployed in one release, instead of being spread out in stages.
Many of the factors that the five agencies identified as inhibiting delivery  of functionality in 6-month increments were consistent with our work on  the challenges we identified in the application of Agile software  development methods.deploying software developed using an Agile development method is that  traditional governance and oversight activities were difficult to execute   For example, we noted that one challenge in  within an iterative time frame. In particular, one agency official stated that  completing the necessary reviews within the short, fixed time frame of an  Agile iteration was difficult because reviewers followed a slower waterfall  schedule with reviews that could take months to perform after the  completion of an iteration. This caused delays for iterations that needed  such reviews within the few weeks of the iteration. We also noted that  procurement practices sometimes delay Agile development.
Further, OMB’s IT Reform Plan also identified some of the same  problems that inhibit incremental development and described solutions to   In particular, as previously mentioned, one item of the  address them.plan was to improve funding of programs by working with Congress to  create IT budget models that promote IT budget flexibility (such as  multiyear budgets or revolving funds). In addition, the plan addresses the  importance of streamlining IT governance by strengthening the quality  and timing of oversight activities.
The factors identified in this report as enabling and inhibiting incremental  development could help agencies address the challenges they face in  incrementally acquiring IT.
section_title: Conclusions
paragraphs: Given the enormous size of the federal government’s investment in IT  and the often disappointing results from IT development efforts—many of  which are attributable to the use of a “big bang” approach—finding ways  to improve the quality and timeliness of agencies’ investments is  important. Congress, OMB, and our work support the use of incremental  development practices. However, although the selected agencies have  developed policies that address incremental development, most of the  agencies’ policies have significant weaknesses. With the exception of VA,  these policies do not fully define functionality, and do not have a complete  process for ensuring that the agencies’ investments are developed  incrementally, including the use of TechStat sessions to enforce  compliance with incremental development policies. In the absence of  such policies, agencies continue to run the risk of failing to deliver major  investments in a cost-effective and efficient manner.
Regarding implementation of incremental development, slightly more than  one-fourth of selected investments planned to deliver functionality every 6  months—and less than one-half planned to do so every 12 months. Thus,  delivering functionality every 6 months is not an appropriate requirement  for all agencies given current performance. Requiring the delivery of  functionality every 12 months, consistent with OMB’s IT Reform Plan,  would be an appropriate starting point and be a substantial improvement.  Further, since there are three types of investments for which it may not  always be practical or necessary to expect functionality to be delivered in  6-month cycles, this raises questions about whether OMB’s requirement  for shorter delivery cycles should be applied to all investments or whether  investments should be allowed the latitude to determine a more  appropriate time frame.
The lack of progress in updating policies and implementing OMB’s  guidance was enabled by weaknesses in OMB’s guidance. In the  absence of agency and OMB use of TechStat sessions to ensure  compliance with incremental development policy, investments will  continue to be at risk of not delivering promised capabilities on time and  within budget. Until OMB clearly and explicitly disseminates guidance with  realistic goals and clear expectations, and agencies update their policies  to reflect this guidance, agencies may not consistently adopt incremental  development approaches, and IT expenditures will continue to produce  disappointing results—including sizable cost overruns and schedule  slippages, and questionable progress in meeting mission goals and  outcomes. Additionally, without useful information on how often projects  are delivering functionality, it will be difficult for OMB to identify  investments that are not implementing its guidance. Further,  dissemination of the factors identified in this report as enabling and  inhibiting incremental development may help federal agencies address  the challenges they face in acquiring IT investments faster and more  efficiently.
section_title: Recommendations for Executive Action
paragraphs: We recommend that the Director of the Office of Management and  Budget direct the Federal Chief Information Officer to take the following  two actions.
Update, and clearly and explicitly issue incremental development  guidance that addresses the following three components:  requires projects associated with major IT investments to deliver  incremental functionality at least every 12 months, with the  exception of the three types of investments identified in this report;  specifies how agencies are to define the project functionality that  is to be delivered; and requires agencies to define a process for enforcing compliance  with incremental functionality delivery, such as the use of  TechStat sessions.
Require agencies to clearly identify on exhibit 300 submissions  whether, for each project, functionality will be delivered within the time  frames called for by this incremental development guidance, and to  provide justification for projects that do not plan to do so.
We further recommend that the Secretaries of Defense, Health and  Human Services, Homeland Security, and Transportation take the  following two actions: modify, finalize, and implement their agencies’ policies governing  incremental development to ensure that those policies comply with  OMB’s guidance, once that guidance is made available; and when updating their policies, consider the factors identified in this  report as enabling and inhibiting incremental development.
We also recommend that the Secretary of Veterans Affairs consider  incorporating the factors identified in this report as enabling and inhibiting  incremental development in the department’s related policy.
section_title: Agency Comments and Our Evaluation
paragraphs: We received comments on a draft of this report from OMB and the five  agencies in our review. OMB agreed with one recommendation and  partially disagreed with the other; Defense generally concurred with the  report; HHS neither agreed nor disagreed with the report’s  recommendations; DHS agreed with our recommendations;  Transportation did not agree with the recommendations in that it did not  believe the department should be dependent on OMB first taking action;  and VA generally agreed with the report’s conclusions and concurred with  our recommendation. Each agency’s comments that we received are  discussed in more detail below.
In comments provided via e-mail on April 15, 2014, staff in OMB’s  Office of General Counsel, on behalf of OMB, stated that the agency  agreed with one of our recommendations and partially disagreed with  the other. Specifically, OMB agreed with our recommendation to  require agencies to clearly identify on exhibit 300 submissions  whether functionality will be delivered within the time frames called for  by this incremental development guidance.
OMB stated that it agreed with our recommendation to update and  issue incremental development guidance, but did not agree that this  guidance should require major IT investments to deliver incremental  functionality at least every 12 months. OMB explained that changing  the requirement from 6 to 12 months would reduce the emphasis on  incremental development that it has been advocating. OMB also  noted that it believes requiring investments to deliver functionality  every 6 months is an appropriate governmentwide goal and said that  updating and clarifying its guidance on incremental development will  make it easier for agencies to meet this target. However, as we state  in this report, slightly more than one-fourth of selected investments  planned to deliver functionality every 6 months—and less than one- half planned to do so every 12 months. Additionally, there are three  types of investments for which it may not always be practical or  necessary to expect functionality to be delivered in 6-month cycles.  Thus, we continue to believe that delivering functionality every 6  months is not an appropriate requirement for all agencies and that  requiring the delivery of functionality every 12 months, consistent with  OMB’s IT Reform Plan, is a more appropriate starting point. We  therefore maintain that OMB should require projects associated with  major IT investments to deliver functionality at least every 12 months.
In written comments, Defense stated that it generally concurred with  the report and outlined planned actions to address the  recommendations. However, Defense explained that many of its IT  investments are consistent with the three types of investments for  which it may not always be practical or necessary to expect  functionality to be delivered in 6-month cycles or have other  exceptional circumstances. Given these issues, Defense stated that  requiring the delivery of functionality every 12 months is not a useful  management constraint. However, as we state in our report, many  failed projects have been broadly scoped in that they aim to deliver  their capabilities several years after initiation. Additionally, the  Defense Science Board reported that Defense’s acquisition process  for IT systems was too long, was ineffective, and did not  accommodate the rapid evolution of IT. In order to resolve these  issues, we continue to believe that investments and their projects  should deliver functionality more frequently. In addition, the existence  of many investments that are consistent with the three types of  investments identified in this report for which it may not be practical to  deliver functionality quickly does not excuse Defense from the  requirement of delivering functionality every 12 months. Consistent  with our recommendation to OMB, investments and projects that meet  these three exceptions should be exempt from the requirement, but all  other investments and projects should aim to meet this requirement.  Further, we expect that Defense’s actions to implement our  recommendation to consider the factors identified in this report that  enable and inhibit incremental development will help its investments  and projects deliver functionality more rapidly. As such, we continue  to believe that requiring the delivery of functionality every 12 months  is an appropriate goal for all federal agencies, including Defense.  Defense’s comments are reprinted in appendix III. The department  also provided technical comments, which we have incorporated in the  report as appropriate.
In comments provided via e-mail on April 15, 2014, an official from  HHS’s Office of the Assistant Secretary for Legislation, on behalf of  HHS, stated that the department had no comments.
In written comments, DHS stated that it concurred with our  recommendations and outlined planned actions to address the  recommendations. DHS’s comments are reprinted in appendix IV. The  department also provided technical comments, which we have  incorporated in the report as appropriate.
In comments provided via e-mail on April 15, 2014, Transportation’s  Deputy Director of Audit Relations, on behalf of Transportation, stated  that the department would prefer to have specific recommendations  and deliverables so that it can achieve success in closing them.  Specifically, the department explained that relying on another agency  to concur with one of our recommendations before Transportation can  take action leaves the department with the potential challenge of a  recommendation that cannot be implemented. However, as previously  stated, OMB agrees with our recommendation to update and issue  incremental guidance, meaning that OMB has committed to taking the  actions necessary to enable Transportation to begin addressing our  recommendation. Additionally, our recommendation to consider the  factors identified in this report as enabling and inhibiting incremental  development when updating incremental development policies does  not require another agency to take action before Transportation can  implement it. Accordingly, we continue to believe that our  recommendations are warranted and can be implemented.
In written comments, VA stated that it generally agreed with the  report’s conclusions, that it concurred with our recommendation to the  department, and that it will review its existing policy for incremental  development and consider incorporating the factors identified as  enabling and inhibiting incremental development. VA’s comments are  reprinted in appendix V. The department also provided technical  comments, which we have incorporated in the report as appropriate.
As agreed with your offices, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 30 days from the  report date. At that time, we will send copies of this report to interested  congressional committees; the Director of the Office of Management and  Budget; and the Secretaries of the Departments of Defense, Health and  Human Services, Homeland Security, Transportation, and Veterans  Affairs. In addition, the report will also be available at no charge on our  website at http://www.gao.gov/.
If you or your staffs have any questions on matters discussed in this  report, please contact me at (202) 512-9286 or pownerd@gao.gov.  Contact points for our Offices of Congressional Relations and Public  Affairs may be found on the last page of this report. GAO staff who made  major contributions to this report are listed in appendix VI.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: Our objectives for this review were to (1) assess whether selected  agencies have established policies for incremental information technology  (IT) development, (2) determine whether selected agencies are using  incremental development approaches to manage their IT investments,  and (3) identify the key factors that enabled and inhibited the selected  agencies’ abilities to effectively use incremental development approaches  to manage their IT investments.
In conducting our review, we selected five agencies and a total of 89  investments from those agencies. To choose the agencies, we identified  the five agencies with the largest IT budgets for development,  modernization, and enhancement on major IT investments in fiscal years  2013 and 2014 as reported in the Office of Management and Budget’s  (OMB) fiscal year 2014 exhibit 53. Those agencies are the Departments  of Defense (Defense), Health and Human Services (HHS), Homeland  Security (DHS), Transportation (Transportation), and Veterans Affairs  (VA). To choose the agencies’ investments, we identified the 98 major IT  investments for which the selected agencies planned to spend more than  50 percent of the investments’ fiscal year 2013 and 2014 budgets on  development, modernization, and enhancement as reported in OMB’s  fiscal year 2014 exhibit 53. We removed 9 investments because, after  reporting their planned budgets for OMB’s fiscal year 2014 exhibit 53,  these investments changed their budgets so that they no longer planned  to spend more than 50 percent of the investments’ fiscal year 2013 and   The  2014 budgets on development, modernization, and enhancement.final 89 investments are identified in appendix II. The investments  selected for the review account for about 58 percent of the development,  modernization, and enhancement spending on all federal agencies’ major  IT investments for fiscal years 2013 and 2014 reported in OMB’s exhibit  53 for fiscal year 2014.
To address our first objective, we reviewed OMB guidance related to the  use of incremental development, as well as industry guidance, and  identified three key components of incremental development that  agencies should include in their policies. In order to identify these  components, we reviewed OMB’s guidance and leading industry  guidance on institutionalizing processes throughout an organization.analysis identified three key policy components that will help agencies  effectively implement OMB’s requirement for incremental development.
Require delivery of functionality every 6 months. According to the  Software Engineering Institute’s (SEI) Capability Maturity Model®  Integration for Acquisition (CMMI-ACQ), as part of institutionalizing a  process, organizations should document their processes, to include  defining standards and requirements. According to OMB budget  guidance, projects associated with major IT investments must deliver  functionality every 6 months.
Define functionality. As previously stated, according to SEI’s CMMI- ACQ, as part of institutionalizing a process, organizations should  document that process, to include defining standards and  requirements.
Define a process for enforcing compliance. According to SEI’s CMMI- ACQ, as part of institutionalizing a process, organizations should  document that process, including management review activities (e.g.,  taking corrective action when requirements and objectives are not  being satisfied). Additionally, OMB Circular A-130 requires agencies  to ensure that investments comply with their policies. Further,  according to the President’s Fiscal Year 2014 Budget, agencies are to  use their TechStat processes to identify investments that are not  being acquired incrementally and undertake corrective actions.
At each selected agency, we then analyzed agency policies for  incremental development and compared these policies to the three  components identified above.
For each agency, each policy component was assessed as either being  not met—the agency did not provide evidence that addressed the  component or provided evidence that minimally addressed the  component; partially met—the agency provided evidence that addressed  about half or a large portion of the component; or fully met—the agency  provided evidence that addressed the component. We also interviewed  officials from OMB and the five selected agencies to obtain information  about their current and future incremental development policies.
To address our second objective, we administered a data collection  instrument to each of the selected investments about how often each  investment planned to deliver functionality during fiscal years 2013 and  2014. We then analyzed information obtained from data collection  instruments describing how often the selected investments planned to  deliver functionality. We prepopulated these instruments with data  obtained from OMB’s fiscal year 2014 exhibit 53, as well as each  investment’s exhibit 300 data, which describe the investments’ projects  and how often each project plans to deliver functionality. We asked  officials from each investment to verify the accuracy and completeness of  the data and to make corrections where needed. Because the exhibit 300  data did not always describe the investments’ plans for delivering  functionality in fiscal year 2014, we asked the officials to indicate for each  project whether they planned to deliver functionality in the first half of  fiscal year 2014 (i.e., from October 1, 2013, to March 31, 2014) and in the  second half of fiscal year 2014 (i.e., from April 1, 2014, to September 30,  2014). We also asked the investments to provide their life-cycle cost  estimates and, for each project, the development methodology used and  phase of the acquisition life cycle.
Using the information obtained through the data collection instruments,  we determined the extent to which the selected investments and their  projects planned to meet OMB’s guidance on incremental development.  To assess whether investments had planned to deliver functionality every  6 months, we determined whether the selected investments planned to  deliver functionality in each of the following four time frames: (1) the first  half of fiscal year 2013 (i.e., from October 1, 2012, to March 31, 2013), (2)  the second half of fiscal year 2013 (i.e., from April 1, 2013, to September  30, 2013), (3) the first half of fiscal year 2014, and (4) the second half of  fiscal year 2014. To determine whether investments had planned to  deliver functionality every 12 months, we analyzed whether the selected  investments planned to deliver functionality in each of the following two  time frames: (1) fiscal year 2013 and (2) fiscal year 2014. We presented  our results to the five selected agencies and OMB and solicited their input  and explanations for the results.
To determine the reliability of the exhibit 300 data, we performed three  steps. First, as previously mentioned, we asked officials from each  investment to verify the accuracy and completeness of these data and  provide the correct information where needed. Second, we removed  projects that were not intended to deliver functionality and activities that  were inaccurately classified as resulting in the delivery of functionality. To  do so, we compared the descriptions of the projects and activities with a  definition of functionality that we developed. To develop this definition, we   reviewed OMB and agency guidance, as well as leading practices.We defined functionality as follows: the implementation of IT requirements  that is intended to either (1) ultimately be used by one or more customers  in production (actual deployment may occur at a later date than when the  functionality is reported as being delivered) or (2) be a delivery of a  prototype or pilot. We then assessed the descriptions of the activities and  projects agencies reported in their exhibit 300 submissions against our  definition. We presented the activities and projects that did not meet our  definition to the selected agencies and solicited their input and  explanations. Third, where there was a conflict between the exhibit 300  data and agencies’ answers to our questions regarding plans for  delivering functionality in fiscal year 2014, we presented the conflict to the  agencies and obtained clarification. We determined that the data were  sufficiently reliable for the purpose of this report, which is to determine the  extent to which the selected investments planned to deliver functionality  every 6 and 12 months, during fiscal years 2013 and 2014.
GAO, Information Technology: Critical Factors Underlying Successful Major  Acquisitions, GAO-12-7 (Washington, D.C.: Oct. 21, 2011). methods. Additionally, we compared the factors to OMB’s 25 Point  Implementation Plan to Reform Federal Information Technology  Management.
Further, because Defense guidance encourages investments to deliver  functionality every 12-18 months where possible, we asked the selected  investments and officials from Defense’s Office of the CIO to identify the  key factors that have both enabled and inhibited their efforts to deliver  functionality for their major IT investments every 12-18 months. We  compared the information we received to the eight factors the five  selected agencies commonly identified as enabling incremental  development during a 6-month period and the seven factors commonly  identified by the five agencies as inhibiting incremental development  during a 6-month period. We also performed a content analysis of the  information we received in order to identify additional factors.
We conducted this performance audit from May 2013 to May 2014, in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
section_title: Appendix II: Selected Investments’ Plans for Delivering Functionality
paragraphs: Below is the list of investments that are included in this review, as well as  whether each planned to deliver functionality every 6 and 12 months for  fiscal years 2013 and 2014.
section_title: Appendix III: Comments from the Department of Defense
section_title: Appendix IV: Comments from the Department of Homeland Security
section_title: Appendix V: Comments from the Department of Veterans Affairs
section_title: Appendix VI: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact name above, individuals making contributions to  this report included Dave Hinchman (Assistant Director), Deborah A.  Davis (Assistant Director), Rebecca Eyler, Kaelin Kuhn, Jamelyn Payan,  Meredith Raymond, Kevin Smith, Andrew Stavisky, and Kevin Walsh.