Farm Program Modernization: Farm Service Agency Needs to Demonstrate the Capacity to Manage IT Initiatives
Published: 2015-06-18T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Since 2004, FSA has spent about $423 million to modernize IT systems through a program known as MIDAS. FSA planned for this program to replace aging hardware and software applications and to provide a single platform to manage all of the agency's farm programs. However, the agency experienced significant challenges in managing this program. In July 2014, the Secretary of Agriculture decided to halt MIDAS after the completion of a second software release.
GAO was asked to review the MIDAS program. This report (1) describes what led to the decision to halt further MIDAS development, (2) compares the functionality that MIDAS has implemented to its original plans, and (3) evaluates the adequacy of key program management disciplines in place for MIDAS and successor programs.
To do so, GAO analyzed agency policies and guidance; evaluated program management plans and related artifacts, program and contractor status reports, program milestone artifacts, and lessons learned; obtained a live demonstration of MIDAS; and interviewed agency and contractor officials.
section_title: What GAO Found
paragraphs: The key factors that led to the decision to halt the Modernize and Innovate the Delivery of Agricultural Systems (MIDAS) program were poor program performance and uncertainty regarding future plans. The Farm Service Agency (FSA) experienced significant cost overruns and schedule delays, deferred the majority of the envisioned features, skipped key tests, and deployed software in April 2013 that was slow and inaccurate. In addition, FSA struggled to establish a new program baseline as estimates grew from $330 million to $659 million and time frames were delayed from early 2014 to late 2016. The United States Department of Agriculture (USDA) and FSA did not approve three different baseline proposals by the time the program was halted. By March 2015, MIDAS had overrun its baseline cost estimate by $93 million.
FSA has delivered about 20 percent of the functionality that was originally planned for MIDAS. FSA envisioned MIDAS as a single platform to host data, tools, and applications for administering farm program benefits that would be integrated with USDA financial, geospatial, and data warehouse systems. However, FSA delivered a platform that hosts data for administering farm program benefits and is integrated with USDA's geospatial system; it does not host tools and applications for administering benefits, and is not integrated with USDA's financial system or data warehouse.
FSA did not have key program management disciplines in place for MIDAS, and lacks the capacity to effectively manage successor programs. Of 18 key practices associated with sound IT acquisition and investment management and required by USDA or FSA policy, FSA implemented 2, partially implemented 7 practices, and did not implement 9 others. For example, USDA and FSA did not establish a complete set of requirements, perform key tests before deploying the system, or provide effective oversight as the program floundered for 2 years. Moving forward, FSA has begun planning how it will continue to automate, integrate, and modernize its farm program services through additional system development initiatives. However, the agency has not yet established plans to improve its management capabilities. Until FSA establishes and implements such a plan, the agency will continue to lack the fundamental capacity to manage IT acquisitions. Further, until FSA addresses shortfalls in key program management disciplines on successor programs to MIDAS, the agency will be at an increased risk of having additional projects that overrun cost and schedule estimates and contribute little to mission-related outcomes.
section_title: What GAO Recommends
paragraphs: GAO is making five recommendations to FSA, including establishing and implementing a plan for adopting recognized best practices. GAO received written comments from the FSA administrator. While the agency did not explicitly agree or disagree with the recommendations, it cited steps it has taken or plans to take to implement best practices.

section_title: Letter
paragraphs: Since 2004, the United States Department of Agriculture’s (USDA) Farm  Service Agency (FSA) has spent about $423 million through March 2015  to modernize the information technology (IT) systems that deliver benefits  to farmers and ranchers through its Modernize and Innovate the Delivery  of Agricultural Systems (MIDAS) program. MIDAS was originally  envisioned to replace aging hardware and associated software  applications by developing a single platform to manage all of FSA’s farm  programs. However, the agency has experienced significant challenges in  managing this program, including problems in developing requirements,  establishing reliable cost and schedule estimates, and implementing  sound governance mechanisms. We have previously reported on FSA’s  shortfalls and made recommendations to address them.
After nearly a decade of planning and development, in April 2013, FSA  deployed initial MIDAS functionality. However, senior managers  expressed concerns regarding the program’s performance and delays in  defining the cost, schedule, and scope for the remaining elements of  MIDAS. As a result, in July 2014, the Secretary of Agriculture halted any  new development on MIDAS after its second software release (in  December 2014) and FSA deferred remaining development to future IT  projects.
Subsequently, you asked us to review the circumstances surrounding the  decision to halt further development on USDA’s MIDAS program. Our  objectives are to (1) describe what led to the decision to halt further  MIDAS development, (2) compare the functionality that MIDAS has  implemented to its original plans, and (3) evaluate the adequacy of key  program management disciplines in place for MIDAS and successor  programs.
To describe what led to the decision to halt further MIDAS development,  we reviewed documentation such as program planning artifacts, status  reports, key milestone reviews, and departmental or external reviews of  MIDAS. We identified events and decisions that had significant impacts  on MIDAS’s cost, schedule, scope, and performance from the program’s  initial requirements review in December 2011 through the July 2014  decision to halt further development on MIDAS. To compare the intended  functionality that MIDAS has implemented to its original plans, we  identified what features were delivered by obtaining a demonstration of  the MIDAS system, interviewing service center employees, and reviewing  program artifacts—such as system test reports, program milestone  documentation, and requirements artifacts. We compared the features  that were delivered to those outlined in the program’s initial set of  requirements.
To evaluate the extent to which FSA has implemented key IT program  management disciplines, we assessed the implementation of key  practices and standards identified by the Project Management Institute,  the Software Engineering Institute at Carnegie Mellon University, and  GAO in the areas of (a) requirements development and management, (b)  system testing, (c) project planning and monitoring, and (d) executive  governance.and FSA policies and guidance related to these practices; the  requirements traceability matrix and other requirements artifacts; baseline  program plans and reports used in monitoring the program’s progress;  system test plans and final test reports; and governance board charters,  governance board meeting minutes, and reviews to identify lessons  learned. For our assessment of each management discipline, we  assessed the extent to which USDA and FSA had implemented, partially  implemented, or not implemented key practices. For each objective, we  interviewed cognizant program and contractor officials.
In doing so, we reviewed documentation such as USDA  We conducted this performance audit from October 2014 to June 2015 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. Additional details  on our objectives, scope, and methodology are provided in appendix I.
subsections: section_title: Background
paragraphs: Among other missions, USDA manages benefit programs that support  farm and ranch production, natural resources and environmental  conservation, and rural development. FSA is one of three USDA service  center agencies that manage benefit programs for farmers and ranchers.  Currently, FSA manages 23 farm benefit programs identified by, among  other legislation, the Agricultural Act of 2014 (commonly referred to as the  2014 Farm Bill).assistance for livestock, honeybees, and farm-raised fish to providing  incentives for resource conservation. Appendix II provides a brief  description for each of the 23 farm benefit programs.
FSA benefit programs fall into four core categories:  farm loan programs, which are to provide direct loans or loan  guarantees to family farmers who could not otherwise obtain  agricultural credit;  income support and disaster assistance programs, which are to  provide farmers and ranchers with an economic safety net to help  them maintain their operations during difficult times; commodity operations programs, which are to expand market  opportunities for farmers; and conservation programs, which are to help maintain and enhance the  nation’s natural resources and environment.
subsections: section_title: Overview of the Services FSA Provides Its Customers
paragraphs: Over the last two decades, FSA has provided services to customers  supporting the farm benefit programs at its approximately 2,100 local  offices. To participate in FSA programs, customers may need to visit local  service center offices multiple times throughout the year because certain  transactions cannot be done electronically via e-mail or the Internet.
A new customer would typically go through several steps to enroll in a  benefit program:
At first, a customer needs to establish a relationship with the agency  by providing certain basic information about his operation that will be  used in determining eligibility.
Based on this information, an FSA agent is to create a master farm  record for the customer. The farm record is to include specific  information about the farm such as identification numbers for fields  and tracts (a tract is one or more contiguous fields), location  information, and a list of commodities that the farm is able to produce.
Once a customer has established a relationship, customers learn  about available FSA programs, receive information on eligibility and  estimated benefits under a particular program, and generate a draft  agreement for participation in a particular program.
The next step is for the customer to submit a final agreement to FSA  for their participation.
Subsequently, throughout the year, the customer documents information  about crops in an acreage report, which must be prepared for each  applicable tract and growing season. The customer brings acreage  reports and other forms to the FSA service center in person. Because  different crops have different reporting deadlines, a customer may need  to visit the service center multiple times to fill out reports for different  crops.
In addition to establishing relationships and administering benefits  programs, there are two other key activities that service centers perform:  handling acreage reports and printing maps.
Handling acreage reports. This is one of the most critical functions  for a service center. The FSA agent verifies that a customer is eligible  for a benefit and compares the acreage amount in the acreage report  against the acreage amount in the master farm record. The agent  then computes the payment amount and authorizes the payment to be  made to the customer.
Printing maps. Customers also often request maps of their tracts  from FSA service centers to help plan for the next growing season  because the maps can only be produced in hard copy. According to  FSA service center officials, this is one of the customer’s most  requested services.
In order to provide these services, FSA staff use a variety of computing  environments and software applications, including a central “web farm,” consisting of an array of interconnected  computer servers that exchange data in support of data storage and  web-based applications; a central IBM mainframe that hosts non-Web applications and data; a distributed network of IBM Application System 400 computers and a  common computing environment of personal and server computers at  each local office.
section_title: Goals and History of MIDAS
paragraphs: In early 2004, FSA began planning the MIDAS program to streamline and  automate farm program processes and to replace obsolete hardware and  software. FSA identified these goals for the program:
Replace aging hardware: Replace Application System 400  computers, which date to the 1980s and are obsolete and difficult to  maintain, with a hosting infrastructure to meet business needs,  internal controls, and security requirements.
Reengineer business processes: Streamline outmoded work  processes by employing common functions across farm programs.  For example, determining benefits eligibility could be redesigned  (using business process reengineering) as a structured series of work  steps that would remain consistent regardless of the benefits  requested.
Improve data management: Make data more readily available to  FSA personnel and farmers and ranchers—including online self- service capabilities—and increase data accuracy and security.
Improve interoperability with other USDA and FSA systems:  Integrate with other USDA and FSA modernization initiatives,  including the Financial Management Modernization Initiative for core  financial services that meet federal accounting and systems  standards, the Geospatial Information Systems to obtain farm imagery  and mapping information, and the Enterprise Data Warehouse to  provide enterprise reporting.
From 2004 through 2010, FSA went through several changes in direction  before selecting a technical solution for MIDAS:
FSA drafted initial requirements for MIDAS in January 2004.
FSA halted requirements development in early 2006 when program  officials decided that the proposed customized solution would not  meet future business needs.
FSA subsequently changed its approach in the Summer of 2006 from  acquiring customized software to acquiring commercial off-the-shelf  enterprise resource planning software. The program estimated that it  would cost $305 million to implement MIDAS, but this estimate had a  high degree of uncertainty.
In February 2008, FSA analyzed how its farm program functions  would map to functions available in a commercial off-the-shelf  enterprise resource planning software suite from vendor SAP, which  had been selected for two other USDA modernization initiatives—the  Financial Management Modernization Initiative and the Web Based  Supply Chain Management program. This analysis concluded that  MIDAS processes generally mapped to the SAP software.
Based on that analysis and a software alternatives analysis conducted  in mid-2008, FSA decided to proceed with SAP enterprise resource  planning software as the solution for MIDAS. FSA also decided to  accelerate the time frame for implementing the solution from the 10  years originally planned to 2 years. To accomplish this, FSA planned  to compress the requirements analysis phase from 4 years to 5  months, and reduce the analysis and design phase from 3.5 years to  9 months.
A request for quotations for the MIDAS system integrator contract was  released in July 2009, and a contract based on this request was awarded  to SRA International in December 2009. After a short delay due to a bid  protest, the system integrator began work in May 2010 with an initial firm  fixed price task order for $4.4 million through December 2010. By this  point, FSA had also awarded six other contracts for services to support  additional aspects of this initiative, including software licenses, project  management support, and technical support. As of October 2010, FSA  planned to spend $169 million—more than half of the program’s $305  million estimate—on the system integrator contract through fiscal year  2012.
Figure 1 depicts a timeline of key milestones for MIDAS from its inception  through the initiation of work by the system integrator.
In the years after the system integrator began to work on developing  MIDAS, the program ran into cost, schedule, and technical problems.  These issues ultimately resulted in a July 2014 decision to halt further  development on the MIDAS program. We discuss the events that led to  this decision later in this report.
section_title: History of MIDAS Program Management and Governance
paragraphs: In September 2007, FSA established a MIDAS executive program  manager and a program office to oversee the program and its supporting  contracts. According to FSA officials, the program office reported to a  Senior Management Oversight Committee on a regular basis. The  committee was chaired by the USDA Under Secretary for Farm and  Foreign Agricultural Services and included the USDA Chief Information  Officer, USDA Chief Financial Officer, and Administrator of the Farm  Service Agency as additional board members. The committee had the  following responsibilities, among others, in providing departmental  oversight and support for the MIDAS program:  communicating and providing strategic direction for FSA’s enterprise  modernization; approving the MIDAS acquisition strategy; approving the program’s cost, schedule, and requirements baseline; ensuring MIDAS integration with departmental requirements and  related initiatives and significant interdependencies; approving updates to business cases; and addressing issues escalated by a program-level review board.
At the department level, USDA had an IT governance process that was  overseen by the Executive Information Technology Investment Review  Board, which was chaired by the department’s Chief Operating Officer  and included the department’s Under Secretary for Farm and Foreign  Agricultural Services, Chief Information Officer, Chief Financial Officer,  and other senior executives. The board was to approve IT investments  that aligned with USDA’s mission and enterprise IT goals; provide  executive management oversight, approval, and commitment to selected  IT investments; and recommend to the Secretary a ranked group of IT  investments proposed for funding.
In addition to USDA governance, the Office of Management and Budget  (OMB) was involved in providing routine oversight for this program. OMB  requested monthly status briefings on MIDAS’s progress after USDA’s  Office of the Chief Information Officer conducted a TechStat review of the  program in November 2012. These monthly briefings continued until  October 2014, when the program was preparing to deploy its second and  final software release.
section_title: Prior GAO Reviews Found Weaknesses in MIDAS Management and Oversight
paragraphs: In May 2008, at the request of the House and Senate Committees on  Appropriations, we reported that MIDAS was in the planning phase and  that FSA had begun gathering information and analyzing products to  integrate its existing systems.adequately assessed the program’s cost estimate, in that the estimate  had been based on an unrelated USDA IT investment. Moreover, the  agency had not adequately assessed its schedule estimate because  business requirements had not been considered when FSA reduced the  implementation time frame from 10 years to 2 years.
We determined that the agency had not  As a result, we reported that it was uncertain whether the department  could deliver the program within the cost and schedule time frames it had  proposed and recommended that FSA establish effective and reliable  cost estimates using industry leading practices and establish a realistic  and reliable implementation schedule that was based on complete  business requirements. USDA generally agreed with our  recommendations and implemented our recommendation to improve its  tracking of user problems and clarifying roles and responsibilities between  FSA and USDA’s Information Technology Services. However, it did not  implement our other recommendations to establish reliable cost and  schedule estimates based on complete business requirements.
Subsequently, in July 2011, we reported that MIDAS was in the proof-of- concept and system design phase, and noted that the scope included  modernization of FSA’s systems for all of its (at that time) 37 farm  programs by March 2014. We determined that the program’s cost  estimate had a large degree of uncertainty. In particular, it did not yet  reflect decisions that had occurred since the estimate was developed in  2007 and that the completion date of its current development phase was  uncertain because of delays to key system design milestones.
In addition, we found that FSA had plans in place for MIDAS that  incorporated selected leading practices and had defined governance  bodies to provide oversight, but it had not implemented other key  management practices, including forming an integrated team with  representatives from IT programs that MIDAS depended on for its  success, developing a schedule that reflected dependencies with relevant  IT programs, and tracking the status of risks as planned. Moreover, it had  not clearly defined the roles and coordination among the program’s  governance bodies. We recommended that USDA update cost and  schedule estimates, address management weaknesses in plans and  program execution, and clarify the roles and coordination among the  governance bodies. The department agreed with our recommendations  and identified plans to address them. However, the agency did not  complete efforts to address these recommendations before the decision  was made to halt the program.
section_title: Poor Program Performance and Uncertainty Regarding Future Plans Led to the Decision to Halt Further MIDAS Development
paragraphs: In a July 2014 memo, the Secretary of Agriculture decided to halt the  MIDAS program after deploying minimal functionality due to performance  challenges in the early months after the system became operational, and  delays in determining the remaining scope, schedule, and cost for the  program. Our analysis similarly found that the key factors that led to this  decision were poor program performance—characterized by rising costs,  schedule overruns, reduced functionality, and problems with the system  after it was deployed—as well as uncertainty regarding future plans for  MIDAS.
Poor program performance: FSA experienced significant cost and  schedule delays in developing MIDAS, which led it to defer or remove  expected functionality and to eliminate key system tests prior to deploying  the system. Once the initial MIDAS functionality was deployed, FSA  employees encountered serious problems in using the system. A timeline  of key events and decisions that factored into MIDAS’s poor program  performance include:  In December 2011, MIDAS was envisioned to deliver significant  functionality in phases, with the majority of functions to be delivered in  the first phase. In further designing the system in March 2012, FSA  decided to remove selected functionality from the first phase, and to  deliver the remaining functionality in two deployments. As of June  2012, FSA estimated that development costs would be $330 million.
As FSA began to develop the system, however, it experienced cost  and schedule overruns. For example, by August 2012, FSA had  overrun its cost estimates by 11 percent and schedule estimates by  10 percent. These overruns were due, in part, to delays in completing  customization of the commercial software, redesign work on  interfaces, delays in testing individual system components as a result  of including more customization than planned, and delays in data  conversion and remediation efforts.
In order to help meet cost and schedule demands, in September  2012, FSA decided to split the two deployments into three  deployments and to focus primarily on the first deployment. After  continuing to experience schedule delays as it moved into system  testing, FSA decided to remove additional functionality from the first  deployment (including acreage reporting and customer records  functions).
To try to stay on schedule, the MIDAS program also obtained  approval from senior USDA and FSA management in early 2013 to  defer key testing activities—including performance testing and user  acceptance testing—until after the system became operational. These  tests were not performed after deployment.
When FSA implemented its first software release in April 2013,  MIDAS experienced significant technical problems, which is not  surprising given its lack of testing. For example, users experienced  significant problems with the system such as the geospatial  information system (GIS) functionality, accuracy of farm record data,  and system response time. Also, within 3 months after the  deployment, there were 62 critical, 172 major, 236 average, and 69  minor defects that needed to be addressed. As a result, the time  allotted to fix problems doubled in length—from 3 to 6 months—to  accommodate the fixes required by the system.
Uncertainty regarding future plans: FSA was unable to establish a  revised baseline for the program after experiencing cost and schedule  overruns in developing the initial system release because the proposals  were too costly or not aligned with the department’s budget and IT plans.  A timeline of key events include:
After conducting a TechStat review in November 2012, USDA’s Office  of the Chief Information Officer (CIO) directed FSA to establish a new  program baseline by January 2013. However, FSA did not deliver a  new baseline by the deadline. According to FSA officials, the agency  made three different attempts to salvage what it could from MIDAS:  In May 2013, the program submitted a proposed baseline to FSA  management that included delivering the full set of envisioned  MIDAS functionality by late 2016 at an increased cost. At $659  million, this new cost estimate was almost twice as expensive as  the earlier baseline estimate of $330 million. FSA management  did not approve the proposed baseline due to the cost.
In August 2013, FSA submitted a proposal to USDA for delivering  less than the full set of MIDAS functionality at a reduced cost to  implement and with a shorter development schedule. Specifically,  the program’s scope no longer included applications for the 2014  Farm Bill programs, which FSA decided to transfer to its web farm.  The cost estimate was reduced to $583 million through fiscal year  2015 and development was to be completed by mid-2015.  However, USDA’s Office of the CIO rejected this rebaseline  request because it required revisions in order to align with the  department’s budget plans.
In February 2014, FSA submitted a proposal for the same  functionality as the prior proposal at roughly the same cost and  with a shorter development schedule. Under this new proposal,  development was to be completed in mid-2015 at a cost of $584  million. This proposal also included a life cycle cost estimate for  MIDAS of $1.026 billion through fiscal year 2021. However, in  June 2014, the department’s Executive Information Technology  Investment Review Board placed the program’s third rebaseline  request on hold. The board wanted FSA to build the remaining  functionality in smaller increments and to work in partnership with  other agencies to develop an enterprisewide solution for acreage  reporting and customer portal tools consistent with the 2014 Farm  Bill and department priorities.
After the third rebaseline proposal was not approved, the  department’s review board recommended to the Secretary of  Agriculture that MIDAS halt development after the completion of the  customer records release. In July 2014, the Secretary decided to  approve the board’s recommendation.
FSA deployed its customer records release in December 2014. As of  March 2015, FSA had spent about $423 million on MIDAS, which was  $93 million higher than the 2012 baseline estimate of $330 million. Of the  $423 million, about half was spent on the system integrator contract.
Moving forward, FSA estimates that it will cost roughly $50 million to $60  million to continue to operate and maintain the system each year. As a  result, MIDAS could cost approximately $825 million through the end of  its useful life in 2021. Figure 2 illustrates the key events and decisions  affecting MIDAS that led to the decision to halt further development, and  table 1 identifies changes in MIDAS cost and schedule estimates over  time.
section_title: FSA Delivered a Fraction of the Functionality Envisioned for MIDAS
paragraphs: FSA has delivered a fraction of what was originally planned for MIDAS.  FSA first documented high-level plans for the functionality that MIDAS  was to deliver when it completed a system requirements review in  December 2011. At that time, MIDAS was envisioned to provide a single  SAP platform to host data, applications, and business processes for  administering farm program benefits, and advanced tools for customers  and FSA employees.seamlessly with other USDA systems, including USDA’s financial system,   In addition, it was expected to integrate  geospatial information system, and enterprise data warehouse. Figure 3  provides an overview of FSA’s planned key features for MIDAS.
However, as the program ran into problems, FSA continued to remove  planned features. Specifically, in June 2013, the MIDAS Senior  Management Oversight Committee decided to develop applications for  administering farm program benefits on FSA’s web farm rather than on  the SAP platform, as originally intended, because it could no longer wait  to transition from legacy systems. In addition, the ability for FSA  employees to use critical acreage reporting tools with the data—a  function that affects 85 percent of tasks—was removed from the  program’s scope. Other key features were also removed from MIDAS,  such as an online portal for farmers and other customers as well as  integration with USDA’s financial system and enterprise data warehouse.
As a result of removing key features from MIDAS, FSA delivered a  fraction of the originally envisioned functionality. MIDAS currently  provides farm and customer record data on a SAP platform that is  integrated with USDA’s geospatial information system. As a result of this  partial implementation, FSA employees currently access, visualize, and  edit data in MIDAS. They then turn to the web farm to run acreage  reporting, administer benefits, and process payments.
FSA did not quantify what percentage of the originally envisioned MIDAS  functions were delivered, and this task is complicated by the fact that  there is not a complete set of requirements. However, if one were to  weigh the key features equally, MIDAS has delivered about 20 percent of  what FSA planned. That figure would be lower if one were to include the  comparative importance of the functionality. For example, FSA has cited  acreage reporting as a key feature affecting 85 percent of what FSA  employees do and this is not included in MIDAS. In addition, integration  with the financial system was one of the key reasons for going with the  SAP solution, and this, too, was not delivered. Figure 4 compares the  functionality planned for MIDAS in 2011 to what has been delivered.
section_title: FSA Did Not Adequately Implement Key Program Management Disciplines on MIDAS and Lacks the Capacity to Effectively Manage Successor Programs
paragraphs: FSA did not adequately implement program management disciplines on  MIDAS in four key areas—requirements development and management,  project planning and monitoring, system testing, and executive-level  governance—and lacks the demonstrated capacity to manage successor  programs. Leading government and industry organizations call for best  practices such as obtaining commitment to a requirements baseline and  ensuring requirements are prioritized and traceable; managing changes  to project plans and conducting progress monitoring; testing the system to  determine whether it is acceptable to users; and implementing executive- level governance to include comparing performance against expectations  and assessing maturity at key checkpoints based on predefined criteria.
USDA and FSA policies are consistent with these best practices.
However, in developing MIDAS, FSA did not adequately develop and  manage requirements, effectively manage project plan changes, conduct  meaningful progress monitoring, execute critical tests before the system  became operational, and implement effective executive-level governance  to prevent MIDAS from falling short of expectations. FSA and contractor  officials explained that key practices were not always implemented  because, among other things, the program’s scope was not well- understood, USDA and FSA did not follow its own policies, and  management allowed the program to continue despite known  weaknesses. Moreover, while FSA officials have acknowledged  weaknesses in each of these management disciplines, the agency has  not established plans to improve its management of successor programs.
Until FSA addresses shortfalls in key program management disciplines on  successor programs to MIDAS, the agency will be at an increased risk of  producing additional projects with cost overruns and schedule slippages  while contributing little to mission-related outcomes. Further, until FSA  establishes improvement plans, it will be difficult for the agency to  demonstrate that it has the capacity to effectively manage IT acquisitions  and it will be at a higher risk of failure for any new or ongoing IT  initiatives.
subsections: section_title: FSA Did Not Adequately Develop and Manage MIDAS Requirements
paragraphs: Requirements establish what the system is to do, how well it is to do it,  and how it is to interact with other systems. Leading industry  organizations such as the Software Engineering Institute have  recommended practices for the effective development and management  of requirements such as eliciting stakeholder needs, ensuring that  requirements are complete and unambiguous, prioritizing them, obtaining  formal commitment to them, assessing any gaps with the proposed  solution, and ensuring that each requirement traces back to the business  need and forward to its design and testing. FSA has established policies  and guidance for developing and managing requirements that are  consistent with these recognized practices.
Of six key practices in requirements development and management, FSA  implemented one practice, partially implemented two practices, and did  not implement three practices. Specifically, FSA documented  requirements for MIDAS based on needs gathered from stakeholders  prior to a system requirements review in December 2011 and throughout  the development of the system. The agency also identified its process for  addressing software gaps with the SAP solution and documented  workarounds for certain capabilities. However, FSA did not adequately  develop and manage MIDAS requirements because the agency did not  always develop complete requirements, prioritize its requirements, obtain  commitment on a requirements baseline, document solutions to gaps with  SAP software that had been known for years and were required for the  program’s success, and ensure that requirements were traceable to  development products.
Table 2 identifies the extent to which FSA implemented key practices for  developing and managing requirements for MIDAS.
FSA officials and supporting documentation show several reasons for the  lack of requirements development and management discipline on MIDAS.  For example:
FSA officials noted that problems with the completeness and  specificity of requirements persisted because guidance on how to  ensure requirements completeness and specificity was not  implemented until shortly before the system requirements review and  it took time for changes to be made to the requirements. Also, officials  cited challenges in the complexity of writing requirements for business  processes related to GIS capabilities.
For selected key milestone reviews and decision points, the program’s  executive governance board did not verify that key requirements  artifacts and processes were mature enough to proceed because  USDA and FSA did not establish a governance process that required  the board to perform such reviews.
In August 2012, the IV&V contractor reported that the program’s gap  analysis lacked specific details to fully understand activities for  identifying, reviewing, assessing, and validating gaps. In addition, GIS  was not part of the system integrator’s initial scope and additional  resources and expertise had to be acquired during MIDAS’s  development.
FSA and contractor staff had ongoing trouble getting access to one of  the program’s two requirements management tools because system  access rights were controlled by a different USDA agency. While FSA  worked on this issue, it continued to be a problem throughout the  development of release 1. The IV&V contractor reported in March  2012 that this lack of access limited their ability to perform  requirements traceability. This made it more difficult to manage the  baseline scope and configuration of the release. In addition, the IV&V  contractor reported in February 2014 that the program office did not  follow adequate document configuration management processes to  control changes, thereby making it difficult to maintain traceability  between requirements and design documents.
FSA officials noted that the program did not obtain a requirements  baseline approval or prioritize its requirements because the program  lacked the necessary discipline and rigor for requirements  management activities during the first software release.
FSA’s lack of requirements development and management discipline on  MIDAS impacted the program in several ways. For example, by not  establishing a requirements baseline the agency did not have a firm  commitment on the mission-related outcomes MIDAS would satisfy. In  addition, not having prioritized requirements limited the agency’s ability to  make decisions on which scope to defer or remove from the program  when faced with cost and schedule overruns.
Both FSA and IV&V contractor officials noted that the program had  demonstrated improvements in practices associated with requirements  management for the second software release in December 2014. For  example, an official for the IV&V contractor stated that requirements  traceability was significantly better and that a requirements baseline was  established for the second release. Unless FSA ensures that successor  programs to MIDAS are fully implementing key requirements  development and management practices, the agency will not have  reasonable assurance that its IT modernization efforts will meet  stakeholder needs and contribute to mission-related outcomes.
section_title: FSA Established Project Plans for MIDAS, but Did Not Effectively Manage Plan Changes or Monitor Progress
paragraphs: Leading organizations such as the Project Management Institute and  Software Engineering Institute have recommended best practices for  project planning and monitoring. FSA also has policies and guidance  that are consistent with recognized practices. Project planning maintains  plans as the basis for managing the project’s activities. Recommended  best practices call for documenting and evaluating changes to established  project plans to determine whether they require updates to initial planning  estimates for cost, schedule, and scope. Project monitoring provides an  understanding of the project’s progress by comparing actual work  completed to a plan consisting of predefined expectations for cost,  schedule, and deliverables. Best practices state that monitoring progress  is important because it helps project managers take timely corrective  actions when performance deviates significantly from plans.
Of three key practices in project planning and monitoring, FSA partially  implemented one practice and did not implement two practices. The  agency established a project plan for MIDAS with predefined expectations  for cost, schedule, and scope based on its integrated baseline review in  March 2012. For several months following this review, the program  executed to these plans and tracked certain technical and programmatic  changes in its change control log.
However, the agency did not effectively manage plan changes or monitor  progress. For example, FSA did not update its baseline plans when it  revised the solution architecture and when it deferred planned testing  activities before the initial software release. In addition, FSA continued to  develop deferred functionality for approximately 20 months without an  approved rebaseline for these efforts. Also, FSA’s initial monitoring of  contractor performance lacked insight into the progress of deliverables  and contractor performance reporting was halted from December 2012  through October 2014. Table 3 identifies the extent to which FSA  implemented key practices for project planning and monitoring for MIDAS.
USDA and FSA officials provided several explanations for the agency’s  shortfalls in project planning and monitoring. FSA officials acknowledged  that they did not update baseline project plans to reflect changes in the  solution architecture and testing phases prior to the initial software  release, but noted that they briefed the Senior Management Oversight  Committee on these changes. The USDA CIO noted that MIDAS halted  progress monitoring of contractors in December 2012 because managers  were already aware that the program was performing poorly and was in  need of a rebaseline. Also, the Director of the MIDAS business  management office stated that while the program’s cost baseline was not  at a detailed level, the contractors’ cost estimates included additional  details on work products and deliverables. However, the program was not  monitoring progress based on those details.
By not revising the project plan after making significant revisions to its  approach, the program’s cost, schedule, and scope were no longer  effective benchmarks for measuring performance. Without meaningful  progress monitoring initially and as the program shifted its focus, program  managers and executive stakeholders had less insight into the  deliverables being produced by contractors and less control over the  program’s outcomes.
According to FSA and IV&V contractor officials, the program provided a  baselined cost, schedule, and scope for its second software release and  executed improved discipline in managing plan changes. However, until  FSA ensures that successor programs to MIDAS are fully implementing  key project planning and monitoring practices, the agency will be at an  increased risk that future projects will experience cost and schedule  overruns and achieve less than expected outcomes.
section_title: FSA Defined Test Plans for MIDAS, but Did Not Execute Critical Tests before the System Became Operational
paragraphs: According to relevant leading industry practices and government  guidance, system testing should be progressive, meaning that it should  consist of well-defined test plans and a series of test events that build on  and complement previous events in the series. Testing should first focus  on the performance of individual system components, then on the  performance of integrated system components, followed by system-level  tests that focus on whether the system (or major system increments) is  acceptable, interoperable with related systems, and operationally suitable  to users. FSA established policies and guidance for system testing on  MIDAS that are consistent with recognized practices.
Of four key practices in system testing, FSA implemented one practice,  partially implemented two, and did not implement one. FSA defined test  plans for MIDAS, but the agency did not execute critical performance and  user testing before the system became operational. FSA had test plans in  place that generally defined key elements, such as the roles and  responsibilities of groups that were to conduct testing, hardware and  software to support testing, and a schedule that defined how long and in  what order test events were to occur. In addition, FSA conducted testing  on individual and integrated components. However, MIDAS’s test plans  were missing a key element—traceability between system test events and  requirements. Also, integration testing took longer than planned and the  program decided to defer testing that was to validate whether system  performance met requirements and was acceptable to users until after the  system went live. Table 4 identifies the extent to which FSA implemented  key system testing practices for MIDAS.
FSA’s shortfalls in system testing were due in part to technical problems  and delays in developing GIS capabilities and a desire by department and  agency management to keep the target deadline for the initial release of  MIDAS. Early in the development of the GIS capabilities, FSA ran into  technical problems that required additional time and resources to  address. Since GIS development had been delayed, integration testing  with the GIS capabilities also had to be delayed. By April 2013,  integration testing was still ongoing and FSA had to make a decision  whether to delay the implementation of MIDAS or allow the system to go  live while accepting the risk of not conducting performance/stress,  regression, and user acceptance testing. While the program warned of  the risks of deploying MIDAS with outstanding defects and incomplete  testing, senior department and agency officials decided to accept these  risks in order to deploy the system by April 2013.
Incomplete testing on MIDAS did not provide users an opportunity to  identify key problems with the system and whether it met their needs  before it went live. After the system went live, users experienced  significant problems such as GIS functionality, accuracy of farm record  data, and system response time. Within 3 months after the initial MIDAS  release, there were 62 critical, 172 major, 236 average, and 69 minor  defects that needed to be addressed. The program had a plan in place to  address performance problems after the system went live, but, due to the  number of problems, it had to extend the contract for addressing system  defects from 3 months to 6 months.
FSA officials and the program’s IV&V contractor have acknowledged the  shortfalls in system testing practices and stated that the program has  taken steps to improve system testing on the second and final MIDAS  release. For example, FSA and IV&V contractor officials noted that the  program conducted user acceptance testing prior to deploying  functionality for managing customer records (also called business partner  functionality). While the agency recognized the need to improve its testing  on the second MIDAS release, it has not demonstrated that it has  institutionalized sound system testing practices. Until it does so, the  agency will be at higher risk of delivering systems that have performance  issues and do not fully meet users’ expectations.
section_title: Executive-level Governance Was Ineffective in Preventing MIDAS from Falling Short of Expectations
paragraphs: We assessed best practices used in industry, academia, and government  to develop the IT Investment Management Framework to provide a  method for evaluating and assessing how well an agency is selecting and  managing its IT resources. Efforts to build a foundation for IT  governance involve establishing specific critical processes, such as  instituting investment boards, selecting investments, controlling  investments as they are developed and deployed, and reviewing  investments after they are deployed.
Instituting investment boards. Successful organizations establish  an IT investment board comprised of senior executives who are  responsible for operating according to documented guidance, policies,  and procedures that align with existing IT governance processes,  identify decision gates to be reviewed and approved by the board, and  establish entry/exit criteria to be reviewed at each decision gate. The  board is also responsible for ensuring that investment decisions  address stakeholder needs and are made in the best interest of the  organization.
Selecting IT investments. Successful organizations identify, use,  and store comprehensive data—including a business case that  defines the life cycle cost estimate and benefits to be realized—in  order to support investment decision making. Reselecting ongoing  projects is an important part of this critical process; if a project is not  meeting established goals and objectives, the organization must make  a decision on whether or not to continue to fund it.
Controlling IT investments. Organizations should have a  documented, well-defined process for overseeing ongoing  investments once they have been selected. Effective investment  oversight and evaluation involves, among other things, (1) comparing  actual performance against cost and schedule estimates; and (2)  assessing whether projects are meeting expectations against  developmental milestones using predefined criteria and decision  gates, and taking corrective actions when expectations are not being  met.
Reviewing IT investments after deployment. Once the project has  transitioned from the development phase to the operations and  maintenance phase, organizations should conduct a post- implementation review to compare actual investment results with  decision makers’ expectations for cost, schedule, performance, and  mission improvement outcomes. The lessons learned from these  reviews can be used to modify future investment management  decision making. In 2013, USDA issued updated policies and  guidance that are generally consistent with these practices.
Of five key practices in executive-level IT governance, FSA partially  implemented two practices and did not implement three practices.  Specifically, FSA partially implemented steps to institute a governance  board. It established a governance structure and process for MIDAS;  however, its governance process was ineffective in preventing MIDAS  from falling short of expectations. Specifically, FSA did not implement key  steps for selecting and controlling investments, including establishing a  comprehensive business case or life cycle cost estimate and comparing  actual performance against estimates. Also, FSA partially implemented  post-implementation review practices. The agency tasked a contractor  with assessing the results and lessons learned from portions of MIDAS  that were implemented; however, it did not conduct a comprehensive  review of the lessons learned on the program as a whole.
Table 5 identifies the extent to which USDA and FSA implemented key  executive governance practices for MIDAS.
The governance of MIDAS was ineffective, in part, because USDA’s  Office of the CIO did not ensure that MIDAS followed its policies and  guidance for IT governance. In 2011, we reported that governance boards  had not been reviewing MIDAS at key decision points using criteria  defined in department guidance and recommended that the department  and agency collaborate to document how the department is meeting its  policy for IT investment management for MIDAS, to include investment  reviews.not address it while MIDAS was in development. According to FSA  officials, USDA did not have guidance for IT governance providing  defined decision gates with standard criteria and documentation   While the department agreed with our recommendation, it did  requirements. FSA officials noted that from 2011 until 2013, the agency  used a governance process involving a program-specific gate review plan  for MIDAS based on SAP’s system development methodology.
USDA’s Under Secretary for Farm and Foreign Agriculture Services and  the Chief Financial Officer stated that there was a breakdown in the  governance process for MIDAS, particularly on its initial development.  The Under Secretary noted that the Senior Management Oversight  Committee made the best decisions it could based on the information it  had, but the information that FSA had reported to the committee did not  adequately portray the extent of the cost, schedule, and technical  problems or decisions that had been made on scope changes. For  example, the Under Secretary and the Chief Financial Officer stated that  they were not informed that FSA had been developing key functions in  both MIDAS and on the agency’s web farm—a key change in the original  scope—until early 2013. In addition, the Under Secretary and the Chief  Financial Officer noted that they were not informed until early 2013 that  FSA had made decisions to remove or defer additional scope—including  acreage reporting—from the first software release.
Subsequently, in 2013, MIDAS began piloting USDA’s new IT governance  process, called the Integrated IT Governance Framework. This framework  required MIDAS to report its performance to and obtain approval from a  department-level investment review board. In following this new  governance framework, the investment review board approved the final  decision to implement the second MIDAS software release and  recommended to the Secretary of Agriculture to halt further development  on MIDAS.
While the recently updated governance framework established by USDA  has potential for improving FSA’s IT modernization efforts, unless USDA  and FSA take additional steps or develop a mechanism to help ensure  that successor programs to MIDAS programs are fully implementing key  executive IT governance practices—including practices for selecting,  controlling, and reviewing investments—department and agency  management will not have reasonable assurance that oversight is  effective in preventing future IT investments from falling short of  expectations.
section_title: FSA Is Planning Next Steps after MIDAS as Required, but Has Not Demonstrated the Capacity to Manage Future IT Initiatives
paragraphs: Required by law to automate, integrate, and modernize its farm program  services, FSA has begun planning how it will do so. In an explanatory  statement accompanying the 2015 appropriations act, Congress directed  USDA to, among other things, deliver a modernized functional system  that builds existing farm program applications into an integrated system,  delivers increased efficiency and security, retires redundant legacy  systems, eliminates the path of siloed legacy applications, capitalizes on  the investment that USDA has already made in the enterprise platform,  addresses the new requirements of the 2014 Farm Bill, and improves on  the capabilities originally proposed to Congress and the nation’s farmers  and ranchers. The appropriations act also mandated that FSA develop a  plan for IT related to MIDAS and other farm program delivery systems  prior to obligating more than 50 percent of the $132 million made  available in fiscal year 2015. This plan is to identify each investment’s  capabilities and mission benefits, estimated life cycle cost, key  milestones, and alignment with FSA’s IT Roadmap.
In addition, the 2014 Farm Bill includes provisions for streamlining  acreage reporting to reduce the administrative burden on farmers and  producers. This is to be done by, among other things, requiring the  Secretary of Agriculture to ensure that producers may report information  electronically (including geospatial data) and that improvements are made  in the areas of coordination, information sharing, and administrative work  with FSA, the Risk Management Agency, and the Natural Resources  Conservation Service.
FSA has begun planning how it will move forward in its modernization  efforts to fulfill the functionality that was envisioned—but not delivered— by MIDAS. According to FSA officials, the agency plans to document its  decisions for addressing acreage reporting tools, online customer tools,  and other functionality that was removed from MIDAS in its IT Roadmap  by the end of Spring 2015. Those plans may include decisions to partner  with other USDA agencies, acquire new commercial off-the-shelf  software, and/or develop and enhance functionality on the agency’s web  farm.
In its fiscal year 2016 budget request, FSA noted that, while the mix of  investments may fluctuate based on its prioritization process and  business requirements, the agency intends to pursue incremental,  modular investments such as the following.
Customer self-service tools: Expanding on existing online services  and partnering with other USDA agencies (including the Acreage Crop  Reporting Streamlining Initiative) to provide farmers and ranchers  online access to relevant information, including remote and/or mobile  access to their data and programs.
Expanded customer service: Piloting a program to find new ways to  deliver programs and service support through the agency’s repository  of geospatial and farm information.
Increased IT investments to support FSA process improvements:  Delivering incremental improvements to address pain points and  inefficiencies identified by field office staff as impacting their  effectiveness in servicing customers. Improvements in the pipeline  could range from simple items such as simplifying the printing of farm  maps or customized reports to continuing the incremental integration  of stove-piped systems through establishing or enhancing common  eligibility, payment, and obligation frameworks.
In addition, FSA officials stated that the agency is incorporating lessons  learned into future plans, including building smaller, incremental releases with a defined scope, cost, and  schedule and defined benefits for the customer; extending an organizational change agency network to provide input  on pain points and process improvements; driving the prioritization of investments through business needs  instead of technology; and  integrating technology capabilities, including SAP, into decision- making processes and alternatives analyses, so the technologies for  each project will be determined based on what best matches the  business requirements.
However, FSA has not established plans to improve its ability to  successfully manage major IT investments. Specifically, FSA officials  have not committed to improving agency practices in the four areas we  reviewed because they believe that they have already addressed the  problems. While agency officials acknowledge that mistakes were made  on the first MIDAS release, they stated that they did a better job delivering  the second release. For example, the MIDAS Program Executive reported  that the agency established requirements for the second release,  established a schedule for developing and deploying the release,  performed adequate testing prior to deploying the release, and that  oversight bodies were kept informed. While the second release was more  successful than the first, it was much less complex. The second release  involved a limited amount of functionality that had been in development  for several years before it was deferred from the first release. Further, the  relatively discrete amount of work involved and the establishment of  baseline plans 3 months prior to the release allowed the project to deliver  near cost and schedule estimates.
These efforts, however, are not sufficient to demonstrate that FSA will  adhere to departmental policy or that it has practices in place to  successfully plan, develop, and oversee future complex IT investments.  USDA’s Under Secretary for Farm and Foreign Agriculture Services and  Chief Financial Officer agreed that plans are needed to improve FSA’s  ability to successfully manage IT investments. Until FSA establishes and  implements improvement plans, it will be difficult to demonstrate that it  has the capacity to manage IT acquisitions and the agency will have a  higher risk of failure in future IT initiatives.
section_title: Conclusions
paragraphs: After spending about $423 million through March 2015, the MIDAS  program was halted about 10 years after it was initiated. Key factors that  led to the decision to halt the program included cost overruns totaling $93  million more than planned, schedule delays, performance issues, and  management’s inability to decide on how to restructure the program for  success.
In deploying the two MIDAS releases, FSA delivered about one-fifth of the  functionality it had planned to deliver. MIDAS was envisioned to provide a  seamless, integrated system that would allow farmers and ranchers to  submit information electronically and allow FSA employees to process  farm program benefits with built-in tools and access to GIS and other  enterprise systems. However, due to the limited functionality that MIDAS  provided, farmers and ranchers continue to submit information to FSA  service centers in person while employees continue to use separate  systems for processing acreage reports, farm program applications, and  payments.
Even though USDA and FSA have system acquisition policies that are  consistent with best practices in the areas of requirements development  and management, project planning and monitoring, system testing, and  executive-level governance, FSA did not implement the majority of these  policies and practices in developing MIDAS and has not established plans  to improve its approach. Until FSA establishes and implements a plan to  adhere to agency policies and best practices, it will be difficult to  demonstrate that it has the capacity to effectively manage IT acquisitions.  Further, until the agency adheres to system acquisition policies and  sound IT practices, it will have a higher risk of failure in future IT  initiatives.
section_title: Recommendations for Executive Action
paragraphs: In order to institutionalize sound IT management practices and build  FSA’s IT management capacity while improving service to the Nation’s  farmers and ranchers, we are making five recommendations to the  Secretary of Agriculture to:
Direct the FSA Administrator to establish and implement an  improvement plan to guide the agency in adopting recognized best  practices and following agency policy.
Direct the FSA Administrator to adhere to recognized best practices  and agency policy in developing and managing system requirements  before proceeding with any further system development to deliver  previously envisioned MIDAS functionality. Specifically, the  Administrator should ensure that requirements are complete,  unambiguous, and prioritized; commitment to requirements is  obtained through a formal requirements baseline; differences (or  gaps) between the requirements and capabilities of the intended  solution (including commercial off-the-shelf solutions) are analyzed;  strategies to address any gaps are developed; and requirements are  traced forward and backward among development products.
Direct the FSA Administrator to adhere to recognized best practices  and agency policy in planning and monitoring projects. Specifically,  the Administrator should ensure that project plans include predefined  expectations for cost, schedule, and deliverables before proceeding  with any further system development; updates to the project plan are  made through change control processes; and progress against the  project plan, including work performed by contractors, is monitored.
Direct the FSA Administrator to adhere to recognized best practices  and agency policy in system testing. Specifically, the Administrator  should establish well-defined test plans before proceeding with any  further system development, and ensure that testing of (a) individual  system components, (b) the integration of system components, and  (c) the end-to-end system are conducted.
Direct the FSA Administrator to adhere to recognized best practices  and agency policy in executive-level IT governance before proceeding  with any further system development. Specifically, an executive-level  governance board should  review and approve a comprehensive business case that includes  a life cycle cost estimate, a cost-benefit analysis, and an analysis  of alternatives for proposed solutions that are to provide former  MIDAS requirements prior to their implementation; ensure that any programs that are to accommodate former MIDAS  requirements are fully implementing the IT program management  disciplines and practices identified in this report;  conduct a post-implementation review and document lessons  learned for the MIDAS investment; and  reassess the viability of the MIDAS technical solution before  investing in further modernization technologies.
section_title: Agency Comments and Our Evaluation
paragraphs: We sought comments on a draft of this report from USDA. We  subsequently received written comments from the FSA Administrator.  While the agency did not explicitly agree or disagree with the  recommendations, it cited steps it had taken and plans to take to  implement best practices in the areas of requirements management,  project planning and monitoring, system testing, and executive IT  governance. However, the agency did not cite steps it would take to  establish and implement an improvement plan to guide the agency in  adopting recognized best practices and following agency policy. Because  the agency is moving to implement best practices, we continue to believe  that a plan—with steps, milestones, and performance measures—is  warranted. Without such a plan, it will be difficult for the agency to  demonstrate its progress and ensure that it has the capacity to manage IT  acquisitions.
In its overall comments, FSA noted the following:
FSA stated that it has taken active steps to address the issues raised  in the draft report by selecting a new CIO and initiating steps to  acquire a third party assessor to holistically evaluate the technology  solution for MIDAS and to make recommendations to inform a  coherent IT strategy. We agree that selecting a CIO and obtaining  recommendations on how to improve FSA’s IT strategy are sound  steps. However, these steps are not enough to address the issues  raised in this report. FSA must take additional steps to establish an  improvement plan, and to implement practices and follow agency  policy.
FSA stated that the recommendation by the USDA Executive IT  Governance Board and the July 2014 decision by the Secretary of  Agriculture to halt MIDAS development beyond release 2 (which FSA  established to deliver the residual portion of the customer records  functionality) allowed the agency to (1) focus attention and resources  on applying lessons learned from release 1 and (2) apply program  management best practices across key disciplines such as planning,  requirements management, cost and schedule management, and  system testing. We agree that the decision to halt MIDAS was a  sound one, and that it allowed the agency to focus on the residual  deliverables provided by release 2. However, we identified several  management shortfalls that continued to persist after release 1 was  deployed in April 2013. For example, FSA did not update its baseline  cost, schedule, and scope plans from March 2012 to October 2014,  even though it made significant decisions affecting scope and  schedule. Also, from November 2012 through October 2014, the  program did not have a project plan for monitoring progress due to  numerous scope and schedule changes and relied on status reporting  from draft schedules. In addition, while the USDA CIO gave the  investment a “red” (high-risk) rating on the Federal IT Dashboard in  December 2012, the MIDAS Senior Management Oversight  Committee allowed MIDAS to continue until July 2014 without any  improvement to the CIO’s rating. These findings are discussed in this  report.
FSA stated that the organizational alignment around comprehensive  improvement and quality of the MIDAS program (associated with  release 2) is a clear demonstration of the agency’s capability to  properly manage and deliver IT systems. However, until FSA  establishes and implements a plan to adhere to agency policies and  best practices, we believe the agency has not yet demonstrated that it  has the capacity to effectively manage IT acquisitions.
FSA noted that the MIDAS program demonstrated an improvement in  testing practices on release 2. Specifically, FSA stated that the period  of time for transitioning from deployment to steady state concluded  with zero critical defects and five major defects. We agree and  acknowledged the agency’s improvements on system testing  associated with the development of customer records in the report.  Specifically, we noted that the program conducted user acceptance  testing prior to deploying functionality for managing customer records.  However, it has been our experience that it takes time to change an  organization’s culture to adopt best practices. The agency will need to  build upon this experience to ensure it consistently implements sound  practices and follows agency policies in all future IT initiatives. We  continue to believe that establishing and implementing an  improvement plan, as we recommended, will aid the agency in doing  so.
FSA stated that, since the first deployment of MIDAS functionality in  April 2013, the agency implemented top-down organizational  transformation to bolster FSA's ability to consistently deliver IT  investments that provide their intended business value, within the  targeted schedule and budget. The agency also stated that the  MIDAS initiative identified a number of best practices that are being  emulated to improve IT management agencywide. However, FSA did  not provide supporting evidence for these efforts and our previously  stated findings show that FSA did not sufficiently monitor project  progress well beyond the first MIDAS release.
FSA stated that while our report acknowledges some of its  improvements, our assessment of the extent to which USDA and FSA  had implemented each management discipline reflects findings based  on MIDAS release 1 activities, and therefore is not truly representative  of FSA’s capacity to more broadly manage IT initiatives.
We believe our report accurately evaluates the implementation of key  program management disciplines on the MIDAS acquisition. Our  review assessed processes and practices over roughly 3 years (from  December 2011 to October 2014), which included a significant  amount of work on the customer records functionality. Specifically, the  customer records functionality represented 1 of the 24 unique features  FSA had originally planned for MIDAS as of December 2011. FSA  had begun working on customer records in December 2011, and  delivered about 30 percent of the customer records functionality with  the initial MIDAS software release in April 2013. When faced with the  firm commitment to deploy release 1 in April 2013, the agency  decided to defer the remaining customer records functionality to  release 2. Further demonstrating the limited scope of release 2, FSA  established baseline project plans for release 2 in October 2014, just  3 months before deploying it in December 2014. While our report  acknowledges that FSA improved selected practices in developing  and deploying release 2, we do not believe that the scope or  timeframe associated with this initiative provides sufficient evidence  that FSA has established the capacity to manage large, complex  acquisitions.
In addition, FSA provided the following comments regarding our  recommendations:
With respect to our recommendation to establish and implement an  improvement plan to guide the agency in adopting recognized best  practices and following agency policy, FSA stated that the agency has  undergone leadership transformation efforts over the last 12 months,  including appointing a new Administrator, CIO, MIDAS Program  Executive, and MIDAS Program Director. FSA noted that it gave  additional reporting authority to the MIDAS Program Executive and  moved the FSA CIO position from Kanas City, Missouri to  Washington, D.C. to improve communication with the Administrator on  agencywide initiatives. FSA stated that over the past year, FSA  leadership placed additional emphasis, funding, and staff resources  on ensuring that IT investments, decisions, dependencies, and  operational plans are driven by business needs across the agency.  The agency also stated that with its Business Strategy and IT  Strategy, it is maturing IT planning and management capabilities  needed for integrated IT solutions for Farm Programs and all of FSA’s  lines of business. Finally, FSA noted that it is using a Strategic IT  Roadmap to ensure IT programs are supporting the Business  Strategy. We agree that FSA has taken steps over the past year to  improve its IT management capabilities as we discuss in the report.  However, these actions do not establish and implement an  improvement plan to guide the agency in adopting recognized best  practices and following agency policy. Until FSA does so, it will be  difficult to demonstrate that it has the capacity to manage IT  acquisitions. Thus, as previously discussed, we believe the agency  should continue to establish and implement such an improvement  plan.
Regarding our recommendation to adhere to recognized best  practices and agency policy in developing and managing system  requirements before proceeding with any further system development  to deliver previously envisioned MIDAS functionality, FSA stated that  the MIDAS program implemented all of the key practices for release  2. As previously stated, our report acknowledges that FSA improved  selected practices in developing and deploying release 2. However,  we do not believe that the scope or timeframe associated with this  initiative provide sufficient evidence that FSA has improved its  capacity to manage large, complex acquisitions. Further, we identified  selected shortfalls in requirements management for release 2,  including weaknesses in requirements traceability and prioritization.  Moving forward, FSA stated that it will improve the rigor and  adherence to key requirements management processes for all IT  projects. We will continue to monitor the agency’s efforts to implement  our recommendation.
Regarding adhering to recognized best practices and agency policy in  planning and monitoring projects, FSA stated that the MIDAS program  implemented all of the key practices for release 2. As previously  stated, our report acknowledges that FSA improved selected practices  in developing and deploying release 2. However, we do not believe  that the scope or timeframe associated with this initiative provide  sufficient evidence that FSA has improved its capacity to manage  large, complex acquisitions. Further, our report identified shortfalls in  program monitoring in the run up to deploying release 2, including  weaknesses in updating project baselines to reflect program changes  and in monitoring progress against a defined project plan. Moving  forward, FSA stated that it would continue to mature and strengthen  its project planning and monitoring practices through a partnership  with a third-party capital planning center of excellence and through  corrective action plans to address identified weaknesses. It also  stated that it is implementing earned value management practices on  MIDAS going forward. We will continue to monitor the agency’s efforts  to implement our recommendation.
With respect to our recommendation to adhere to recognized best  practices and agency policy in system testing, FSA stated that it  established renewed commitment to MIDAS testing efforts and  implemented all of the key practices for release 2. As previously  stated, our report acknowledges that FSA improved selected practices  in developing and deploying release 2. However, we do not believe  that the scope or timeframe associated with this initiative provide  sufficient evidence that FSA has improved its capacity to manage  large, complex acquisitions. Moving forward, FSA noted that it plans  to adhere to recognized best practices and agency policy in pursuing  consistent or increased rigor around system testing to demonstrate  the agency’s testing capabilities are consistent and repeatable across  all IT projects. We will continue to monitor the agency’s efforts to  implement our recommendation.
Regarding our recommendation to adhere to best practices and  agency policy in executive-level IT governance before proceeding with  any further system development, FSA stated that it is evaluating its  governance structure to potentially include establishing work groups  that would evaluate IT initiatives at a more granular level of detail.  FSA also stated that it is working with USDA’s Office of the CIO to  determine how MIDAS will align with the department’s governance  framework and to identify the appropriate gate reviews, artifacts, and  level of oversight. We will continue to monitor the agency’s efforts to  implement our recommendation.
Overall, FSA’s poor performance and lack of results for more than 2 years  contributed to its inability to deliver most of the intended functionality and  led the Secretary of Agriculture to direct the agency to halt further  development after release 2. The efforts that continued after USDA  decided to halt further development on MIDAS in July 2014 and through  the delivery of release 2 in December 2014 were to salvage a feature  (customer records) that was almost fully developed by the time the  department made this decision. Our assessments of project planning and  monitoring and executive IT governance practices already include FSA’s  efforts to manage the overall program and to continue developing  customer records through October 2014. Nonetheless, if we were to  consider FSA’s efforts on release 2 beginning in October 2014, we would  have altered just 1 of the 18 key practices (conducting user testing) due  to weaknesses that persisted beyond the deployment of release 1 in April  2013.
To its credit, FSA has (1) acknowledged that management improvements  are needed and identified steps the agency plans to take; (2) made  changes in key leadership positions; and (3) committed to delivering  smaller, iterative IT projects going forward. However, our experience in  reviewing federal IT acquisitions has shown that it takes time to build  repeatable, robust processes. Implementing improvements during the last  few months of a 3-year effort is not enough to demonstrate repeatable IT  management capacity. As we recommended, FSA needs an improvement  plan to guide the agency in adopting recognized best practices and  following agency policy as well as a long-term institutional commitment to  comprehensively build these processes going forward. Given the  complexity and challenges in reengineering and improving FSA services,  the agency also needs to demonstrate on an ongoing basis that it can  follow policy, manage acquisitions, and deliver needed functionality.
FSA’s comments are reprinted in appendix III. The agency also provided  technical comments, which we incorporated as appropriate.
We are sending copies of this report to interested congressional  committees, the Secretary of Agriculture, the Director of the Office of  Management and Budget, and other interested parties. In addition, this  report will be available on the GAO Web site at http://www.gao.gov.
If you or your staffs have any questions on the matters discussed in this  report, please contact me at (202) 512-9286 or at pownerd@gao.gov.
Contact points for our Offices of Congressional Relations and Public  Affairs may be found on the last page of this report. GAO staff who made  major contributions to this report are listed in appendix IV.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: Our objectives were to (1) describe what led to the recent decision to halt  further development on MIDAS, (2) compare the functionality that MIDAS  has implemented to its original plans, and (3) evaluate the adequacy of  key program management disciplines in place for MIDAS and successor  programs.
To describe what led to the decision to halt further MIDAS development,  we reviewed documentation such as program planning artifacts, status  reports, key milestone reviews, and departmental or external reviews of  MIDAS. We identified key events and decisions from the program’s  December 2011 requirements review through the July 2014 decision to  halt further development on MIDAS. We analyzed the impact of these  events and decisions on MIDAS’s cost, schedule, scope, and  performance. Based on our analysis of key events and decisions, we  summarized the data in a timeline and identified key factors that led to the  decision to halt further development. We compared our assessments with  rationale provided by USDA for its decision to determine whether it was  similar to the factors we identified. We also interviewed relevant agency  and contractor officials to obtain their perspectives on what led to the  decision to halt further development of MIDAS.
We compared the functionality that MIDAS has implemented to its original  plans by reviewing the program’s December 2011 requirements and  identifying 24 unique features planned for MIDAS across 6 categories:  architecture, data, employee tools, customer tools, system integration,  and applications. We confirmed the delivered functionality by reviewing  program artifacts—including system test reports; program design  documentation; requirements traceability matrices; change request logs;  system architecture illustrations; status reports to the program’s Senior  Management Oversight Committee, the Office of Management and  Budget, and Congress; Exhibit 300 updates; budget requests;  assessments by the program’s independent verification and validation  contractor; and proposals to rebaseline program scope—for evidence that  the features had been implemented, deferred, or removed from scope.  We also obtained a live demonstration of the MIDAS system in a FSA  service center. We then compared the delivered functionality with what  was originally planned and developed graphics to illustrate what was  planned, delivered, and removed from the program. We also interviewed  relevant agency officials to discuss the original plans for MIDAS and  obtain clarification on functionality that FSA implemented.
To evaluate the extent to which USDA and FSA implemented key IT  program management disciplines, we assessed the implementation of  key practices and standards identified by the Project Management  Institute, the Software Engineering Institute at Carnegie Mellon University,  and GAO in the areas of (a) requirements development and  management, (b) project planning and monitoring, (c) system testing, and  (d) executive governance. Specifically, we assessed the extent to which  USDA and FSA had implemented each of the following 18 practices on  the program from December 2011 through October 2014.
Requirements development and management: elicit stakeholder needs and expectations, ensure requirements are complete and unambiguous, ensure requirements are prioritized, obtain commitment to requirements through a formal requirements analyze differences between the requirements and capabilities of  the intended solution (including commercial off-the-shelf solutions)  and address gaps, and ensure that requirements trace forward and backward among  development products.
Project planning and monitoring: establish a project plan with predefined expectations for cost,  schedule, and deliverables; update the project plan through change control procedures; and monitor progress against the project plan, including work  performed by contractors. establish well-defined test plans to include key elements such as  roles and responsibilities, test environment and infrastructure,  tested items and approach, a requirements traceability matrix  linked to test cases, risk and mitigation strategies, a testing  schedule, and quality assurance procedures;  test individual system components;  test the integration of system components; and perform end-to-end system testing to determine whether the  system is acceptable, interoperable with related systems, and  operationally suitable to users. establish a board and document a well-defined structure and  process for investment oversight; ensure that investments have a comprehensive business case  and use it to compare and select among alternative investments;  compare actual performance against estimates; assess whether projects are meeting expectations using  predefined criteria and checkpoints and take corrective action  when expectations are not being met; and  conduct post-implementation reviews to validate actual investment  results as compared to decision makers’ expectations for cost,  schedule, performance, and mission improvement outcomes and  to identify lessons learned that can be applied to future  investments.
We reviewed relevant USDA and FSA policies and guidance to determine  whether they were consistent with the best practices. We then assessed  the extent to which USDA and FSA implemented, partially implemented,  or did not implement the practices. To do so, we analyzed the following.
Requirements development and management artifacts such as  requirements traceability matrices, analyses of software gaps and  needed workarounds, gate review documentation on the status of  requirements, a USDA decision memorandum for the system  requirements review, letters from the system integrator, and  assessments of requirements practices by the program’s independent  verification and validation contractor.
Project planning and monitoring artifacts such as cost, schedule,  and scope baselines defined at the program’s March 2012 integrated  baseline review; earned value management reports from contractors,  program office status reports, Federal IT Dashboard updates; Exhibit  300 updates; change request logs; an assessment of project  management practices by the program’s independent verification and  validation contractor; and a TechStat review by the USDA Office of  the CIO.
System testing artifacts such as the program’s testing strategy and  more detailed test plans, program status reports on key phases of  testing, the program’s independent verification and validation  contractor’s assessment of integration testing adherence to best  practices, the program’s risk and issue list, Senior Management  Oversight Committee briefings that discussed deferment of  performance and user testing, reports on system defects prior to and  after the system was operational, and summary reports by USDA and  contractor experts on key problems with the system after it became  operational.
Executive governance artifacts such as the program’s governance  concept of operations; review board charters; program business  cases and associated life cycle cost estimates; monthly status  briefings to the Senior Management Oversight Committee on the  program’s performance against estimates; documentation from the  program’s system requirements review, critical design review, test  readiness review, go-live (implementation) review for the first software  release, including conditions and corrective actions identified in  decision memoranda; a post-implementation review by the program’s  independent validation and verification contractor; and draft plans to  identify lessons learned.
We also interviewed relevant agency and contractor officials to discuss  the implementation of management disciplines on MIDAS.
We performed our work at USDA, FSA, and contractor offices in  Fredericksburg and Hanover, Virginia, and in the Washington, D.C. area.  We conducted this performance audit from October 2014 to June 2015 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
section_title: Appendix II: FSA Farm Programs
paragraphs: Appendix II: FSA Farm Programs Description  Makes payments related to the difference between commodity crop market prices and  farm program prices.
Provides a financial incentive to produce bio-energy crops.
Provides a financial incentive to environmentally conserve farm or ranch land.
Provides a financial incentive to environmentally conserve farm or ranch land.
Provides a financial incentive to environmentally conserve farm or ranch land.
Provides a financial incentive to environmentally conserve farm or ranch land.
Offers assistance for transferring environmentally conserved farm or ranch land to  beginning, veteran, or socially disadvantaged producers.
Makes payments related to transitioning certain cotton crops to other alternatives.
Grants compensation for livestock, honeybee, and fish production losses related to  weather, disease, or other emergencies.
Furnishes payments and technical assistance to rehabilitate farmland damaged by natural  disaster.
Emergency Forest Restoration Program  Makes payments to restore forest land damaged by natural disaster.
Offers loans related to building or improving farm storage and handling facilities.
Grants compensation for transportation costs related to disadvantaged farm or ranch  geography.
Implements voluntary practices taken to environmentally protect source water.
Grants compensation for ranching losses due to drought or fire on grazing land.
Grants compensation for livestock losses due to weather or certain predators.
Makes payments in lieu of applying for loans for which the producer is eligible.
Makes payments related to the difference in actual and threshold dairy margins.
Offers loans using commodity crops as collateral.
Makes payments related to uninsurable crops lost to natural disaster.
Offers loans to processors of domestically produced sugarcane and sugar beets.
Offers loans to construct or upgrade sugar cane and sugar beet storage facilities.
Makes payments for replanting or rehabilitating eligible trees, bushes, and vines damaged  by natural disaster.
section_title: Appendix III: Comments from the Department of Agriculture
section_title: Appendix IV: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, the following staff made key  contributions to this report: Colleen Phillips (assistant director),  Christopher Businsky, Claudia Fletcher, Nancy Glover, Joshua Leiling,  Jamelyn Payan, and Edward Varty.