Farm Program Modernization: Farm Service Agency Needs to Demonstrate the Capacity to Manage IT Initiatives
Released: 2015-06-18T00:00:00

Letter

Background

Among other missions, USDA manages benefit programs that support farm and ranch production, natural resources and environmental conservation, and rural development. FSA is one of three USDA service center agencies that manage benefit programs for farmers and ranchers. Currently, FSA manages 23 farm benefit programs identified by, among other legislation, the Agricultural Act of 2014 (commonly referred to as the 2014 Farm Bill).assistance for livestock, honeybees, and farm-raised fish to providing incentives for resource conservation. Appendix II provides a brief description for each of the 23 farm benefit programs.

FSA benefit programs fall into four core categories: farm loan programs, which are to provide direct loans or loan guarantees to family farmers who could not otherwise obtain agricultural credit; income support and disaster assistance programs, which are to provide farmers and ranchers with an economic safety net to help them maintain their operations during difficult times; commodity operations programs, which are to expand market opportunities for farmers; and conservation programs, which are to help maintain and enhance the nation’s natural resources and environment.

Overview of the Services FSA Provides Its Customers

Over the last two decades, FSA has provided services to customers supporting the farm benefit programs at its approximately 2,100 local offices. To participate in FSA programs, customers may need to visit local service center offices multiple times throughout the year because certain transactions cannot be done electronically via e-mail or the Internet.

A new customer would typically go through several steps to enroll in a benefit program:

At first, a customer needs to establish a relationship with the agency by providing certain basic information about his operation that will be used in determining eligibility.

Based on this information, an FSA agent is to create a master farm record for the customer. The farm record is to include specific information about the farm such as identification numbers for fields and tracts (a tract is one or more contiguous fields), location information, and a list of commodities that the farm is able to produce.

Once a customer has established a relationship, customers learn about available FSA programs, receive information on eligibility and estimated benefits under a particular program, and generate a draft agreement for participation in a particular program.

The next step is for the customer to submit a final agreement to FSA for their participation.

Subsequently, throughout the year, the customer documents information about crops in an acreage report, which must be prepared for each applicable tract and growing season. The customer brings acreage reports and other forms to the FSA service center in person. Because different crops have different reporting deadlines, a customer may need to visit the service center multiple times to fill out reports for different crops.

In addition to establishing relationships and administering benefits programs, there are two other key activities that service centers perform: handling acreage reports and printing maps.

Handling acreage reports. This is one of the most critical functions for a service center. The FSA agent verifies that a customer is eligible for a benefit and compares the acreage amount in the acreage report against the acreage amount in the master farm record. The agent then computes the payment amount and authorizes the payment to be made to the customer.

Printing maps. Customers also often request maps of their tracts from FSA service centers to help plan for the next growing season because the maps can only be produced in hard copy. According to FSA service center officials, this is one of the customer’s most requested services.

In order to provide these services, FSA staff use a variety of computing environments and software applications, including a central “web farm,” consisting of an array of interconnected computer servers that exchange data in support of data storage and web-based applications; a central IBM mainframe that hosts non-Web applications and data; a distributed network of IBM Application System 400 computers and a common computing environment of personal and server computers at each local office.

Goals and History of MIDAS

In early 2004, FSA began planning the MIDAS program to streamline and automate farm program processes and to replace obsolete hardware and software. FSA identified these goals for the program:

Replace aging hardware: Replace Application System 400 computers, which date to the 1980s and are obsolete and difficult to maintain, with a hosting infrastructure to meet business needs, internal controls, and security requirements.

Reengineer business processes: Streamline outmoded work processes by employing common functions across farm programs. For example, determining benefits eligibility could be redesigned (using business process reengineering) as a structured series of work steps that would remain consistent regardless of the benefits requested.

Improve data management: Make data more readily available to FSA personnel and farmers and ranchers—including online self- service capabilities—and increase data accuracy and security.

Improve interoperability with other USDA and FSA systems: Integrate with other USDA and FSA modernization initiatives, including the Financial Management Modernization Initiative for core financial services that meet federal accounting and systems standards, the Geospatial Information Systems to obtain farm imagery and mapping information, and the Enterprise Data Warehouse to provide enterprise reporting.

From 2004 through 2010, FSA went through several changes in direction before selecting a technical solution for MIDAS:

FSA drafted initial requirements for MIDAS in January 2004.

FSA halted requirements development in early 2006 when program officials decided that the proposed customized solution would not meet future business needs.

FSA subsequently changed its approach in the Summer of 2006 from acquiring customized software to acquiring commercial off-the-shelf enterprise resource planning software. The program estimated that it would cost $305 million to implement MIDAS, but this estimate had a high degree of uncertainty.

In February 2008, FSA analyzed how its farm program functions would map to functions available in a commercial off-the-shelf enterprise resource planning software suite from vendor SAP, which had been selected for two other USDA modernization initiatives—the Financial Management Modernization Initiative and the Web Based Supply Chain Management program. This analysis concluded that MIDAS processes generally mapped to the SAP software.

Based on that analysis and a software alternatives analysis conducted in mid-2008, FSA decided to proceed with SAP enterprise resource planning software as the solution for MIDAS. FSA also decided to accelerate the time frame for implementing the solution from the 10 years originally planned to 2 years. To accomplish this, FSA planned to compress the requirements analysis phase from 4 years to 5 months, and reduce the analysis and design phase from 3.5 years to 9 months.

A request for quotations for the MIDAS system integrator contract was released in July 2009, and a contract based on this request was awarded to SRA International in December 2009. After a short delay due to a bid protest, the system integrator began work in May 2010 with an initial firm fixed price task order for $4.4 million through December 2010. By this point, FSA had also awarded six other contracts for services to support additional aspects of this initiative, including software licenses, project management support, and technical support. As of October 2010, FSA planned to spend $169 million—more than half of the program’s $305 million estimate—on the system integrator contract through fiscal year 2012.

Figure 1 depicts a timeline of key milestones for MIDAS from its inception through the initiation of work by the system integrator.

In the years after the system integrator began to work on developing MIDAS, the program ran into cost, schedule, and technical problems. These issues ultimately resulted in a July 2014 decision to halt further development on the MIDAS program. We discuss the events that led to this decision later in this report.

History of MIDAS Program Management and Governance

In September 2007, FSA established a MIDAS executive program manager and a program office to oversee the program and its supporting contracts. According to FSA officials, the program office reported to a Senior Management Oversight Committee on a regular basis. The committee was chaired by the USDA Under Secretary for Farm and Foreign Agricultural Services and included the USDA Chief Information Officer, USDA Chief Financial Officer, and Administrator of the Farm Service Agency as additional board members. The committee had the following responsibilities, among others, in providing departmental oversight and support for the MIDAS program: communicating and providing strategic direction for FSA’s enterprise modernization; approving the MIDAS acquisition strategy; approving the program’s cost, schedule, and requirements baseline; ensuring MIDAS integration with departmental requirements and related initiatives and significant interdependencies; approving updates to business cases; and addressing issues escalated by a program-level review board.

At the department level, USDA had an IT governance process that was overseen by the Executive Information Technology Investment Review Board, which was chaired by the department’s Chief Operating Officer and included the department’s Under Secretary for Farm and Foreign Agricultural Services, Chief Information Officer, Chief Financial Officer, and other senior executives. The board was to approve IT investments that aligned with USDA’s mission and enterprise IT goals; provide executive management oversight, approval, and commitment to selected IT investments; and recommend to the Secretary a ranked group of IT investments proposed for funding.

In addition to USDA governance, the Office of Management and Budget (OMB) was involved in providing routine oversight for this program. OMB requested monthly status briefings on MIDAS’s progress after USDA’s Office of the Chief Information Officer conducted a TechStat review of the program in November 2012. These monthly briefings continued until October 2014, when the program was preparing to deploy its second and final software release.

Prior GAO Reviews Found Weaknesses in MIDAS Management and Oversight

In May 2008, at the request of the House and Senate Committees on Appropriations, we reported that MIDAS was in the planning phase and that FSA had begun gathering information and analyzing products to integrate its existing systems.adequately assessed the program’s cost estimate, in that the estimate had been based on an unrelated USDA IT investment. Moreover, the agency had not adequately assessed its schedule estimate because business requirements had not been considered when FSA reduced the implementation time frame from 10 years to 2 years.

We determined that the agency had not As a result, we reported that it was uncertain whether the department could deliver the program within the cost and schedule time frames it had proposed and recommended that FSA establish effective and reliable cost estimates using industry leading practices and establish a realistic and reliable implementation schedule that was based on complete business requirements. USDA generally agreed with our recommendations and implemented our recommendation to improve its tracking of user problems and clarifying roles and responsibilities between FSA and USDA’s Information Technology Services. However, it did not implement our other recommendations to establish reliable cost and schedule estimates based on complete business requirements.

Subsequently, in July 2011, we reported that MIDAS was in the proof-of- concept and system design phase, and noted that the scope included modernization of FSA’s systems for all of its (at that time) 37 farm programs by March 2014. We determined that the program’s cost estimate had a large degree of uncertainty. In particular, it did not yet reflect decisions that had occurred since the estimate was developed in 2007 and that the completion date of its current development phase was uncertain because of delays to key system design milestones.

In addition, we found that FSA had plans in place for MIDAS that incorporated selected leading practices and had defined governance bodies to provide oversight, but it had not implemented other key management practices, including forming an integrated team with representatives from IT programs that MIDAS depended on for its success, developing a schedule that reflected dependencies with relevant IT programs, and tracking the status of risks as planned. Moreover, it had not clearly defined the roles and coordination among the program’s governance bodies. We recommended that USDA update cost and schedule estimates, address management weaknesses in plans and program execution, and clarify the roles and coordination among the governance bodies. The department agreed with our recommendations and identified plans to address them. However, the agency did not complete efforts to address these recommendations before the decision was made to halt the program.

Poor Program Performance and Uncertainty Regarding Future Plans Led to the Decision to Halt Further MIDAS Development

In a July 2014 memo, the Secretary of Agriculture decided to halt the MIDAS program after deploying minimal functionality due to performance challenges in the early months after the system became operational, and delays in determining the remaining scope, schedule, and cost for the program. Our analysis similarly found that the key factors that led to this decision were poor program performance—characterized by rising costs, schedule overruns, reduced functionality, and problems with the system after it was deployed—as well as uncertainty regarding future plans for MIDAS.

Poor program performance: FSA experienced significant cost and schedule delays in developing MIDAS, which led it to defer or remove expected functionality and to eliminate key system tests prior to deploying the system. Once the initial MIDAS functionality was deployed, FSA employees encountered serious problems in using the system. A timeline of key events and decisions that factored into MIDAS’s poor program performance include: In December 2011, MIDAS was envisioned to deliver significant functionality in phases, with the majority of functions to be delivered in the first phase. In further designing the system in March 2012, FSA decided to remove selected functionality from the first phase, and to deliver the remaining functionality in two deployments. As of June 2012, FSA estimated that development costs would be $330 million.

As FSA began to develop the system, however, it experienced cost and schedule overruns. For example, by August 2012, FSA had overrun its cost estimates by 11 percent and schedule estimates by 10 percent. These overruns were due, in part, to delays in completing customization of the commercial software, redesign work on interfaces, delays in testing individual system components as a result of including more customization than planned, and delays in data conversion and remediation efforts.

In order to help meet cost and schedule demands, in September 2012, FSA decided to split the two deployments into three deployments and to focus primarily on the first deployment. After continuing to experience schedule delays as it moved into system testing, FSA decided to remove additional functionality from the first deployment (including acreage reporting and customer records functions).

To try to stay on schedule, the MIDAS program also obtained approval from senior USDA and FSA management in early 2013 to defer key testing activities—including performance testing and user acceptance testing—until after the system became operational. These tests were not performed after deployment.

When FSA implemented its first software release in April 2013, MIDAS experienced significant technical problems, which is not surprising given its lack of testing. For example, users experienced significant problems with the system such as the geospatial information system (GIS) functionality, accuracy of farm record data, and system response time. Also, within 3 months after the deployment, there were 62 critical, 172 major, 236 average, and 69 minor defects that needed to be addressed. As a result, the time allotted to fix problems doubled in length—from 3 to 6 months—to accommodate the fixes required by the system.

Uncertainty regarding future plans: FSA was unable to establish a revised baseline for the program after experiencing cost and schedule overruns in developing the initial system release because the proposals were too costly or not aligned with the department’s budget and IT plans. A timeline of key events include:

After conducting a TechStat review in November 2012, USDA’s Office of the Chief Information Officer (CIO) directed FSA to establish a new program baseline by January 2013. However, FSA did not deliver a new baseline by the deadline. According to FSA officials, the agency made three different attempts to salvage what it could from MIDAS: In May 2013, the program submitted a proposed baseline to FSA management that included delivering the full set of envisioned MIDAS functionality by late 2016 at an increased cost. At $659 million, this new cost estimate was almost twice as expensive as the earlier baseline estimate of $330 million. FSA management did not approve the proposed baseline due to the cost.

In August 2013, FSA submitted a proposal to USDA for delivering less than the full set of MIDAS functionality at a reduced cost to implement and with a shorter development schedule. Specifically, the program’s scope no longer included applications for the 2014 Farm Bill programs, which FSA decided to transfer to its web farm. The cost estimate was reduced to $583 million through fiscal year 2015 and development was to be completed by mid-2015. However, USDA’s Office of the CIO rejected this rebaseline request because it required revisions in order to align with the department’s budget plans.

In February 2014, FSA submitted a proposal for the same functionality as the prior proposal at roughly the same cost and with a shorter development schedule. Under this new proposal, development was to be completed in mid-2015 at a cost of $584 million. This proposal also included a life cycle cost estimate for MIDAS of $1.026 billion through fiscal year 2021. However, in June 2014, the department’s Executive Information Technology Investment Review Board placed the program’s third rebaseline request on hold. The board wanted FSA to build the remaining functionality in smaller increments and to work in partnership with other agencies to develop an enterprisewide solution for acreage reporting and customer portal tools consistent with the 2014 Farm Bill and department priorities.

After the third rebaseline proposal was not approved, the department’s review board recommended to the Secretary of Agriculture that MIDAS halt development after the completion of the customer records release. In July 2014, the Secretary decided to approve the board’s recommendation.

FSA deployed its customer records release in December 2014. As of March 2015, FSA had spent about $423 million on MIDAS, which was $93 million higher than the 2012 baseline estimate of $330 million. Of the $423 million, about half was spent on the system integrator contract.

Moving forward, FSA estimates that it will cost roughly $50 million to $60 million to continue to operate and maintain the system each year. As a result, MIDAS could cost approximately $825 million through the end of its useful life in 2021. Figure 2 illustrates the key events and decisions affecting MIDAS that led to the decision to halt further development, and table 1 identifies changes in MIDAS cost and schedule estimates over time.

FSA Delivered a Fraction of the Functionality Envisioned for MIDAS

FSA has delivered a fraction of what was originally planned for MIDAS. FSA first documented high-level plans for the functionality that MIDAS was to deliver when it completed a system requirements review in December 2011. At that time, MIDAS was envisioned to provide a single SAP platform to host data, applications, and business processes for administering farm program benefits, and advanced tools for customers and FSA employees.seamlessly with other USDA systems, including USDA’s financial system, In addition, it was expected to integrate geospatial information system, and enterprise data warehouse. Figure 3 provides an overview of FSA’s planned key features for MIDAS.

However, as the program ran into problems, FSA continued to remove planned features. Specifically, in June 2013, the MIDAS Senior Management Oversight Committee decided to develop applications for administering farm program benefits on FSA’s web farm rather than on the SAP platform, as originally intended, because it could no longer wait to transition from legacy systems. In addition, the ability for FSA employees to use critical acreage reporting tools with the data—a function that affects 85 percent of tasks—was removed from the program’s scope. Other key features were also removed from MIDAS, such as an online portal for farmers and other customers as well as integration with USDA’s financial system and enterprise data warehouse.

As a result of removing key features from MIDAS, FSA delivered a fraction of the originally envisioned functionality. MIDAS currently provides farm and customer record data on a SAP platform that is integrated with USDA’s geospatial information system. As a result of this partial implementation, FSA employees currently access, visualize, and edit data in MIDAS. They then turn to the web farm to run acreage reporting, administer benefits, and process payments.

FSA did not quantify what percentage of the originally envisioned MIDAS functions were delivered, and this task is complicated by the fact that there is not a complete set of requirements. However, if one were to weigh the key features equally, MIDAS has delivered about 20 percent of what FSA planned. That figure would be lower if one were to include the comparative importance of the functionality. For example, FSA has cited acreage reporting as a key feature affecting 85 percent of what FSA employees do and this is not included in MIDAS. In addition, integration with the financial system was one of the key reasons for going with the SAP solution, and this, too, was not delivered. Figure 4 compares the functionality planned for MIDAS in 2011 to what has been delivered.

FSA Did Not Adequately Implement Key Program Management Disciplines on MIDAS and Lacks the Capacity to Effectively Manage Successor Programs

FSA did not adequately implement program management disciplines on MIDAS in four key areas—requirements development and management, project planning and monitoring, system testing, and executive-level governance—and lacks the demonstrated capacity to manage successor programs. Leading government and industry organizations call for best practices such as obtaining commitment to a requirements baseline and ensuring requirements are prioritized and traceable; managing changes to project plans and conducting progress monitoring; testing the system to determine whether it is acceptable to users; and implementing executive- level governance to include comparing performance against expectations and assessing maturity at key checkpoints based on predefined criteria.

USDA and FSA policies are consistent with these best practices.

However, in developing MIDAS, FSA did not adequately develop and manage requirements, effectively manage project plan changes, conduct meaningful progress monitoring, execute critical tests before the system became operational, and implement effective executive-level governance to prevent MIDAS from falling short of expectations. FSA and contractor officials explained that key practices were not always implemented because, among other things, the program’s scope was not well- understood, USDA and FSA did not follow its own policies, and management allowed the program to continue despite known weaknesses. Moreover, while FSA officials have acknowledged weaknesses in each of these management disciplines, the agency has not established plans to improve its management of successor programs.

Until FSA addresses shortfalls in key program management disciplines on successor programs to MIDAS, the agency will be at an increased risk of producing additional projects with cost overruns and schedule slippages while contributing little to mission-related outcomes. Further, until FSA establishes improvement plans, it will be difficult for the agency to demonstrate that it has the capacity to effectively manage IT acquisitions and it will be at a higher risk of failure for any new or ongoing IT initiatives.

FSA Did Not Adequately Develop and Manage MIDAS Requirements

Requirements establish what the system is to do, how well it is to do it, and how it is to interact with other systems. Leading industry organizations such as the Software Engineering Institute have recommended practices for the effective development and management of requirements such as eliciting stakeholder needs, ensuring that requirements are complete and unambiguous, prioritizing them, obtaining formal commitment to them, assessing any gaps with the proposed solution, and ensuring that each requirement traces back to the business need and forward to its design and testing. FSA has established policies and guidance for developing and managing requirements that are consistent with these recognized practices.

Of six key practices in requirements development and management, FSA implemented one practice, partially implemented two practices, and did not implement three practices. Specifically, FSA documented requirements for MIDAS based on needs gathered from stakeholders prior to a system requirements review in December 2011 and throughout the development of the system. The agency also identified its process for addressing software gaps with the SAP solution and documented workarounds for certain capabilities. However, FSA did not adequately develop and manage MIDAS requirements because the agency did not always develop complete requirements, prioritize its requirements, obtain commitment on a requirements baseline, document solutions to gaps with SAP software that had been known for years and were required for the program’s success, and ensure that requirements were traceable to development products.

Table 2 identifies the extent to which FSA implemented key practices for developing and managing requirements for MIDAS.

FSA officials and supporting documentation show several reasons for the lack of requirements development and management discipline on MIDAS. For example:

FSA officials noted that problems with the completeness and specificity of requirements persisted because guidance on how to ensure requirements completeness and specificity was not implemented until shortly before the system requirements review and it took time for changes to be made to the requirements. Also, officials cited challenges in the complexity of writing requirements for business processes related to GIS capabilities.

For selected key milestone reviews and decision points, the program’s executive governance board did not verify that key requirements artifacts and processes were mature enough to proceed because USDA and FSA did not establish a governance process that required the board to perform such reviews.

In August 2012, the IV&V contractor reported that the program’s gap analysis lacked specific details to fully understand activities for identifying, reviewing, assessing, and validating gaps. In addition, GIS was not part of the system integrator’s initial scope and additional resources and expertise had to be acquired during MIDAS’s development.

FSA and contractor staff had ongoing trouble getting access to one of the program’s two requirements management tools because system access rights were controlled by a different USDA agency. While FSA worked on this issue, it continued to be a problem throughout the development of release 1. The IV&V contractor reported in March 2012 that this lack of access limited their ability to perform requirements traceability. This made it more difficult to manage the baseline scope and configuration of the release. In addition, the IV&V contractor reported in February 2014 that the program office did not follow adequate document configuration management processes to control changes, thereby making it difficult to maintain traceability between requirements and design documents.

FSA officials noted that the program did not obtain a requirements baseline approval or prioritize its requirements because the program lacked the necessary discipline and rigor for requirements management activities during the first software release.

FSA’s lack of requirements development and management discipline on MIDAS impacted the program in several ways. For example, by not establishing a requirements baseline the agency did not have a firm commitment on the mission-related outcomes MIDAS would satisfy. In addition, not having prioritized requirements limited the agency’s ability to make decisions on which scope to defer or remove from the program when faced with cost and schedule overruns.

Both FSA and IV&V contractor officials noted that the program had demonstrated improvements in practices associated with requirements management for the second software release in December 2014. For example, an official for the IV&V contractor stated that requirements traceability was significantly better and that a requirements baseline was established for the second release. Unless FSA ensures that successor programs to MIDAS are fully implementing key requirements development and management practices, the agency will not have reasonable assurance that its IT modernization efforts will meet stakeholder needs and contribute to mission-related outcomes.

FSA Established Project Plans for MIDAS, but Did Not Effectively Manage Plan Changes or Monitor Progress

Leading organizations such as the Project Management Institute and Software Engineering Institute have recommended best practices for project planning and monitoring. FSA also has policies and guidance that are consistent with recognized practices. Project planning maintains plans as the basis for managing the project’s activities. Recommended best practices call for documenting and evaluating changes to established project plans to determine whether they require updates to initial planning estimates for cost, schedule, and scope. Project monitoring provides an understanding of the project’s progress by comparing actual work completed to a plan consisting of predefined expectations for cost, schedule, and deliverables. Best practices state that monitoring progress is important because it helps project managers take timely corrective actions when performance deviates significantly from plans.

Of three key practices in project planning and monitoring, FSA partially implemented one practice and did not implement two practices. The agency established a project plan for MIDAS with predefined expectations for cost, schedule, and scope based on its integrated baseline review in March 2012. For several months following this review, the program executed to these plans and tracked certain technical and programmatic changes in its change control log.

However, the agency did not effectively manage plan changes or monitor progress. For example, FSA did not update its baseline plans when it revised the solution architecture and when it deferred planned testing activities before the initial software release. In addition, FSA continued to develop deferred functionality for approximately 20 months without an approved rebaseline for these efforts. Also, FSA’s initial monitoring of contractor performance lacked insight into the progress of deliverables and contractor performance reporting was halted from December 2012 through October 2014. Table 3 identifies the extent to which FSA implemented key practices for project planning and monitoring for MIDAS.

USDA and FSA officials provided several explanations for the agency’s shortfalls in project planning and monitoring. FSA officials acknowledged that they did not update baseline project plans to reflect changes in the solution architecture and testing phases prior to the initial software release, but noted that they briefed the Senior Management Oversight Committee on these changes. The USDA CIO noted that MIDAS halted progress monitoring of contractors in December 2012 because managers were already aware that the program was performing poorly and was in need of a rebaseline. Also, the Director of the MIDAS business management office stated that while the program’s cost baseline was not at a detailed level, the contractors’ cost estimates included additional details on work products and deliverables. However, the program was not monitoring progress based on those details.

By not revising the project plan after making significant revisions to its approach, the program’s cost, schedule, and scope were no longer effective benchmarks for measuring performance. Without meaningful progress monitoring initially and as the program shifted its focus, program managers and executive stakeholders had less insight into the deliverables being produced by contractors and less control over the program’s outcomes.

According to FSA and IV&V contractor officials, the program provided a baselined cost, schedule, and scope for its second software release and executed improved discipline in managing plan changes. However, until FSA ensures that successor programs to MIDAS are fully implementing key project planning and monitoring practices, the agency will be at an increased risk that future projects will experience cost and schedule overruns and achieve less than expected outcomes.

FSA Defined Test Plans for MIDAS, but Did Not Execute Critical Tests before the System Became Operational

According to relevant leading industry practices and government guidance, system testing should be progressive, meaning that it should consist of well-defined test plans and a series of test events that build on and complement previous events in the series. Testing should first focus on the performance of individual system components, then on the performance of integrated system components, followed by system-level tests that focus on whether the system (or major system increments) is acceptable, interoperable with related systems, and operationally suitable to users. FSA established policies and guidance for system testing on MIDAS that are consistent with recognized practices.

Of four key practices in system testing, FSA implemented one practice, partially implemented two, and did not implement one. FSA defined test plans for MIDAS, but the agency did not execute critical performance and user testing before the system became operational. FSA had test plans in place that generally defined key elements, such as the roles and responsibilities of groups that were to conduct testing, hardware and software to support testing, and a schedule that defined how long and in what order test events were to occur. In addition, FSA conducted testing on individual and integrated components. However, MIDAS’s test plans were missing a key element—traceability between system test events and requirements. Also, integration testing took longer than planned and the program decided to defer testing that was to validate whether system performance met requirements and was acceptable to users until after the system went live. Table 4 identifies the extent to which FSA implemented key system testing practices for MIDAS.

FSA’s shortfalls in system testing were due in part to technical problems and delays in developing GIS capabilities and a desire by department and agency management to keep the target deadline for the initial release of MIDAS. Early in the development of the GIS capabilities, FSA ran into technical problems that required additional time and resources to address. Since GIS development had been delayed, integration testing with the GIS capabilities also had to be delayed. By April 2013, integration testing was still ongoing and FSA had to make a decision whether to delay the implementation of MIDAS or allow the system to go live while accepting the risk of not conducting performance/stress, regression, and user acceptance testing. While the program warned of the risks of deploying MIDAS with outstanding defects and incomplete testing, senior department and agency officials decided to accept these risks in order to deploy the system by April 2013.

Incomplete testing on MIDAS did not provide users an opportunity to identify key problems with the system and whether it met their needs before it went live. After the system went live, users experienced significant problems such as GIS functionality, accuracy of farm record data, and system response time. Within 3 months after the initial MIDAS release, there were 62 critical, 172 major, 236 average, and 69 minor defects that needed to be addressed. The program had a plan in place to address performance problems after the system went live, but, due to the number of problems, it had to extend the contract for addressing system defects from 3 months to 6 months.

FSA officials and the program’s IV&V contractor have acknowledged the shortfalls in system testing practices and stated that the program has taken steps to improve system testing on the second and final MIDAS release. For example, FSA and IV&V contractor officials noted that the program conducted user acceptance testing prior to deploying functionality for managing customer records (also called business partner functionality). While the agency recognized the need to improve its testing on the second MIDAS release, it has not demonstrated that it has institutionalized sound system testing practices. Until it does so, the agency will be at higher risk of delivering systems that have performance issues and do not fully meet users’ expectations.

Executive-level Governance Was Ineffective in Preventing MIDAS from Falling Short of Expectations

We assessed best practices used in industry, academia, and government to develop the IT Investment Management Framework to provide a method for evaluating and assessing how well an agency is selecting and managing its IT resources. Efforts to build a foundation for IT governance involve establishing specific critical processes, such as instituting investment boards, selecting investments, controlling investments as they are developed and deployed, and reviewing investments after they are deployed.

Instituting investment boards. Successful organizations establish an IT investment board comprised of senior executives who are responsible for operating according to documented guidance, policies, and procedures that align with existing IT governance processes, identify decision gates to be reviewed and approved by the board, and establish entry/exit criteria to be reviewed at each decision gate. The board is also responsible for ensuring that investment decisions address stakeholder needs and are made in the best interest of the organization.

Selecting IT investments. Successful organizations identify, use, and store comprehensive data—including a business case that defines the life cycle cost estimate and benefits to be realized—in order to support investment decision making. Reselecting ongoing projects is an important part of this critical process; if a project is not meeting established goals and objectives, the organization must make a decision on whether or not to continue to fund it.

Controlling IT investments. Organizations should have a documented, well-defined process for overseeing ongoing investments once they have been selected. Effective investment oversight and evaluation involves, among other things, (1) comparing actual performance against cost and schedule estimates; and (2) assessing whether projects are meeting expectations against developmental milestones using predefined criteria and decision gates, and taking corrective actions when expectations are not being met.

Reviewing IT investments after deployment. Once the project has transitioned from the development phase to the operations and maintenance phase, organizations should conduct a post- implementation review to compare actual investment results with decision makers’ expectations for cost, schedule, performance, and mission improvement outcomes. The lessons learned from these reviews can be used to modify future investment management decision making. In 2013, USDA issued updated policies and guidance that are generally consistent with these practices.

Of five key practices in executive-level IT governance, FSA partially implemented two practices and did not implement three practices. Specifically, FSA partially implemented steps to institute a governance board. It established a governance structure and process for MIDAS; however, its governance process was ineffective in preventing MIDAS from falling short of expectations. Specifically, FSA did not implement key steps for selecting and controlling investments, including establishing a comprehensive business case or life cycle cost estimate and comparing actual performance against estimates. Also, FSA partially implemented post-implementation review practices. The agency tasked a contractor with assessing the results and lessons learned from portions of MIDAS that were implemented; however, it did not conduct a comprehensive review of the lessons learned on the program as a whole.

Table 5 identifies the extent to which USDA and FSA implemented key executive governance practices for MIDAS.

The governance of MIDAS was ineffective, in part, because USDA’s Office of the CIO did not ensure that MIDAS followed its policies and guidance for IT governance. In 2011, we reported that governance boards had not been reviewing MIDAS at key decision points using criteria defined in department guidance and recommended that the department and agency collaborate to document how the department is meeting its policy for IT investment management for MIDAS, to include investment reviews.not address it while MIDAS was in development. According to FSA officials, USDA did not have guidance for IT governance providing defined decision gates with standard criteria and documentation While the department agreed with our recommendation, it did requirements. FSA officials noted that from 2011 until 2013, the agency used a governance process involving a program-specific gate review plan for MIDAS based on SAP’s system development methodology.

USDA’s Under Secretary for Farm and Foreign Agriculture Services and the Chief Financial Officer stated that there was a breakdown in the governance process for MIDAS, particularly on its initial development. The Under Secretary noted that the Senior Management Oversight Committee made the best decisions it could based on the information it had, but the information that FSA had reported to the committee did not adequately portray the extent of the cost, schedule, and technical problems or decisions that had been made on scope changes. For example, the Under Secretary and the Chief Financial Officer stated that they were not informed that FSA had been developing key functions in both MIDAS and on the agency’s web farm—a key change in the original scope—until early 2013. In addition, the Under Secretary and the Chief Financial Officer noted that they were not informed until early 2013 that FSA had made decisions to remove or defer additional scope—including acreage reporting—from the first software release.

Subsequently, in 2013, MIDAS began piloting USDA’s new IT governance process, called the Integrated IT Governance Framework. This framework required MIDAS to report its performance to and obtain approval from a department-level investment review board. In following this new governance framework, the investment review board approved the final decision to implement the second MIDAS software release and recommended to the Secretary of Agriculture to halt further development on MIDAS.

While the recently updated governance framework established by USDA has potential for improving FSA’s IT modernization efforts, unless USDA and FSA take additional steps or develop a mechanism to help ensure that successor programs to MIDAS programs are fully implementing key executive IT governance practices—including practices for selecting, controlling, and reviewing investments—department and agency management will not have reasonable assurance that oversight is effective in preventing future IT investments from falling short of expectations.

FSA Is Planning Next Steps after MIDAS as Required, but Has Not Demonstrated the Capacity to Manage Future IT Initiatives

Required by law to automate, integrate, and modernize its farm program services, FSA has begun planning how it will do so. In an explanatory statement accompanying the 2015 appropriations act, Congress directed USDA to, among other things, deliver a modernized functional system that builds existing farm program applications into an integrated system, delivers increased efficiency and security, retires redundant legacy systems, eliminates the path of siloed legacy applications, capitalizes on the investment that USDA has already made in the enterprise platform, addresses the new requirements of the 2014 Farm Bill, and improves on the capabilities originally proposed to Congress and the nation’s farmers and ranchers. The appropriations act also mandated that FSA develop a plan for IT related to MIDAS and other farm program delivery systems prior to obligating more than 50 percent of the $132 million made available in fiscal year 2015. This plan is to identify each investment’s capabilities and mission benefits, estimated life cycle cost, key milestones, and alignment with FSA’s IT Roadmap.

In addition, the 2014 Farm Bill includes provisions for streamlining acreage reporting to reduce the administrative burden on farmers and producers. This is to be done by, among other things, requiring the Secretary of Agriculture to ensure that producers may report information electronically (including geospatial data) and that improvements are made in the areas of coordination, information sharing, and administrative work with FSA, the Risk Management Agency, and the Natural Resources Conservation Service.

FSA has begun planning how it will move forward in its modernization efforts to fulfill the functionality that was envisioned—but not delivered— by MIDAS. According to FSA officials, the agency plans to document its decisions for addressing acreage reporting tools, online customer tools, and other functionality that was removed from MIDAS in its IT Roadmap by the end of Spring 2015. Those plans may include decisions to partner with other USDA agencies, acquire new commercial off-the-shelf software, and/or develop and enhance functionality on the agency’s web farm.

In its fiscal year 2016 budget request, FSA noted that, while the mix of investments may fluctuate based on its prioritization process and business requirements, the agency intends to pursue incremental, modular investments such as the following.

Customer self-service tools: Expanding on existing online services and partnering with other USDA agencies (including the Acreage Crop Reporting Streamlining Initiative) to provide farmers and ranchers online access to relevant information, including remote and/or mobile access to their data and programs.

Expanded customer service: Piloting a program to find new ways to deliver programs and service support through the agency’s repository of geospatial and farm information.

Increased IT investments to support FSA process improvements: Delivering incremental improvements to address pain points and inefficiencies identified by field office staff as impacting their effectiveness in servicing customers. Improvements in the pipeline could range from simple items such as simplifying the printing of farm maps or customized reports to continuing the incremental integration of stove-piped systems through establishing or enhancing common eligibility, payment, and obligation frameworks.

In addition, FSA officials stated that the agency is incorporating lessons learned into future plans, including building smaller, incremental releases with a defined scope, cost, and schedule and defined benefits for the customer; extending an organizational change agency network to provide input on pain points and process improvements; driving the prioritization of investments through business needs instead of technology; and integrating technology capabilities, including SAP, into decision- making processes and alternatives analyses, so the technologies for each project will be determined based on what best matches the business requirements.

However, FSA has not established plans to improve its ability to successfully manage major IT investments. Specifically, FSA officials have not committed to improving agency practices in the four areas we reviewed because they believe that they have already addressed the problems. While agency officials acknowledge that mistakes were made on the first MIDAS release, they stated that they did a better job delivering the second release. For example, the MIDAS Program Executive reported that the agency established requirements for the second release, established a schedule for developing and deploying the release, performed adequate testing prior to deploying the release, and that oversight bodies were kept informed. While the second release was more successful than the first, it was much less complex. The second release involved a limited amount of functionality that had been in development for several years before it was deferred from the first release. Further, the relatively discrete amount of work involved and the establishment of baseline plans 3 months prior to the release allowed the project to deliver near cost and schedule estimates.

These efforts, however, are not sufficient to demonstrate that FSA will adhere to departmental policy or that it has practices in place to successfully plan, develop, and oversee future complex IT investments. USDA’s Under Secretary for Farm and Foreign Agriculture Services and Chief Financial Officer agreed that plans are needed to improve FSA’s ability to successfully manage IT investments. Until FSA establishes and implements improvement plans, it will be difficult to demonstrate that it has the capacity to manage IT acquisitions and the agency will have a higher risk of failure in future IT initiatives.

Conclusions

After spending about $423 million through March 2015, the MIDAS program was halted about 10 years after it was initiated. Key factors that led to the decision to halt the program included cost overruns totaling $93 million more than planned, schedule delays, performance issues, and management’s inability to decide on how to restructure the program for success.

In deploying the two MIDAS releases, FSA delivered about one-fifth of the functionality it had planned to deliver. MIDAS was envisioned to provide a seamless, integrated system that would allow farmers and ranchers to submit information electronically and allow FSA employees to process farm program benefits with built-in tools and access to GIS and other enterprise systems. However, due to the limited functionality that MIDAS provided, farmers and ranchers continue to submit information to FSA service centers in person while employees continue to use separate systems for processing acreage reports, farm program applications, and payments.

Even though USDA and FSA have system acquisition policies that are consistent with best practices in the areas of requirements development and management, project planning and monitoring, system testing, and executive-level governance, FSA did not implement the majority of these policies and practices in developing MIDAS and has not established plans to improve its approach. Until FSA establishes and implements a plan to adhere to agency policies and best practices, it will be difficult to demonstrate that it has the capacity to effectively manage IT acquisitions. Further, until the agency adheres to system acquisition policies and sound IT practices, it will have a higher risk of failure in future IT initiatives.

Recommendations for Executive Action

In order to institutionalize sound IT management practices and build FSA’s IT management capacity while improving service to the Nation’s farmers and ranchers, we are making five recommendations to the Secretary of Agriculture to:

Direct the FSA Administrator to establish and implement an improvement plan to guide the agency in adopting recognized best practices and following agency policy.

Direct the FSA Administrator to adhere to recognized best practices and agency policy in developing and managing system requirements before proceeding with any further system development to deliver previously envisioned MIDAS functionality. Specifically, the Administrator should ensure that requirements are complete, unambiguous, and prioritized; commitment to requirements is obtained through a formal requirements baseline; differences (or gaps) between the requirements and capabilities of the intended solution (including commercial off-the-shelf solutions) are analyzed; strategies to address any gaps are developed; and requirements are traced forward and backward among development products.

Direct the FSA Administrator to adhere to recognized best practices and agency policy in planning and monitoring projects. Specifically, the Administrator should ensure that project plans include predefined expectations for cost, schedule, and deliverables before proceeding with any further system development; updates to the project plan are made through change control processes; and progress against the project plan, including work performed by contractors, is monitored.

Direct the FSA Administrator to adhere to recognized best practices and agency policy in system testing. Specifically, the Administrator should establish well-defined test plans before proceeding with any further system development, and ensure that testing of (a) individual system components, (b) the integration of system components, and (c) the end-to-end system are conducted.

Direct the FSA Administrator to adhere to recognized best practices and agency policy in executive-level IT governance before proceeding with any further system development. Specifically, an executive-level governance board should review and approve a comprehensive business case that includes a life cycle cost estimate, a cost-benefit analysis, and an analysis of alternatives for proposed solutions that are to provide former MIDAS requirements prior to their implementation; ensure that any programs that are to accommodate former MIDAS requirements are fully implementing the IT program management disciplines and practices identified in this report; conduct a post-implementation review and document lessons learned for the MIDAS investment; and reassess the viability of the MIDAS technical solution before investing in further modernization technologies.

Agency Comments and Our Evaluation

We sought comments on a draft of this report from USDA. We subsequently received written comments from the FSA Administrator. While the agency did not explicitly agree or disagree with the recommendations, it cited steps it had taken and plans to take to implement best practices in the areas of requirements management, project planning and monitoring, system testing, and executive IT governance. However, the agency did not cite steps it would take to establish and implement an improvement plan to guide the agency in adopting recognized best practices and following agency policy. Because the agency is moving to implement best practices, we continue to believe that a plan—with steps, milestones, and performance measures—is warranted. Without such a plan, it will be difficult for the agency to demonstrate its progress and ensure that it has the capacity to manage IT acquisitions.

In its overall comments, FSA noted the following:

FSA stated that it has taken active steps to address the issues raised in the draft report by selecting a new CIO and initiating steps to acquire a third party assessor to holistically evaluate the technology solution for MIDAS and to make recommendations to inform a coherent IT strategy. We agree that selecting a CIO and obtaining recommendations on how to improve FSA’s IT strategy are sound steps. However, these steps are not enough to address the issues raised in this report. FSA must take additional steps to establish an improvement plan, and to implement practices and follow agency policy.

FSA stated that the recommendation by the USDA Executive IT Governance Board and the July 2014 decision by the Secretary of Agriculture to halt MIDAS development beyond release 2 (which FSA established to deliver the residual portion of the customer records functionality) allowed the agency to (1) focus attention and resources on applying lessons learned from release 1 and (2) apply program management best practices across key disciplines such as planning, requirements management, cost and schedule management, and system testing. We agree that the decision to halt MIDAS was a sound one, and that it allowed the agency to focus on the residual deliverables provided by release 2. However, we identified several management shortfalls that continued to persist after release 1 was deployed in April 2013. For example, FSA did not update its baseline cost, schedule, and scope plans from March 2012 to October 2014, even though it made significant decisions affecting scope and schedule. Also, from November 2012 through October 2014, the program did not have a project plan for monitoring progress due to numerous scope and schedule changes and relied on status reporting from draft schedules. In addition, while the USDA CIO gave the investment a “red” (high-risk) rating on the Federal IT Dashboard in December 2012, the MIDAS Senior Management Oversight Committee allowed MIDAS to continue until July 2014 without any improvement to the CIO’s rating. These findings are discussed in this report.

FSA stated that the organizational alignment around comprehensive improvement and quality of the MIDAS program (associated with release 2) is a clear demonstration of the agency’s capability to properly manage and deliver IT systems. However, until FSA establishes and implements a plan to adhere to agency policies and best practices, we believe the agency has not yet demonstrated that it has the capacity to effectively manage IT acquisitions.

FSA noted that the MIDAS program demonstrated an improvement in testing practices on release 2. Specifically, FSA stated that the period of time for transitioning from deployment to steady state concluded with zero critical defects and five major defects. We agree and acknowledged the agency’s improvements on system testing associated with the development of customer records in the report. Specifically, we noted that the program conducted user acceptance testing prior to deploying functionality for managing customer records. However, it has been our experience that it takes time to change an organization’s culture to adopt best practices. The agency will need to build upon this experience to ensure it consistently implements sound practices and follows agency policies in all future IT initiatives. We continue to believe that establishing and implementing an improvement plan, as we recommended, will aid the agency in doing so.

FSA stated that, since the first deployment of MIDAS functionality in April 2013, the agency implemented top-down organizational transformation to bolster FSA's ability to consistently deliver IT investments that provide their intended business value, within the targeted schedule and budget. The agency also stated that the MIDAS initiative identified a number of best practices that are being emulated to improve IT management agencywide. However, FSA did not provide supporting evidence for these efforts and our previously stated findings show that FSA did not sufficiently monitor project progress well beyond the first MIDAS release.

FSA stated that while our report acknowledges some of its improvements, our assessment of the extent to which USDA and FSA had implemented each management discipline reflects findings based on MIDAS release 1 activities, and therefore is not truly representative of FSA’s capacity to more broadly manage IT initiatives.

We believe our report accurately evaluates the implementation of key program management disciplines on the MIDAS acquisition. Our review assessed processes and practices over roughly 3 years (from December 2011 to October 2014), which included a significant amount of work on the customer records functionality. Specifically, the customer records functionality represented 1 of the 24 unique features FSA had originally planned for MIDAS as of December 2011. FSA had begun working on customer records in December 2011, and delivered about 30 percent of the customer records functionality with the initial MIDAS software release in April 2013. When faced with the firm commitment to deploy release 1 in April 2013, the agency decided to defer the remaining customer records functionality to release 2. Further demonstrating the limited scope of release 2, FSA established baseline project plans for release 2 in October 2014, just 3 months before deploying it in December 2014. While our report acknowledges that FSA improved selected practices in developing and deploying release 2, we do not believe that the scope or timeframe associated with this initiative provides sufficient evidence that FSA has established the capacity to manage large, complex acquisitions.

In addition, FSA provided the following comments regarding our recommendations:

With respect to our recommendation to establish and implement an improvement plan to guide the agency in adopting recognized best practices and following agency policy, FSA stated that the agency has undergone leadership transformation efforts over the last 12 months, including appointing a new Administrator, CIO, MIDAS Program Executive, and MIDAS Program Director. FSA noted that it gave additional reporting authority to the MIDAS Program Executive and moved the FSA CIO position from Kanas City, Missouri to Washington, D.C. to improve communication with the Administrator on agencywide initiatives. FSA stated that over the past year, FSA leadership placed additional emphasis, funding, and staff resources on ensuring that IT investments, decisions, dependencies, and operational plans are driven by business needs across the agency. The agency also stated that with its Business Strategy and IT Strategy, it is maturing IT planning and management capabilities needed for integrated IT solutions for Farm Programs and all of FSA’s lines of business. Finally, FSA noted that it is using a Strategic IT Roadmap to ensure IT programs are supporting the Business Strategy. We agree that FSA has taken steps over the past year to improve its IT management capabilities as we discuss in the report. However, these actions do not establish and implement an improvement plan to guide the agency in adopting recognized best practices and following agency policy. Until FSA does so, it will be difficult to demonstrate that it has the capacity to manage IT acquisitions. Thus, as previously discussed, we believe the agency should continue to establish and implement such an improvement plan.

Regarding our recommendation to adhere to recognized best practices and agency policy in developing and managing system requirements before proceeding with any further system development to deliver previously envisioned MIDAS functionality, FSA stated that the MIDAS program implemented all of the key practices for release 2. As previously stated, our report acknowledges that FSA improved selected practices in developing and deploying release 2. However, we do not believe that the scope or timeframe associated with this initiative provide sufficient evidence that FSA has improved its capacity to manage large, complex acquisitions. Further, we identified selected shortfalls in requirements management for release 2, including weaknesses in requirements traceability and prioritization. Moving forward, FSA stated that it will improve the rigor and adherence to key requirements management processes for all IT projects. We will continue to monitor the agency’s efforts to implement our recommendation.

Regarding adhering to recognized best practices and agency policy in planning and monitoring projects, FSA stated that the MIDAS program implemented all of the key practices for release 2. As previously stated, our report acknowledges that FSA improved selected practices in developing and deploying release 2. However, we do not believe that the scope or timeframe associated with this initiative provide sufficient evidence that FSA has improved its capacity to manage large, complex acquisitions. Further, our report identified shortfalls in program monitoring in the run up to deploying release 2, including weaknesses in updating project baselines to reflect program changes and in monitoring progress against a defined project plan. Moving forward, FSA stated that it would continue to mature and strengthen its project planning and monitoring practices through a partnership with a third-party capital planning center of excellence and through corrective action plans to address identified weaknesses. It also stated that it is implementing earned value management practices on MIDAS going forward. We will continue to monitor the agency’s efforts to implement our recommendation.

With respect to our recommendation to adhere to recognized best practices and agency policy in system testing, FSA stated that it established renewed commitment to MIDAS testing efforts and implemented all of the key practices for release 2. As previously stated, our report acknowledges that FSA improved selected practices in developing and deploying release 2. However, we do not believe that the scope or timeframe associated with this initiative provide sufficient evidence that FSA has improved its capacity to manage large, complex acquisitions. Moving forward, FSA noted that it plans to adhere to recognized best practices and agency policy in pursuing consistent or increased rigor around system testing to demonstrate the agency’s testing capabilities are consistent and repeatable across all IT projects. We will continue to monitor the agency’s efforts to implement our recommendation.

Regarding our recommendation to adhere to best practices and agency policy in executive-level IT governance before proceeding with any further system development, FSA stated that it is evaluating its governance structure to potentially include establishing work groups that would evaluate IT initiatives at a more granular level of detail. FSA also stated that it is working with USDA’s Office of the CIO to determine how MIDAS will align with the department’s governance framework and to identify the appropriate gate reviews, artifacts, and level of oversight. We will continue to monitor the agency’s efforts to implement our recommendation.

Overall, FSA’s poor performance and lack of results for more than 2 years contributed to its inability to deliver most of the intended functionality and led the Secretary of Agriculture to direct the agency to halt further development after release 2. The efforts that continued after USDA decided to halt further development on MIDAS in July 2014 and through the delivery of release 2 in December 2014 were to salvage a feature (customer records) that was almost fully developed by the time the department made this decision. Our assessments of project planning and monitoring and executive IT governance practices already include FSA’s efforts to manage the overall program and to continue developing customer records through October 2014. Nonetheless, if we were to consider FSA’s efforts on release 2 beginning in October 2014, we would have altered just 1 of the 18 key practices (conducting user testing) due to weaknesses that persisted beyond the deployment of release 1 in April 2013.

To its credit, FSA has (1) acknowledged that management improvements are needed and identified steps the agency plans to take; (2) made changes in key leadership positions; and (3) committed to delivering smaller, iterative IT projects going forward. However, our experience in reviewing federal IT acquisitions has shown that it takes time to build repeatable, robust processes. Implementing improvements during the last few months of a 3-year effort is not enough to demonstrate repeatable IT management capacity. As we recommended, FSA needs an improvement plan to guide the agency in adopting recognized best practices and following agency policy as well as a long-term institutional commitment to comprehensively build these processes going forward. Given the complexity and challenges in reengineering and improving FSA services, the agency also needs to demonstrate on an ongoing basis that it can follow policy, manage acquisitions, and deliver needed functionality.

FSA’s comments are reprinted in appendix III. The agency also provided technical comments, which we incorporated as appropriate.

We are sending copies of this report to interested congressional committees, the Secretary of Agriculture, the Director of the Office of Management and Budget, and other interested parties. In addition, this report will be available on the GAO Web site at http://www.gao.gov.

If you or your staffs have any questions on the matters discussed in this report, please contact me at (202) 512-9286 or at pownerd@gao.gov.

Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix IV.

Appendix I: Objectives, Scope, and Methodology

Our objectives were to (1) describe what led to the recent decision to halt further development on MIDAS, (2) compare the functionality that MIDAS has implemented to its original plans, and (3) evaluate the adequacy of key program management disciplines in place for MIDAS and successor programs.

To describe what led to the decision to halt further MIDAS development, we reviewed documentation such as program planning artifacts, status reports, key milestone reviews, and departmental or external reviews of MIDAS. We identified key events and decisions from the program’s December 2011 requirements review through the July 2014 decision to halt further development on MIDAS. We analyzed the impact of these events and decisions on MIDAS’s cost, schedule, scope, and performance. Based on our analysis of key events and decisions, we summarized the data in a timeline and identified key factors that led to the decision to halt further development. We compared our assessments with rationale provided by USDA for its decision to determine whether it was similar to the factors we identified. We also interviewed relevant agency and contractor officials to obtain their perspectives on what led to the decision to halt further development of MIDAS.

We compared the functionality that MIDAS has implemented to its original plans by reviewing the program’s December 2011 requirements and identifying 24 unique features planned for MIDAS across 6 categories: architecture, data, employee tools, customer tools, system integration, and applications. We confirmed the delivered functionality by reviewing program artifacts—including system test reports; program design documentation; requirements traceability matrices; change request logs; system architecture illustrations; status reports to the program’s Senior Management Oversight Committee, the Office of Management and Budget, and Congress; Exhibit 300 updates; budget requests; assessments by the program’s independent verification and validation contractor; and proposals to rebaseline program scope—for evidence that the features had been implemented, deferred, or removed from scope. We also obtained a live demonstration of the MIDAS system in a FSA service center. We then compared the delivered functionality with what was originally planned and developed graphics to illustrate what was planned, delivered, and removed from the program. We also interviewed relevant agency officials to discuss the original plans for MIDAS and obtain clarification on functionality that FSA implemented.

To evaluate the extent to which USDA and FSA implemented key IT program management disciplines, we assessed the implementation of key practices and standards identified by the Project Management Institute, the Software Engineering Institute at Carnegie Mellon University, and GAO in the areas of (a) requirements development and management, (b) project planning and monitoring, (c) system testing, and (d) executive governance. Specifically, we assessed the extent to which USDA and FSA had implemented each of the following 18 practices on the program from December 2011 through October 2014.

Requirements development and management: elicit stakeholder needs and expectations, ensure requirements are complete and unambiguous, ensure requirements are prioritized, obtain commitment to requirements through a formal requirements analyze differences between the requirements and capabilities of the intended solution (including commercial off-the-shelf solutions) and address gaps, and ensure that requirements trace forward and backward among development products.

Project planning and monitoring: establish a project plan with predefined expectations for cost, schedule, and deliverables; update the project plan through change control procedures; and monitor progress against the project plan, including work performed by contractors. establish well-defined test plans to include key elements such as roles and responsibilities, test environment and infrastructure, tested items and approach, a requirements traceability matrix linked to test cases, risk and mitigation strategies, a testing schedule, and quality assurance procedures; test individual system components; test the integration of system components; and perform end-to-end system testing to determine whether the system is acceptable, interoperable with related systems, and operationally suitable to users. establish a board and document a well-defined structure and process for investment oversight; ensure that investments have a comprehensive business case and use it to compare and select among alternative investments; compare actual performance against estimates; assess whether projects are meeting expectations using predefined criteria and checkpoints and take corrective action when expectations are not being met; and conduct post-implementation reviews to validate actual investment results as compared to decision makers’ expectations for cost, schedule, performance, and mission improvement outcomes and to identify lessons learned that can be applied to future investments.

We reviewed relevant USDA and FSA policies and guidance to determine whether they were consistent with the best practices. We then assessed the extent to which USDA and FSA implemented, partially implemented, or did not implement the practices. To do so, we analyzed the following.

Requirements development and management artifacts such as requirements traceability matrices, analyses of software gaps and needed workarounds, gate review documentation on the status of requirements, a USDA decision memorandum for the system requirements review, letters from the system integrator, and assessments of requirements practices by the program’s independent verification and validation contractor.

Project planning and monitoring artifacts such as cost, schedule, and scope baselines defined at the program’s March 2012 integrated baseline review; earned value management reports from contractors, program office status reports, Federal IT Dashboard updates; Exhibit 300 updates; change request logs; an assessment of project management practices by the program’s independent verification and validation contractor; and a TechStat review by the USDA Office of the CIO.

System testing artifacts such as the program’s testing strategy and more detailed test plans, program status reports on key phases of testing, the program’s independent verification and validation contractor’s assessment of integration testing adherence to best practices, the program’s risk and issue list, Senior Management Oversight Committee briefings that discussed deferment of performance and user testing, reports on system defects prior to and after the system was operational, and summary reports by USDA and contractor experts on key problems with the system after it became operational.

Executive governance artifacts such as the program’s governance concept of operations; review board charters; program business cases and associated life cycle cost estimates; monthly status briefings to the Senior Management Oversight Committee on the program’s performance against estimates; documentation from the program’s system requirements review, critical design review, test readiness review, go-live (implementation) review for the first software release, including conditions and corrective actions identified in decision memoranda; a post-implementation review by the program’s independent validation and verification contractor; and draft plans to identify lessons learned.

We also interviewed relevant agency and contractor officials to discuss the implementation of management disciplines on MIDAS.

We performed our work at USDA, FSA, and contractor offices in Fredericksburg and Hanover, Virginia, and in the Washington, D.C. area. We conducted this performance audit from October 2014 to June 2015 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: FSA Farm Programs

Appendix II: FSA Farm Programs Description Makes payments related to the difference between commodity crop market prices and farm program prices.

Provides a financial incentive to produce bio-energy crops.

Provides a financial incentive to environmentally conserve farm or ranch land.

Provides a financial incentive to environmentally conserve farm or ranch land.

Provides a financial incentive to environmentally conserve farm or ranch land.

Provides a financial incentive to environmentally conserve farm or ranch land.

Offers assistance for transferring environmentally conserved farm or ranch land to beginning, veteran, or socially disadvantaged producers.

Makes payments related to transitioning certain cotton crops to other alternatives.

Grants compensation for livestock, honeybee, and fish production losses related to weather, disease, or other emergencies.

Furnishes payments and technical assistance to rehabilitate farmland damaged by natural disaster.

Emergency Forest Restoration Program Makes payments to restore forest land damaged by natural disaster.

Offers loans related to building or improving farm storage and handling facilities.

Grants compensation for transportation costs related to disadvantaged farm or ranch geography.

Implements voluntary practices taken to environmentally protect source water.

Grants compensation for ranching losses due to drought or fire on grazing land.

Grants compensation for livestock losses due to weather or certain predators.

Makes payments in lieu of applying for loans for which the producer is eligible.

Makes payments related to the difference in actual and threshold dairy margins.

Offers loans using commodity crops as collateral.

Makes payments related to uninsurable crops lost to natural disaster.

Offers loans to processors of domestically produced sugarcane and sugar beets.

Offers loans to construct or upgrade sugar cane and sugar beet storage facilities.

Makes payments for replanting or rehabilitating eligible trees, bushes, and vines damaged by natural disaster.

Appendix III: Comments from the Department of Agriculture

Appendix IV: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, the following staff made key contributions to this report: Colleen Phillips (assistant director), Christopher Businsky, Claudia Fletcher, Nancy Glover, Joshua Leiling, Jamelyn Payan, and Edward Varty.