Managing for Results: Challenges Agencies Face in Producing Credible Performance Information
Released: 2000-02-04T00:00:00

Letter

Background

The Results Act is aimed at improving performance of government programs by requiring agencies to clarify their missions, establish goals and strategies for reaching them, measure performance, and report on their accomplishments. Beginning with fiscal year 1999, the head of each agency is to prepare and submit to Congress and the president a report on program performance. The first of these annual reports is to be submitted no later than March 31, 2000. These reports are to contain two main parts: a report on the actual performance achieved as compared with the performance goals expressed in the performance plan and the plans and schedules to achieve those goals that were not met. If a performance goal becomes impractical or infeasible to achieve, the agency is to explain why that is the case and what legislative, regulatory, or other actions are needed to accomplish the goal, or whether the goal ought to be modified or discontinued. Finally, the reports should also relate performance measurement information to program evaluation findings, in order to give a clear picture of an agency’s performance and its efforts at improvement.

Scope and Methodology

To address our objectives, we relied on our large body of work on agencies’ performance data problems and related issues. We drew examples from our reviews of agencies’ efforts to implement the Results Act, such as our reviews of agencies’ fiscal years 1999 and 2000 performance plans, and products we have issued on major management challenges and risks, such as our performance and accountability series and high-risk series.

Because this report is based primarily on our previously issued reports, we did not obtain agency comments. Our work on this report was done from October 1999 to January 2000, in Washington, D.C., in accordance with generally accepted government auditing standards.

Challenges in Performance Planning Will Affect Performance Reporting

Agencies need reliable information during their planning efforts to set realistic goals and later, as programs are being implemented, to gauge their progress toward achievement of those goals. In our assessments of annual performance plans, we identified challenges that will affect agencies’ abilities to reliably report on the achievement of program goals and, in cases where goals are not met, either identify opportunities for improvement or whether goals need to be adjusted.

For example, we concluded in our review of fiscal year 1999 performance plans that future plans would be more useful if they would, among other things, (1) more fully articulate how strategies and resources will lead to improved performance and (2) provide much greater confidence that performance information will be credible and useful for decisionmaking. Although, on the whole, the fiscal year 2000 plans showed moderate improvements over the fiscal year 1999 plans, we identified these two issues as continuing key weaknesses common among agencies’ plans.

Few Plans Indicated How Strategies Would Contribute to Accomplishing Expected Level of Performance

Most of the fiscal year 2000 plans related strategies and programs to performance goals; however, few plans indicated how the strategies would contribute to accomplishing the expected level of performance. Agencies need to understand and articulate how what they do on a day-to-day basis contributes to mission-related results. Such an understanding is important for agencies to pinpoint opportunities to improve performance and design and implement appropriate initiatives. This information is also helpful to congressional and other decisionmakers in assessing the degree to which strategies are appropriate and reasonable.

The outcomes of many federal programs are the result of the interplay of several factors, and only some of these are within a program’s control.Further, on a daily basis, agencies do not produce outcomes, but rather outputs, such as activities, products, or services that are intended to contribute to outcomes. Thus, a key analytic challenge for agencies is knowing how their programmatic efforts contribute to their desired outcomes. The inconsistent attention to this critical element undermined the value of agencies’ performance plans and, unless addressed, it also will severely limit the value of their performance reports.

In 1997, we reported that the Department of Justice’s Immigration and Naturalization Service (INS) lacked data on the overall effectiveness of its southwest border strategy. For example, data were insufficient to indicate whether illegal aliens were deterred from entering the United States, whether there had been a decrease in attempted reentries by those who had previously been apprehended, and whether the strategy had reduced border violence. We noted that, despite the investment of billions of dollars in the strategy, INS had amassed only a partial picture of the effects of increased border control and did not know whether the investment was producing the intended results. We reported that a comprehensive, systematic evaluation of the agency’s strategy to deter illegal entry along the southwest border would provide INS with information on whether its border enforcement strategy had produced the intended results. After our report, INS contracted with independent research firms for an evaluation.

Few Plans Provided Confidence That Performance Information Would Be Credible

Another weakness that we identified in our review of agencies’ fiscal year 1999 and fiscal year 2000 performance plans was the limited confidence they provided in the credibility of performance information. Credible performance information is essential for accurately assessing agencies’ progress towards the achievement of their goals—the cornerstone of performance reporting. As shown in figure 1, our analysis of agencies’ fiscal year 2000 performance plans noted that most of the plans provided only limited confidence that performance information would be credible. Only the plans for the Department of Education, the Department of Justice, the Department of Transportation, and the Social Security Administration provided general confidence that their performance information would be credible.

Decisionmakers must have assurance that the program and financial data being used will be sufficiently timely, complete, accurate, useful, and consistent if these data are to inform decisionmaking. However, like the fiscal year 1999 performance plans, most of the fiscal year 2000 plans lacked information on the procedures the agencies would use to verify and validate performance information.

Similar to our findings with the fiscal year 1999 plans, we also found that, in general, the fiscal year 2000 plans failed to include discussions of strategies to address known data limitations. We reported that when performance data are unavailable or of low quality, a performance plan would be more useful to decisionmakers if it briefly discussed how the agency plans to deal with such limitations. Without such a discussion, decisionmakers will have difficulty determining the implications for assessing the subsequent achievement of performance goals that agencies include in their performance reports.

Agencies Face Limitations in Producing Credible Performance Data

In order to successfully measure and report progress toward intended results, agencies need to build the capacity to gather and use performance information. However, our work over the past several years has identified limitations in agencies’ abilities to produce credible performance data. Specifically, those limitations relate to program design issues that may make it difficult to collect timely and consistent national data, the relatively limited level of agencies’ program evaluation capabilities, and long-standing weaknesses in agencies’ financial management capabilities.

Program Design Features Have Implications for the Availability of Performance Information

In several program areas, devolution of program responsibility from the federal level and consolidation of individual federal programs into more comprehensive, multipurpose grant programs have shifted both program management and accountability responsibilities toward the states. These programs vary greatly in the kind and degree of flexibility afforded to state or local entities, distribution of accountability across levels of government, and availability of direct measures of program performance. In our report on grant program design features, we noted that relatively few flexible programs collected uniform data on the outcomes of state or local service activities. Collecting such data requires conditions—such as uniformity of activities, objectives, and measures—that do not exist under many flexible program designs.

For instance, we reported that the block grants enacted as part of the Omnibus Budget Reconciliation Act of 1981 carried no uniform federal information and reporting requirements. States collected a wide range of program information, but the collection efforts were designed to meet the needs of the individual states. Congress had limited information on program activities, services delivered, and clients served. As a result, it was difficult, in many cases, to aggregate state experiences and speak from a national perspective on the block grant activities or their effects. Similarly, without uniform information definitions and collection methodologies, it was difficult to compare state efforts or draw meaningful conclusions about the relative effectiveness of different strategies.

Education is one of many agencies where the interest in having enough information for accountability and federal program management continually competes with the aim of providing local agencies with the flexibility needed to implement their programs on the basis of their local needs. The Safe and Drug-Free Schools program, for example, allows a wide range of activities, such as drug prevention instruction for students; staff training; general violence-prevention instruction; and special one-time events, such as guest speakers and drug- and alcohol-free social activities.States are also permitted to define the information they collect on program activities and effectiveness. Under the Safe and Drug-Free Schools and Communities Act, Education oversees state programs and state agencies monitor local programs. Under the act, each state may establish its own reporting requirements for local education agencies. Although these requirements have some common elements, state requirements vary widely. With no requirements that states use consistent measures, Education faces, as our work has shown, a difficult challenge in assembling the required state reports to develop a nationwide picture of the program’s effectiveness.

Because states, localities, or nongovernmental organizations operate many of its programs, the Department of Health and Human Services (HHS) experiences similar challenges. The Personal Responsibility and Work Opportunity Reconciliation Act dramatically altered the nation’s system for providing assistance to the poor. Among the many changes, the act replaced the existing entitlement program for poor families (Aid to Families With Dependent Children) with fixed block grants to the states to provide Temporary Assistance for Needy Families (TANF). Under the TANF block grant, states have flexibility in designing and implementing their own assistance programs within federal guidelines. Meanwhile, HHS has a broad range of responsibilities for ensuring accountability from the states.

The welfare reform law gives HHS administrative and oversight responsibilities, the performance of which will rely on state-provided data. HHS needs to ensure that it receives comparable and reliable data from the states to help it fulfill its oversight responsibilities, which include ensuring that states enforce the federal 5-year time limit on receiving welfare benefits, meet minimum work participation rates, and maintain a certain level of state welfare spending. Enforcing the time limit, for example, will be difficult because information on the total amount of time that someone has received TANF is not always available in individual states, let alone across states. In addition, the law gives HHS authority to assess penalties if states fail to comply with certain requirements and provides for states to receive bonuses if they meet certain performance standards. HHS needs to collect state data to determine performance penalties and bonuses. In view of the increased flexibility of states in designing their programs, obtaining comparable and reliable data to assess the effect of welfare reform on children and families could be difficult for HHS.

The Environmental Protection Agency (EPA) provides another example of a federal agency that depends on the state and local agencies it is working with to provide the performance information that indicates whether results are being achieved. For example, the state water quality reports required by the Clean Water Act are a key source of information for measuring progress in cleaning up the nation’s lakes, rivers, and streams. However, EPA has found that the wealth of environmental data EPA and states collect are often difficult to compile in a meaningful way.

As contained in the Clean Water Act, Congress left the primary monitoring responsibility to the states for measuring progress in cleaning up the nation’s lakes, rivers, and streams. However, inconsistencies in water quality assessments and in assessment methodologies from state to state make it difficult to aggregate the data and to use the information to conclusively determine whether the quality of rivers, lakes, and streams is getting better or worse over time. Absent this information, it has been difficult for EPA to set priorities, evaluate the success of its programs and activities, and report on its accomplishments in a credible and informed way.

The unavailability of reliable performance information can also be traced to a lack of standards and of common definitions for terms used to evaluate programs. For example, we reported that the agencies involved in wetlands-related activities inconsistently used terms such as protection, restoration, rehabilitation, improvement, and enhancement in describing and reporting on their accomplishments. Even when the same terms are used, the agencies do not define them in the same way. As a result, the consistency and reliability of data on the status of wetlands acreage are questionable. Thus, neither the progress made toward achieving the governmentwide goal of no net loss of the nation’s remaining wetlands nor the contributions made by the agencies in achieving this goal can be accurately measured.

Weaknesses in the availability of direct measures of performance can be overcome by drawing on information from other sources, such as program evaluation studies, research on the effectiveness of service delivery, or aggregate data such as vital statistics that describe the general status of a population. In this regard, our report on grant program design features noted that 13 of the 21 flexible grant programs reviewed used such sources along with, or as a substitute for, performance measures collected from program operations. We found that agencies that made use of multiple sources had information that covered more aspects of program performance than those that relied upon a single source.

Program Evaluation Is Essential, But Federal Capacity Is Limited

Program evaluation studies are important for assessing how well programs are working, determining factors affecting performance, and identifying improvement opportunities. In our report on the analytic challenges facing agencies in measuring performance, we stated that supplementing performance data with impact evaluations might help provide agencies with a more complete picture of program effectiveness. Evaluations can play a critical role in helping to address those measurement and analysis difficulties agencies face that stem from two features common to many federal programs: the interplay of federal, state, and local government activities, and objectives and the aim to influence complex systems or phenomena whose outcomes are largely outside government control. Furthermore, systematic evaluation of how a program was implemented can provide important information about why a program did or did not succeed as well as suggest ways to improve it.

However, as we reported in our assessment of agencies’ fiscal year 1999 performance plans, we continue to be concerned about the lack in many federal agencies of the capacity to undertake the program evaluations that will be vital to the success of the Results Act. In our earlier review of agencies’ strategic plans, we found that many agencies had not given sufficient attention to how program evaluations will be used in implementing the Results Act and improving performance. In another report, we noted that agencies’ program evaluation capabilities would be challenged to meet the new demands for information on program results.We found that the resources allocated for conducting program evaluations were small and unevenly distributed across the 13 departments and 10 independent agencies we surveyed for that report.

Good evaluation information about program effects is difficult to obtain. Each of the tasks involved—measuring outcomes, ensuring the consistency and quality of data collected, establishing the causal connection between outcomes and program activities, and separating out the influence of extraneous factors—raises formidable technical or logistical problems that are not easily resolved. Thus, evaluating program impact generally requires a planned study and, often, considerable time and expense.

The experiences of the Head Start program illustrate the importance—and difficulty—of systematic program evaluation. Head Start, administered by HHS’ Administration for Children and Families, is one of the most popular federal early childhood programs and has long enjoyed both congressional and public support. Between fiscal years 1990 and 1998, annual Head Start funding nearly tripled, from $1.5 billion to almost $4.4 billion. Head Start’s purpose is to improve the social competence of children in low- income families, and in the past 33 years, the program has provided a comprehensive set of services to about 16 million low-income children. Educational, medical, nutritional, mental health, dental, social, and other services have been provided to low-income children and their families in all 50 states, the District of Columbia, Puerto Rico, and the U.S. territories, as well as to migrant and Native American populations. Given the size of the Head Start program and the efforts to expand the program’s annual enrollment to one million children by 2002, investing in studies that will assess its impact is important. Specifically, the challenge for HHS is to determine whether the same outcomes would have occurred if children and families were in other kinds of early childhood programs, or none at all.

HHS has substantially strengthened its emphasis on determining whether Head Start has achieved its purpose. In part in response to the direction of Congress, HHS has new initiatives that will, in the next few years, provide information not previously available on outcomes, such as gains made by children and their families while in the program. In addition, the program is currently designing an impact study to assess whether children and their families would have achieved these gains without participating in Head Start. Congress has required that HHS submit a final report on the impact of the Head Start program by September 30, 2003.

Financial Management Capabilities Are Needed to Support Decisionmaking and Accountability

The long-standing inability of many federal agencies to accurately record and report financial management data on both a year-end and an ongoing basis for decisionmaking and oversight purposes continues to be a serious weakness. Without reliable data on costs, decisionmakers cannot effectively evaluate programs’ financial performance or control and reduce costs. Under the Chief Financial Officers (CFO) Act, agencies are expected to develop and deploy modern financial management systems and to routinely produce sound cost and operating performance information, among other things. Further, the Federal Financial Management Improvement Act (FFMIA) focuses on ensuring greater attention to making much needed improvements in financial management systems. The primary purpose of FFMIA is to ensure that agency financial management systems routinely provide reliable, useful, and timely financial information. With such information, government leaders will be better positioned to invest scarce resources, reduce costs, oversee programs, and hold agency managers accountable for the way they run government programs.

Table 1 shows the financial statement audit results for fiscal year 1998 for the 24 CFO Act agencies.

In addition, financial management systems for 21 of the 24 agencies were found by auditors not to comply substantially with FFMIA’s requirements for fiscal year 1998. The three agencies in compliance were the Department of Energy, National Aeronautics and Space Administration, and the National Science Foundation.

For some agencies, the preparation of financial statements requires considerable reliance on ad hoc programming and analysis of data produced by inadequate financial management systems that are not integrated or reconciled, and that often require significant audit adjustments. The key for agencies is to take steps to continuously improve internal controls and underlying financial and management information systems. These systems must generate timely, accurate, and useful information on an ongoing basis. The overhauling of financial and related management information systems is the overarching challenge for agencies in generating timely, reliable data throughout the year. The following examples illustrate serious financial management weaknesses and systems problems.

While the Department of Defense (DOD) is responsible for vast operations—with an estimated $1 trillion in assets, nearly $1 trillion in liabilities, and a net cost of operations of $280 billion in fiscal year 1998— no major part of the department has been able to pass the test of an independent audit because of pervasive financial management weaknesses. Such weaknesses led us in 1995 to put DOD financial management on our list of high-risk areas vulnerable to waste, fraud, and mismanagement—a designation that continued unchanged in our more recent high-risk update. These financial management weaknesses limit the reliability and timeliness of DOD’s currently available financial information. DOD management and/or auditors have repeatedly found DOD systems to be inadequate for measuring the cost of operations and programs. For example:

DOD has acknowledged that the lack of a cost accounting system is the single largest impediment to controlling and managing weapon systems costs, including costs of acquiring, managing, and disposing of weapon systems.

DOD is unable to provide actual data on the cost associated with functions to be considered for A-76 outsourcing competitions, including the capital costs associated with operations.

DOD has long-standing problems accumulating and reporting the full costs associated with working capital fund operations, which provide goods and services in support of the military services.

As a result of our assessment of DOD’s fiscal year 2000 performance plan, we noted that the lack of adequate cost information impairs the development of cost-based performance measures and indicators across virtually the entire spectrum of DOD’s program operations. While DOD developed 43 unclassified performance measures and indicators to measure a wide variety of activities—from force levels to asset visibility— these measures and indicators contained few efficiency measures based on cost.

In our most recent testimony on DOD financial management, we reported that DOD has started to devote additional resources to correcting its financial management weaknesses. DOD’s Financial Management Improvement Plans represent an ambitious undertaking and are an important step toward long-term improvements in the department’s accountability. However, eliminating DOD’s financial management weaknesses represents a major challenge because they are pervasive and entrenched in an extremely large decentralized organization.

As another example, in January 1999, we designated the Federal Aviation Administration’s (FAA) financial management a high-risk area because of serious and long-standing accounting and financial reporting weaknesses.These weaknesses render FAA vulnerable to waste, fraud, and abuse; undermine its ability to manage its operations; and limit the reliability of financial information it provides to Congress.

Beginning with fiscal year 1994, the Department of Transportation’s Office of the Inspector General has audited FAA’s financial statements and has consistently been unable to determine whether the financial information is reliable. This pattern of negative financial audit results has continued with its most recent report—a disclaimer of opinion—on FAA’s fiscal year 1998 financial statements, citing as a primary reason the inability to verify property, plant, and equipment (PP&E) reported at a cost of $11.9 billion.

We previously reported that many problems in the PP&E accounts affect FAA’s ability to efficiently and effectively manage programs that use these assets. We also reported that many problems in these accounts result from the lack of a reliable system for accumulating project cost accounting information. The inadequacy of FAA’s cost accounting system is a weakness that prevents the agency from having reliable and timely information about the full cost of program activities. The lack of cost accounting information also limits FAA’s ability to, among other things, meaningfully evaluate performance in terms of efficiency and cost- effectiveness.

FAA senior management have indicated that they recognize the urgency of correcting their financial management deficiencies and have taken steps to address them, including efforts to continue to develop a cost accounting system, which FAA expects will be fully operational in 2001. However, as we reported in July 1999, while FAA has taken steps that are likely to lead to or already have resulted in improved accountability for FFA assets, much still remains to be done.

Performance Reports Provide Opportunities to Show Progress in Addressing Data Credibility Issues

Agencies’ March 2000 performance reports will provide them with an opportunity to show the progress they have made in addressing data credibility issues. As far back as our earliest assessment of agencies’ efforts to implement the Results Act, and more recently in our reviews of agencies’ strategic and performance plans, we identified data credibility issues as a persistent and continuing challenge for agencies. In passing the Results Act, however, Congress emphasized that the usefulness of agencies’ performance information depends, to a large degree, on the reliability and validity of their data.

During this past year, we issued several reports on practices and approaches that agencies have proposed or adopted that address data credibility issues. For example, we reported that applied practices, such as identifying actions to compensate for unavailable or low-quality data and discussing implications of data limitations for assessing performance, can help agencies describe their capacity to gather and use performance information. To illustrate, the Department of Transportation stated in its fiscal year 1999 performance plan that one of the most significant limitations of both internal and external data is timeliness. One way the department plans to deal with this limitation is to compile preliminary estimates from the portion of data that is available in time to report on the performance measures. According to the plan, fatality data from the first 6 months of the year could be compared with data from the first 6 months of the previous year for an initial performance measurement.

In our report on reasonable approaches to verify and validate performance information, we identified a wide range of possible approaches that can be organized into four general strategies, as follows:

Management can seek to improve the quality of performance data by fostering an organizational commitment and capacity for data quality.

Verification and validation can include assessing the quality of existing performance data.

Assessments of data quality are of little value unless agencies are responding to identified data limitations.

Building quality into the development of performance data may help prevent future errors and minimize the need to continually fix existing data.

These approaches can help agencies improve the quality, usefulness, and credibility of performance information. However, as noted earlier, making stakeholders aware of significant data limitations allows them to judge the data’s credibility for their intended use and to use the data in appropriate ways. All data have limitations that may hinder their use for certain purposes but still allow them to be used for others. Stakeholders may not have enough familiarity with the data to recognize the significance of their shortcomings. Therefore, appropriate use of performance data may be fostered by clearly communicating how and to what extent data limitations affect assessments of performance. For example, we noted that when the Department of the Treasury’s Customs Service field staff realized management was using performance data to make decisions, the staff began providing explanations for any incorrect data. Customs also said it required each office to establish a data quality function, responsible for verification and validation, which would be inspected annually.

Conclusion

A federal environment that focuses on results relies on new types of information that are different from those that have traditionally been collected by federal agencies. Obtaining more credible results-oriented performance information is essential for agencies to plan their efforts and gauge progress toward the achievement of their goals. However, as we previously reported, agencies have encountered some difficult analytic and technical challenges in obtaining timely and reliable results-oriented performance information and in ensuring that program evaluations that allow for the informed use of that information are undertaken.

The Results Act requires agencies to describe in their annual performance plans how they will verify and validate the performance information that will be collected. Including such information in performance reports can be equally important in helping to assure report users of the quality of the performance data. Discussing data credibility and related issues in performance reports can provide important contextual information to Congress and agencies to help them address the weaknesses in this area. For example, this sort of discussion in an agency’s performance report can alert Congress to the problems the agency has had in collecting needed results-oriented performance information. Agencies can also alert Congress to the cost and data quality trade-offs associated with various collection strategies, such as relying on sources outside the agency to provide performance data and the degree to which those data are expected to be reliable. Finally, in order to give a clear picture of the agency’s performance and its efforts at improvement, annual reports on performance can also relate performance measurement information to program evaluation findings.

We are sending copies of this report to Senator Joseph I. Lieberman, Ranking Minority Member, Senate Governmental Affairs Committee; Representative Richard A. Gephardt, Minority Leader, House of Representatives; Representative Henry A. Waxman, Ranking Minority Member, House Government Reform Committee; Representative John M. Spratt, Jr., Ranking Minority Member, House Budget Committee; and the Honorable Jacob J. Lew, Director, Office of Management and Budget. Copies will be made available to others on request.

Please contact me at (202) 512-8676 if you have any questions. Dottie Self was the key contributor to this report.

J. Christopher Mihm Associate Director, Federal Management and Workforce Issues The first copy of each GAO report and testimony is free. Additional copies are $2 each. Orders should be sent to the following address, accompanied by a check or money order made out to the Superintendent of Documents, when necessary. VISA and MasterCard credit cards are accepted, also. Orders for 100 or more copies to be mailed to a single address are discounted 25 percent.

U.S. General Accounting Office P.O. Box 37050 Washington, DC 20013 Room 1100 700 4th St. NW (corner of 4th and G Sts. NW) U.S. General Accounting Office Washington, DC Orders may also be placed by calling (202) 512-6000 or by using fax number (202) 512-6061, or TDD (202) 512-2537.

Each day, GAO issues a list of newly available reports and testimony. To receive facsimile copies of the daily list or any list from the past 30 days, please call (202) 512-6000 using a touch- tone phone. A recorded menu will provide information on how to obtain these lists.