District of Columbia: Performance Report Reflects Progress and Opportunities for Improvement
Released: 2002-04-15T00:00:00

Letter

Background

The Federal Payment Reauthorization Act of 1994 requires that the mayor of the District of Columbia submit to Congress a statement of measurable and objective performance goals for the significant activities of the District government (i.e., the performance accountability plan). After the end of the each fiscal year, the District is to report on its performance (i.e., the performance accountability report). The District’s performance report is to include a statement of the actual level of performance achieved compared to each of the goals stated in the performance accountability plan for the year, the title of the District of Columbia management employee most directly responsible for the achievement of each goal and the title of the employee’s immediate supervisor or superior, and a statement of the status of any court orders applicable to the government of the District of Columbia during the year and the steps taken by the government to comply with such orders.

The law also requires that GAO, in consultation with the director of the Office of Management and Budget, review and evaluate the District performance accountability report and submit it not later than April 15 to your committees.

Our June 2001 report on the District’s fiscal year 2000 performance accountability report included recommendations that the District (1) settle on a set of results-oriented goals that are more consistently reflected in its performance planning, reporting, and accountability efforts, (2) provide specific information in its performance reports for each goal that changed, including a description of how, when, and why the change occurred, and (3) adhere to the statutory requirement that all significant activities of the District government be addressed in subsequent performance accountability reports. Our review had determined that the District’s fiscal year 2000 report was of limited usefulness because the District had introduced new plans, goals, and measures throughout the year, the goals and measures were in a state of flux due to these changes, and its report did not cover significant activities, such as the District’s public schools, an activity that accounted for more than 15 percent of the District’s budget. In response, the District concurred with our findings and acknowledged that additional work was needed to make the District’s performance management system serve the needs of its citizens and Congress. The comments stated that the District planned, for example, to consolidate its goals and expand the coverage of its fiscal year 2001 report to more fully comply with its mandated reporting requirements.

Objectives, Scope, and Methodology

We examined the progress the District has made in developing its performance accountability report and identified areas where improvements are needed. Specifically, the objectives of this report were to examine (1) the extent to which the District’s performance accountability report includes its significant activities, (2) how well the District reports progress toward a consistent set of goals and explains any changes in the goals, (3) the extent to which the report adheres to the statutory requirements, and (4) areas for future improvement.

To meet these objectives, we reviewed and analyzed the information presented in the District’s fiscal year 2001 performance accountability report and interviewed key District officials.

To examine the extent to which the District’s performance accountability report included significant activities, we compared the information in the 2001 performance and accountability report with budget information on actual expenditures presented in the District’s budget.

To determine how well the District reported progress toward a consistent set of goals, we compared the report’s goals with those contained in the District’s fiscal year 2002 Proposed Budget and Financial Plan which served as the District’s 2001 performance plan and then reviewed any changes.

To determine the extent to which the report adhered to the statutory requirements, we analyzed the information contained in the District’s report in conjunction with the requirements contained in the Federal Payment Reauthorization Act of 1994. We also reviewed the performance contracts for the District’s cabinet-level officials.

To identify areas for future improvement, we compared the fiscal year 2001 report with the District’s fiscal year 2000 and 1999 performance accountability reports to identify baseline and trend information. We based our analysis on the information developed from work addressing our other objectives, recommendations from our June 8, 2001, report commenting on the District’s fiscal year 2000 report, and our other recent work related to performance management issues.

We conducted our work from December 2001 through April 2002 at the Office of the Mayor of the District of Columbia, Washington, D.C., in accordance with generally accepted government auditing standards. In accordance with requirements contained in P.L. 103-373, we consulted with a representative of the director of the Office of Management and Budget concerning our review. We did not verify the accuracy or reliability of the performance data included in the District’s report, including information on the court orders in effect for fiscal year 2001.

We provided a draft of this report to the mayor of the District of Columbia for review and comment. The deputy mayor/city administrator provided oral and written comments that are summarized at the end of this report, along with our response. The written comments are reprinted in their entirety in appendix III.

The 2001 Report Included Most of the District’s Significant Activities

The fiscal year 2001 performance accountability report includes most of the District’s significant activities, providing performance information for 66 District agencies that represent 83 percent of the District’s total expenditures of $5.9 billion during that year. The District included 26 additional agencies in this year’s report, compared with 40 in its prior report for fiscal year 2000. Appendix I lists the 66 agencies included in the District’s 2001 performance accountability report, along with the 2001 actual expenditures for each of these agencies. However, the absence of goals and measures related to educational activities remains the most significant gap.

The District reports that it is continuing its efforts to include performance information on its significant activities in its performance accountability reports. For example, the 2001 performance accountability report notes that the District of Columbia Public Schools (DCPS) did not include performance goals or measures because they were in the early stages of a long-term strategic planning process initiated by the newly installed school board. DCPS accounted for about 14 percent of the District’s fiscal year 2001 actual expenditures, and public charter schools, which also were not included, accounted for another 2 percent of the District’s 2001 expenditures. The 2001 report states that in lieu of a formal performance accountability report for DCPS, the District included a copy of the Superintendent’s testimony before the Subcommittee on the District of Columbia, Committee on Government Reform, U.S. House of Representatives. The District acknowledged that the inclusion of this information does not fully comply with the statutory requirement and set forth a plan to include DCPS performance goals and measures in the fiscal year 2003 proposed budget and financial plan that will serve as the basis for the DCPS performance accountability report for fiscal year 2002.

The 2001 report lists another 10 agencies that were not included, primarily, according to the report, because they did not publish performance goals and measures in the fiscal year 2002 proposed budget. These 10 agencies accounted for about $330 million in fiscal year 2001 actual expenditures, or about 6 percent of the District’s total fiscal year 2001 actual expenditures. These agencies included the Child and Family Services Agency, which was under receivership until June 15, 2001 (with fiscal year 2001 actual expenditures of $189 million) and public charter schools (with fiscal year 2001 expenditures of $137 million). Although it may not be appropriate to include agency performance information in some cases, the performance accountability report should provide a rationale for excluding them. For example, Advisory Neighborhood Commissions, according to the deputy mayor, have a wide range of agendas that cannot be captured in a single set of meaningful measures. Table 3 lists these 10 agencies and their fiscal year 2001 actual expenditures.

In addition to these 10 agencies, the District also did not specifically include other areas constituting 11 percent of the District’s fiscal year 2001 actual expenditures. In view of the District’s interest in tying resources to results, the District could further improve its performance accountability reports by linking these budget activities as appropriate to the agencies that are responsible for these expenditures or provide a rationale for exclusion. For example, the Department of Employment Services administers the unemployment and disability funds (with fiscal year 2001 expenditures totaling about $32 million). Similarly, the Office of the Corporation Counsel administers the settlement and judgments fund, which was set up to settle claims and lawsuits and pay judgments in tort cases entered against the District (with fiscal year 2001 expenditures of about $26 million). Table 4 contains a list of these budget activities and fiscal year 2001 actual expenditures.

The District’s 2001 Plan and Report Addressed a Consistent Set of Performance Goals

The goals in the fiscal year 2001 performance accountability report were consistent with the goals in the District’s 2001 performance plan. Using a consistent set of goals enhanced the understandability of the report by demonstrating how performance measured throughout the year contributed toward achieving the District’s goals. The District also used clear criteria for rating performance on a five-point scale and reported that these ratings were included in the performance evaluations of cabinet agency directors who had performance contracts with the mayor. In addition, according to a District official, the District will be able to provide information on any future changes made to its performance goals through its new performance management database.

The District has made substantial progress in improving its performance planning and reporting efforts by focusing on measuring progress toward achieving a consistent set of goals. In our June 2001 review of the District’s 2000 performance accountability report, we had raised concerns that the District’s performance management process was in flux, with goals changing continually throughout the year. Further, the District did not discuss the reasons for these changes. This year, the goals were consistent and the District provided some information about upcoming changes that could be anticipated in fiscal year 2002 goals. In addition, according to the 2001 report, the District has developed a performance measures database to allow it to document changes to individual goals and measures that are proposed in the agencies’ fiscal year 2003 budget submissions.

One of the District’s enhancements to its 2001 performance accountability report was reporting on a five-point performance rating scale, as compared to the three-point performance rating scale it used in its fiscal year 2000 report. The five-point scale was designed to be consistent with the rating scale used in the District’s Performance Management Program, under which management supervisory service, excepted service, and selected career service personnel develop individual performance plans against which they are evaluated at the end of the year. The five ratings are: (1) below expectations, (2) needs improvement, (3) meets expectations, (4) exceeds expectations, and (5) significantly exceeds expectations. According to the fiscal year 2001 performance accountability report, this scale was used to evaluate the performance of cabinet agency directors who held performance contracts with the mayor. It stated that 60-percent of each director’s performance rating was based on the agency-specific goals included in the agency’s performance accountability report, with the other 40-percent based on operational support requirements such as responsiveness to customers, risk management, and local business contracting. Our work has found that performance agreements can become an increasingly vital part of overall efforts to improve programmatic performance and better achieve results. We found that the use of results-oriented performance agreements: strengthened alignment of results-oriented goals with daily operations, fostered collaboration across organizational boundaries, enhanced opportunities to discuss and routinely use performance information to make program improvements, provided a results-oriented basis for individual accountability, and maintained continuity of program goals during leadership transitions.

The Report Generally Adhered to Statutory Requirements

The District’s fiscal year 2001 performance accountability report reflected improvement in adhering to the statutory requirements in the Federal Payment Reauthorization Act. The District’s 2001 report was timely and included information on the level of performance achieved for most goals listed. It included the titles of the District management employee most directly responsible for the achievement of each of the goals and the title of that employee’s immediate supervisor, as required by the statute. We also found that the names and titles on the performance contracts of the cabinet level officials we reviewed matched the names in the performance report as the immediate supervisor for all of the goals. Although the report contains information on certain court orders, the report could be improved by providing clearer and more complete information on the steps the District government has taken during the reporting year to comply with those orders and by including updated information on the court orders applicable to the District as required by the act.

The Report Identified Performance Levels Achieved toward Most of the District’s Goals

The District identified the level of performance achieved for most of the goals in its 2001 report. The report contains a total of 214 performance goals that are associated with the 66 agencies covered. Of these 214 performance goals, 201 goals (or 94 percent) include information on whether or not the goal was achieved, and only 13 did not include information on the level of performance.

As shown in table 1, the 13 goals that did not include the level of performance were associated with eight agencies. For example, the District’s State Education Office did not provide this information for four of its seven goals because the reports and information needed to achieve the goals had not been completed.

Information the District Included on Court Orders Has Limitations

Although the District’s 2001 performance accountability report included some information on certain court orders imposed upon the District and the status of its compliance with those orders, the act calls for a statement of the status of any court orders applicable to the District of Columbia government during the year and the steps taken by the government to comply with such orders. The 2001 report contains information on the same 12 court orders involving civil actions against the District reported on for fiscal years 1999 and 2000. Among these 12 orders are 2 orders that the fiscal year 2001 report lists as no longer in effect in 2001. One of these court orders involved a receivership that terminated in May 2000. The other involved a maximum-security facility that closed at the end of January 2001. The 2001 report does not disclose whether or not any new court orders were imposed on the District during fiscal year 2001.

The summaries that the District provides on the status of these court orders could be more informative if they contained clearer and more complete information on the steps taken by the District government to comply with the court orders. For example, according to the District’s 2001 report, the case Nikita Petties v. DC relates to DCPS transportation services to special education students and the timely payment of tuition and related services to schools and providers. The report’s summary on the status of this case states: “The School system has resumed most of the transportation responsibilities previously performed by a private contractor. A transportation Administrator with broad powers had been appointed to coordinate compliance with Court orders. He has completed his appointment and this position has been abolished.” This summary does not provide a clear picture of what steps the school system is taking to comply with the requirements resulting from this court order. The act, however, calls for the District to report on the steps taken by the government to comply with such orders.

Steps Are Needed to Improve Future Performance Accountability Reports

The District recognized in its 2001 performance and accountability report that its performance management system is a work-in-progress and stated that there are several fronts on which improvements can be made. In the spirit of building on the progress that the District has made in improving its performance accountability reports over the last 2 years, there are three key areas where we believe that improvements in future performance accountability reports are needed. First, the District needs to be more inclusive in reporting on court orders to more fully comply with the act’s requirements. Second, as part of the District’s emphasis on expanding its performance-based budgeting approach, the District needs to validate and verify the performance data it relies on to measure performance and assess progress, present this information in its performance accountability reports, and describe its strategies to address any known data limitations. Finally, the District needs to continue its efforts to include goals and measures for its major activities, and it should include related expenditure information to provide a more complete picture of the resources targeted toward achieving an agency’s goals and therefore help to enhance transparency and accountability.

The District Should Be More Inclusive in Reporting on Court Orders

Since this is the third year that the District has had to develop performance and accountability reports, the District has had sufficient time to determine how best to present information on the status of any court orders that are applicable to the District of Columbia during the fiscal year and the steps taken to comply with those orders. However, the District has continued to report on the same 12 court orders for fiscal years 1999, 2000, and 2001.

By limiting its presentation to the same 12 court orders, the District’s current report does not provide assurance that the information in its performance accountability report reflects court orders applicable during the fiscal year. Court orders have an important effect on the District’s performance, as reflected by the chief financial officer’s statement that the District’s “unforeseen expenses are often driven by new legislative imperatives, court-ordered mandates, and suits and settlements.” As another indication of their importance, 1 of the 11 general clauses in performance contracts with agency directors addresses the directors’ responsiveness to court orders.

To make future reports more useful, the District should include information on the status of court orders it has not previously reported on as well as those applicable during the fiscal year, including those that may have been vacated during the fiscal year and the steps taken to comply with them. The District should establish objective criteria for determining the types of court orders for which it will provide specific compliance information for future performance accountability reports, and it should consider ways to provide summary information related to any other court orders. In establishing objective criteria, the factors could include the cost, time, and magnitude of effort involved in complying with a court order. If the District government has not acted to comply with a court order it should include an explanation as to why no action was taken.

The District’s 2001 report contains a statement that “Following the publication of the FY 1999 Performance Accountability Report, GAO and the District’s Office of Corporation Counsel agreed upon a list of 12 qualifying orders that should be included in the District’s future Performance Accountability Reports.” We did not intend to limit future reporting to only the 12 court orders first reported by the District for fiscal year 1999. We agreed on the list of 12 court orders because, at that time, the District had difficulty identifying all the court orders as required by statute. However, we believe that the District now has had time to develop criteria and a system for ensuring that updated and accurate information on the status of applicable court orders can be presented in its future performance accountability reports. Therefore, we are recommending that the mayor ensure that such steps are taken.

The District Faces Challenges in Verifying and Validating Its Performance Information

The District has identified data collection standards as one of the areas it is working to improve. As with federal agencies, one of the biggest challenges the District faces is developing performance reports with reliable information to assess whether goals are being met or how performance can be improved. Data must be verified and validated to ensure the performance measures used are complete, accurate, consistent, and of sufficient quality to document performance and support decision making. Data verification and validation are key steps in assessing whether the measures are timely, reliable, and adequately represent actual performance. The District’s performance and accountability reports should include information obtained from verification and validation efforts and should discuss strategies to address known data limitations.

As reported in our June 2001 report on the District’s fiscal year 2000 performance accountability report, the District had planned to issue performance review guidelines by the end of the summer of 2001. These guidelines were to be issued in response to an Inspector General’s finding that the agencies did not maintain records and other supporting documentation for the accomplishments they reported regarding the fiscal year 2000 performance contracts. The District included information in its fiscal year 2003 budget instructions regarding performance measures emphasizing the importance of high quality data. Although not required for agencies’ budget submissions, the guidance called for every agency to maintain, at a minimum, documentation on how it calculated each measure and the data source for each measure.

In its 2001 performance accountability report, the District said it plans to address the development of data collection standards. The District plans to begin developing manuals to document how data for each performance measure is collected, how the measure is calculated, and who is responsible for collecting, analyzing, and reporting the data. A further step the District can consider is ensuring that these data are independently verified and validated. A District official acknowledged that validating and verifying performance information is something the District would deal with in the future. Credible performance information is essential for accurately assessing agencies’ progress toward the achievement of their goals and pinpointing specific solutions to performance shortfalls. Agencies also need reliable information during their planning efforts to set realistic goals.

Decision makers must have reliable and timely performance and financial information to ensure adequate accountability, manage for results, and make timely and well-informed judgments. Data limitations should also be documented and disclosed. Without reliable information on costs, for example, decision makers cannot effectively control and reduce costs, assess performance, and evaluate programs. Toward that end, the District must ensure that its new financial management system is effectively implemented to produce crucial financial information, such as the cost of services at the program level, on a timely and reliable basis.

The District Should Enhance Its Efforts to Include Goals, Measures, and Related Expenditure Information

Although the District has made progress in presenting program performance goals and measures, the 2001 report did not contain goals and measures for all of its major activities and it did not include information on other areas that accounted for 11 percent of its annual expenditures. The District could enhance the transparency and accountability of its reports by continuing its efforts to ensure that agencies establish goals and measures that they will use to track performance during the year and by taking steps to ensure that agencies responsible for other budget activities (as shown in table 4) include these areas in their performance reports.

The District did not include, for example, goals and measures for DCPS, although it did provide a copy of a testimony and stated that this was included, at least in part, to address concerns we had raised in our June 2001 report that the District’s fiscal year 2000 performance accountability report did not cover DCPS. The District also did not include another 10 agencies in its 2001 performance accountability report and indicated that it is taking steps to include relevant goals and measures for some of these agencies in the next year’s report.

In addition to including goals and measures for the District’s significant activities, the District should consider including related expenditure information to help ensure transparency and accountability. We found, for example, that the Department of Employment Services administers the unemployment and disability funds but this information was not linked in the District’s 2001 performance accountability report. By linking expenditures to agencies that are responsible for them, the District can further improve its future performance accountability reports by providing a more complete picture of performance.

Conclusions

The District, like several federal agencies, has found that it needed to change its performance goals—in some cases substantially—as it learned and gained experience during the early years of its performance measurement efforts. The District has continued to make progress in implementing a more results-oriented approach to management and accountability and issuing a timely and more complete performance accountability report. As we have seen with federal agencies, cultural transformations do not come quickly or easily, and improvements in the District’s performance management system are still underway. Despite the important progress that has been made, opportunities exist for the District to strengthen its efforts as it moves forward.

Recommendations

In order to more fully comply with the Federal Payment Reauthorization Act of 1994, which requires the District to provide a statement of the status of any court orders applicable to the government of the District of Columbia during the year and the steps taken by the government to comply with such orders, the mayor should ensure that the District establish objective criteria to determine the types of court orders for which it will provide specific compliance information for future performance accountability reports. In establishing objective criteria, the factors could include the cost, time, and magnitude of effort involved in complying with these court orders. If the District government has not acted to comply with the court orders it should include an explanation as to why no action was taken. In addition, the District should provide summary information related to other applicable court orders in its performance accountability reports.

The Mayor of the District of Columbia should also ensure that future performance accountability reports include information on the extent to which its performance measures and data have been verified and validated and discuss strategies to address known data limitations, and include goals and performance measures for the District’s significant activities and link related expenditure information to help ensure transparency and accountability.

Agency Comments and Our Evaluation

On April 2, 2002, we provided a draft of our report to the mayor of the District of Columbia for his review. In response to our request, the deputy mayor/city administrator met with us on April 4 to discuss the draft and provided us with written comments on April 8. His written comments appear in appendix III.

Overall, the deputy mayor stated that he agreed with the findings of the report and concurred with the report’s recommendations. He stated that clear and meaningful performance reports are essential to communicate the extent to which the District has or has not met its goals and commitments to make those improvements. Further, he stated that the findings and recommendations in this report were consistent with the District government’s intent of further improving its public reporting.

The deputy mayor stated that the District would adopt our recommendation to develop objective criteria to determine the types of court orders for which it will provide specific compliance information for future performance accountability reports. Our recommendation also stated that the District should more fully comply with the statute by reporting information on the steps taken by the District government to comply with these orders. The deputy mayor said that they would provide such additional information although he stated that the statute does not specifically require that this information be provided. However, the Federal Payment Reauthorization Act of 1994 (P.L. 103-373) section 456(b)(C) requires that the District’s performance accountability report contain “a statement of the status of any court orders applicable to the government of the District of Columbia during the year and the steps taken by the government to comply with such orders.” We encourage the District government to comply with this requirement and concur with its comment that providing this information would make the report more informative and useful to Congress and the general public.

The deputy mayor also concurred with our recommendation that the District’s future performance reports include information on the extent to which its performance data have been validated and verified. The deputy mayor said that seven District agencies participating in the District’s performance based budgeting pilot would be developing data collection manuals this summer. We encourage the District to proceed with this effort as well as to develop and report on strategies for addressing limitations in its data collections efforts. We have suggested in prior reports that when federal agencies have low quality or unavailable performance data, they should discuss how they plan to deal with such limitations in their performance plans and reports. Assessments of data quality do not lead to improved data for accountability and program management unless steps are taken to respond to the data limitations that are identified. In addition, alerting decisionmakers and stakeholders to significant data limitations allows them to judge the data’s credibility for their intended use and to use the data in appropriate ways.

Regarding the independent verification of performance data, the deputy mayor stated that the District's ability to secure independent verification of more than selected goals and measures is limited by the resources available to the District's Office of the Inspector General (OIG). He said that the OIG conducted spot-check audits of selected scorecard goals in the fiscal year 2000 performance accountability report and although these limited audits allowed the District to determine the validity of only those particular measures, this effort provided valuable observations and suggestions on how District agencies could improve its data collection practices. He also said that his office has discussed initiating additional spot-check audits of selected goals and measures with the OIG during fiscal year 2002. We agree that such spot checks would be useful. The knowledge that the OIG will be spot-checking some performance data during each fiscal year provides a good incentive to develop and use accurate, high-quality data. In our prior work, we have encouraged federal agencies to use a variety of strategies to verify and validate their performance information, depending upon the unique characteristics of their programs, stakeholder concerns, performance measures, and data resources. In addition to relying on inspector general assessments of data systems and performance measures, the District can use feedback from data users and external stakeholders to help ensure that measures are valid for their intended use. Other approaches can include taking steps to comply with quality standards established by professional organizations and/or using technical or peer review panels to ensure that performance data meet quality specifications. The District can also test the accuracy of its performance data by comparing it with other sources of similar data, such as data obtained from external studies, prior research, and program evaluations.

The deputy mayor said that the District would be making efforts to include additional agencies and budget activities in future performance reports. We encourage the District to proceed with these efforts. Of the 10 agencies that were not included in the fiscal year 2001 performance report, the District has already included 3 agencies (the Office of Asian and Pacific Islander Affairs, the Child and Family Services Agency, and the Office of Veteran Affairs) in its fiscal year 2002 performance plan issued in March 2002. In addition, the deputy mayor stated that three additional agencies (the Office of the Secretary, the Housing Finance Agency, and the National Capital Revitalization Corporation) would be included in the District’s consensus budget to be submitted to the Council of the District of Columbia in June 2002.

With regard to the budget activities that were not included in the District’s fiscal year 2001 performance report, the deputy mayor agreed that it would be appropriate to develop performance measures for six funds, such as settlements and judgments and administration of the disability compensation fund. The deputy mayor acknowledged that establishing performance measures for administering an additional six funds, such as the Public Benefit Corporation, would have been appropriate but they no longer exist. The deputy mayor said that the District of Columbia Retirement Board manages two funds that had relevant performance measures in the District’s 2001 report. We noted, however, that these two retirement funds were not specifically identified in the 2001 performance accountability report.

We are sending copies of this report to the Honorable Anthony A. Williams, Mayor of the District of Columbia. We will make copies available to others upon request.

Key contributors to this report were Katherine Cunningham, Steven Lozano, Sylvia Shanks, and Susan Ragland. Please contact me or Ms. Ragland on (202) 512-6806 if you have any questions on the material in this report.

Expenditures by Agencies Included in the District’s Fiscal Year 2001 Performance Accountability Report

The District’s fiscal year 2001 performance accountability report included 66 agencies accounting for 83 percent of the District’s operating budget for fiscal year 2001. Table 2 lists these agencies and their fiscal year 2001 actual expenditures.

Agencies and Budget Activities Not Included in the District’s Fiscal Year 2001 Performance Accountability Report

The District’s fiscal year 2001 performance accountability report did not include 10 District agencies primarily because they did not publish performance goals in the District’s 2001 performance plan. Table 3 lists these agencies and their fiscal year 2001 actual expenditures.

In addition to these 10 agencies, we identified several budget activities— accounting for 11 percent of the District’s total fiscal year 2001 actual expenditures—that were not included in the fiscal year 2001 performance accountability report. Table 4 lists these activities and related fiscal year 2001 actual expenditures.