Managing for Results: Experiences of Selected Credit Programs
Published: 1998-02-19T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: GAO reviewed major credit agencies' efforts to implement the Government Performance and Results Act of 1993 (GPRA), focusing on: (1) goals and measures established by the selected credit programs that related to the programs' intended purposes; (2) whether the programs had set target levels of performance for assessing their progress in achieving their desired results; (3) the challenges agency officials cited in developing performance information, including goals and measures, for the selected programs and any approaches those programs were taking to address those challenges; and (4) the status of the Federal Credit Policy Working Group's effort to develop common performance measures for federal credit programs.
section_title: What GAO Found
paragraphs: GAO noted that: (1) in their efforts to implement GPRA, the five credit programs established goals and performance measures that appeared to be generally related to the programs' intended purposes; (2) if the selected programs collect accurate corresponding data on their actual performance, they should be able to monitor their progress in achieving desired results on those measures and have fiscal year (FY) 1998 baseline data to use in setting future targets for those measures; (3) although the selected programs have established goals, measures, and targets in their efforts to implement GPRA, GAO identified three general challenges the programs have been facing in developing performance information; (4) according to the Office of Management and Budget (OMB) and the Working Group, comparing results using common measures across credit programs allows program managers and other decisionmakers to identify best practices among those programs that have the potential for improving other credit programs' performance; (5) two general problems have limited the Working Group's progress in developing common performance measures for credit programs; (6) the Working Group anticipated that agencies that administer credit programs could include common financial and programmatic measures in their annual performance plans and reports under the Results Act; (7) however, OMB does not intend to require credit agencies to adopt common performance measures when consensus about the appropriateness of such measures has not been achieved; (8) GAO agreed that OMB should not force the use of common measures when concerns about their appropriateness exist, but the Working Group had not resolved those concerns and had not decided how and when those concerns would be addressed; (9) thus, it is unclear whether OMB and the credit agencies will maintain their current level of attention to developing common measures; (10) also unclear is the extent to which agencies that administer credit programs will include common measures for those programs in their annual performance plans that could provide useful information to decisionmakers interested in making performance and cost comparisons; and (11) GAO believes the potential benefits that could be realized from developing common performance measures, where appropriate, underscore the importance of OMB and the credit agencies continuing their efforts to develop and reach consensus on such measures.

section_title: Letter
paragraphs: The Government Performance and Results Act of 1993, also referred to as “GPRA” or “the Results Act,” seeks to improve the efficiency, effectiveness, and accountability of federal programs by establishing a system to set goals for programs based on their intended purposes, measure the performance of those programs, and use performance information to improve results. As you know, if successfully implemented, the Results Act will help agencies focus on how to improve their programs’ performance in achieving desired results.
The Act requires agencies to establish goals, performance measures, and target levels of performance for fiscal year 1999. However, since the passage of the Act, many agencies have been preparing for its governmentwide implementation, beginning in the fall of 1997. In June 1997, under the Results Act requirement that we report on agencies’ implementation of the Act, we concluded that governmentwide implementation would be uneven. That report included a brief discussion of federal credit programs’ experiences in developing goals and measures under the Results Act.
This report provides additional information and context on major credit agencies’ efforts to implement the Act. It also describes the efforts of the Federal Credit Policy Working Group—an interagency advisory group on credit management policy—to develop common performance measures for credit programs. In July 1995, the Working Group established a task force to develop and enhance credit program performance measures. The report is addressed to you because of the Office of Management and Budget’s (OMB) responsibility for overseeing agencies’ implementation of the Act and because the Deputy Director of Management at OMB chairs the Working Group.
Federal credit, or loan, programs are generally intended to provide financial assistance to borrowers that the private sector will not serve. In doing so, these programs are typically expected to help achieve certain social and economic results, such as reaching underserved populations and neighborhoods and supporting investment that is important to the economy. As reported in the fiscal year 1998 budget, at the end of fiscal year 1996, the outstanding face value of credit programs had reached almost $1 trillion in direct loans and loan guarantees.
For this report, our first objective was to identify goals and measures established by the selected credit programs that related to the programs’ intended purposes and to determine whether the programs had set target levels of performance for assessing their progress in achieving their desired results. Under the Results Act, target levels of performance are to enable decisionmakers to compare planned versus actual results achieved for a given year. Our second objective was to identify the challenges agency officials cited in developing performance information, including goals and measures, for the selected programs and any approaches those programs were taking to address those challenges. Our third objective was to describe the status of the Working Group’s effort to develop common performance measures for federal credit programs.
We selected for our review credit programs that varied in terms of type of program (e.g., housing and education loans); mode of credit delivery (e.g., direct and guaranteed loans); and program size as measured by the amount of outstanding loans. These programs were the Department of Veterans Affairs’ (VA) Loan Guaranty Program, which provides home loans for veterans and active duty personnel; the Department of Education’s William D. Ford Direct Loan Program and its Federal Family Education Loan Program (referred to in this report as Education’s direct and guaranteed student loan programs, respectively), which issue or guarantee loans, respectively, for students enrolled in postsecondary education; and the Department of Agriculture’s (USDA) Single-Family Housing (SFH) direct loan program and guaranteed loan program, which make or guarantee loans to low- and moderate-income rural families, respectively, who are unable to obtain credit from other sources.
According to data reported by OMB in the fiscal year 1998 budget and agency data, of the total amount of federal credit outstanding in fiscal year 1996 for guaranteed loans ($805 billion) and direct loans ($165 billion), these five programs represented about 32 and 18 percent, respectively.
subsections: section_title: Background
paragraphs: In the 1990s, Congress put in place a statutory framework to address the long-standing weaknesses in federal operations, improve federal management practices, and provide greater accountability for achieving results. This framework included as its essential elements the Results Act and key financial management and information technology reform legislation: the Chief Financial Officers Act of 1990 (CFO)—as expanded by the Government Management Reform Act of 1994 (GMRA)—and the Paperwork Reduction Act of 1995 and the Clinger-Cohen Act of 1996, respectively. Taken together, these legislative initiatives seek to respond to a need for accurate, reliable, and integrated budget, financial, and program information for congressional and executive branch decisionmaking, information that much of our prior work has shown to be badly lacking.
The goal-setting and performance measurement and improvement system envisioned by the Results Act is the centerpiece of this framework and starts with the requirement that each executive agency develop and periodically update a strategic plan to lay out its mission, long-term goals and objectives, and strategies for achieving those goals and objectives. Under the Results Act, the first of these plans were due by September 30, 1997. Next, each agency is to develop an annual performance plan, beginning with the agency’s plan for fiscal year 1999, which covers each program activity set forth in the agency’s budget. Agencies were to submit their fiscal year 1999 annual performance plans to OMB in the fall of 1997 with their fiscal year 1999 budget requests and are to submit those plans to Congress after the President’s fiscal year 1999 budget is provided to Congress in February 1998. Among other things, agencies’ annual performance plans are to contain their programs’ goals and measures for fiscal year 1999. Finally, each agency is to report publicly on its programs’ performance, specifically on the degree to which the goals that are laid out in the agency’s annual performance plan are being met and on actions it plans to take to achieve unmet goals. The first of these reports, on programs’ performance for fiscal year 1999, is due by March 31, 2000; subsequent reports are due by March 31 for the years that follow. (For a detailed description of the Results Act’s requirements, see app. I.)
The CFO Act and GMRA are intended to strengthen the reliability of agencies’ financial and programmatic performance information and the reporting of such information by, among other things, having agencies develop better performance measures and cost information and design results-oriented reports that integrate budget, accounting, and program information.Finally, the Paperwork Reduction Act and Clinger-Cohen Act seek to help agencies address long-standing weaknesses in their use of information technology. Under this legislation, each agency is to better link its technology plan and information technology use to achieving the agency’s desired results.
In addition, long-standing concerns about the program and financial management of credit programs have prompted Congress to enact important budget and credit management reform initiatives over the last 15 years. These initiatives include the Debt Collection Act of 1982 (DCA) and amendments, the Debt Collection Improvement Act of 1996 (DCIA), and the Federal Credit Reform Act of 1990 (FCRA). DCA and DCIA are significant pieces of credit management legislation designed to, among other things, facilitate federal efforts to decrease delinquencies and increase collections. Under DCA, agencies were to annually report to the Director of OMB and the Secretary of the Treasury on the status of their debt collection activities. Under DCIA, agencies are now to annually report on those activities only to the Secretary of the Treasury, who in turn is to annually report to Congress, beginning no later than 1999, on such activities governmentwide. FCRA changed the budget treatment of direct loans and loan guarantees made on or after October 1, 1991, to (1) facilitate more accurate reporting by credit agencies on the full cost to the government in the budget for the year in which the programs made or guaranteed the loans so that executive branch and congressional decisionmakers might consider such costs when making budget decisions and (2) permit appropriate cost comparisons between direct and guaranteed credit programs and between credit and noncredit programs intended to achieve similar purposes.
Guided by these legislative initiatives, since 1992, OMB has encouraged federal credit agencies to improve their credit programs’ financial and programmatic performance measures and to adopt a set of common performance measures for those programs. According to OMB and the Federal Credit Policy Working Group, common measures should help credit program managers and other decisionmakers assess how similar functions are performed and promote an atmosphere of cooperation and a sharing of ideas among agency officials on how to improve the performance of credit programs.
In response to the increasing importance being placed on agencies’ integration of performance measurement with budgeting, management improvement, and overall agency accountability, the Working Group has focused since July 1995 on developing common performance measures for credit programs. To do so, the Working Group established a task force to develop measures for credit programs consistent with the Results Act and credit and financial management reform initiatives. In August 1997 we reported that agencies’ annual planning under the Results Act could be used as a vehicle for developing, where appropriate, common performance measures for permitting future comparisons of similar programs’ results and the methods those programs used to achieve those results.
section_title: Scope and Methodology
paragraphs: To meet our first objective—to identify goals and measures established by the selected credit programs that related to the programs’ intended purposes and determine whether the programs had set target levels of performance for assessing their progress in achieving their desired results—we compared the programs’ goals and measures to their respective intended purposes, as identified by the programs or their respective agencies. We interviewed agency officials about the programs’ intended purposes and asked those officials to comment on the relationships we identified between the programs’ goals and measures and intended purposes. To determine whether the programs had set target levels of performance for assessing their progress in achieving their desired results, we identified those measures for which the programs had either (1) identified fiscal year 1998 targets or (2) reported prior year baseline data for those measures and indicated how performance on those measures was to change (i.e., increase or decrease) in fiscal year 1998 relative to the baseline.
The programs’ goals and performance measures that we used for our assessment were those established by the programs as of May 1, 1997, which, according to agency officials, were generally the same ones the programs submitted with their respective agencies’ fiscal year 1998 budget presentations to OMB and Congress. The programs also were proposing to include these goals and measures for their agencies’ fiscal year 1999 annual performance plans under the Results Act. Although we make observations about differences in goals and measures among the programs, our review did not address the reasonableness of the processes or methods the programs used to determine how to assess progress or establish target levels toward achieving the programs’ intended purposes; determine whether other, more appropriate measures existed; or evaluate the feasibility of the targets the programs established.
To meet our second objective—to identify the challenges agency officials cited in developing performance information, including goals and measures, for the selected programs and any approaches those programs were taking to address those challenges—we asked agency officials responsible for and involved in the development of goals and measures for those programs to rate how great a challenge it was to perform each of 49 activities we identified as associated with developing performance information. Examples of these activities included “determining a realistic target level of performance for annual performance goals” and “developing measures for assessing the net effect of the program compared with what would have occurred in the absence of the program.” To identify these activities, we referred to key steps and practices identified in our Executive Guide: Effectively Implementing the Government Performance and Results Act and other work that we had under way assessing the challenges agencies were facing in implementing performance measurement.
We asked officials to use a five-point scale to rate the 49 activities, ranging from “little to no” challenge to a “very great” challenge for the selected credit programs in their agencies. For purposes of this report, we refer to activities that any of the agency officials rated as a “great” or “very great” challenge as significantly challenging activities. We then interviewed agency officials about why they rated certain activities associated with developing performance information as significantly challenging. We analyzed their responses and related documentation to identify general challenges that led agency officials to report those activities as significantly challenging for the selected programs. To identify any approaches those programs were taking to address the challenges they have been facing, we talked with agency officials about such approaches and analyzed agency documentation. We also considered prior and ongoing work we have done on the efforts of VA, Education, USDA, and other credit agencies to implement various credit and financial management reforms.
To meet our third objective—to describe the status of the Working Group’s effort to develop common performance measures for federal credit programs—we reviewed various documentation from OMB and members of the Working Group that described this effort and agency officials’ views about those measures. We also talked to OMB officials and members of the Working Group, including agency officials at those agencies administering the programs we selected for our review, to obtain their views of the common performance measures proposed. Our review did not address the reasonableness of the processes or methods the Working Group’s task force used in determining how to assess progress for federal credit programs or determine whether other measures existed that may be more appropriate. (For a more detailed discussion on our objectives, scope, and methodology, see app. II.)
We did our work from September 1996 to December 1997, in Washington, D.C., in accordance with generally accepted government auditing standards. We requested comments on a draft of this report from the Director of OMB, the Secretaries of Education and Agriculture, and the Acting Secretary of VA or their designees. Comments were provided orally by designees of the four agencies, and those comments are discussed at the end of this report.
section_title: Selected Credit Programs Established Goals, Measures, and Targets to Monitor Their Progress
paragraphs: In their efforts to implement the Results Act, the five credit programs established goals and performance measures that appeared to be generally related to the programs’ intended purposes. In some cases, the programs established goals or measures that addressed intermediate results that the programs expected to lead to their intended purposes. Also, some of the selected programs established a range of measures that should provide a more complete picture of particular aspects of their performance related to the programs’ intended purposes. Finally, the selected programs also set fiscal year 1998 targets for most of their measures.
subsections: section_title: Goals and Performance Measures Generally Related to the Programs’ Intended Purposes
paragraphs: In their fiscal year 1998 budget presentations, each of the five credit programs established goals and performance measures that appeared to be generally related to the programs’ intended purposes. For example, to monitor its performance in achieving one of its intended purposes, namely helping veterans retain their homes, VA’s Loan Guaranty Program established the Foreclosure Avoidance Through Servicing (FATS) Ratio measure. According to VA, the FATS ratio is to provide data on the extent to which foreclosures would have been greater had VA not pursued alternatives to foreclosure, such as intervening with the holder of the loan on behalf of the borrower to set up a repayment plan. For each of the selected credit programs, appendix III shows the intended purposes identified by those programs and their respective agencies and examples of related goals and performance measures the programs established in their fiscal year 1998 budget presentations.
In some cases, the programs’ goals or performance measures addressed intermediate results that the programs expected to lead to an intended purpose. For example, the performance measure that VA’s Loan Guaranty Program established to monitor its performance in achieving its intended purpose of helping veterans purchase homes was the percentage of respondents to VA’s Lender Customer Satisfaction Survey who say they are satisfied with their overall interaction with VA. According to VA, “maximizing lender satisfaction with their dealings with VA employees . . . will encourage lenders to participate in the program, expanding financing opportunities for veterans.” In this way, the intermediate result of increasing lender satisfaction could be expected to contribute to helping veterans and active duty personnel purchase homes.
Similarly, Education’s direct student loan program established as performance measures the rate of (1) institutional (i.e., school) participation; (2) overall satisfaction of schools with the direct student loan program; and (3) institutional retention in the program. The program uses schools as the vehicles for providing loans to students and their families. By providing a streamlined loan delivery system, the program expects to attract schools to participate in the program. Further, by satisfying participating schools, the program expects to encourage those schools to stay as participants in the program. Thus, increases in schools’ participation, satisfaction, and retention in the program are intermediate results that the program expects will lead to broader student access to capital for postsecondary education.
Some of the selected programs established a range of measures to provide a more complete picture of particular aspects of their performance related to their intended purposes. For example, VA’s Loan Guaranty Program established two performance measures for monitoring the timeliness of issuing a certificate of eligibility, which is related to its intended purpose of treating all veterans in a timely manner. These measures were (1) the percentage of veterans responding to a VA veteran survey who say they are satisfied with the time it takes VA to certify veterans’ eligibility for a home loan; and (2) the average time VA calculated it took to issue a certificate of eligibility, which is to supplement the survey data. Similarly, Education established several performance measures for assessing its guaranteed and direct student loan programs’ performance toward successfully managing the programs in a cost-effective manner. These measures included the programs’ lifetime gross dollar default rates; lifetime net default rates (i.e., loss rate); annual delinquency rates; per unit administrative costs; and the annual collection rates.
section_title: The Credit Programs Set Fiscal Year 1998 Targets for Most of Their Measures
paragraphs: The Results Act defines a performance goal as the target level of performance expressed as a tangible, measurable objective against which achievement is to be compared. Thus, annual performance goals should consist of two parts: (1) the performance measure that represents the specific characteristic of the program that the program uses to gauge its performance and (2) the annual target level of performance to be achieved during a given fiscal year for the measure. The Results Act also requires each agency to report to the President and Congress annually, beginning for fiscal year 1999, on the degree to which the agency is meeting its annual performance goals. Thus, under the Act, an agency is to monitor and report on its actual performance during the year compared to the targets it had established for its performance measures for that year.
As shown in table 1, we found that the selected programs set fiscal year 1998 targets for most of their respective measures. These included measures for which the programs either (1) set fiscal year 1998 target levels of performance; or (2) reported prior year baseline data for those measures and indicated how performance on those measures was to change (i.e., increase or decrease) in fiscal year 1998 relative to their baselines. Thus, if the selected programs collect accurate corresponding data on their actual performance, they should be able to monitor their progress in achieving desired results on those measures and have fiscal year 1998 baseline data to use in setting future targets for those measures. For example, one of the fiscal year 1998 performance goals set by Education’s guaranteed student loan program is that the “Level of [overall school] satisfaction will meet or exceed the level of school satisfaction measured last year,  67 percent of the schools reported satisfaction.” However, for other measures, the program did not set targets that could be used to monitor their progress on those measures. For example, Education’s guaranteed student loan program did not establish a fiscal year 1998 target for its annual delinquency rate measure. The program reported that this measure will provide information on the dollar amount of loans “past due” as a percentage of dollars in repayment and that baseline data for the measure will be developed as the definition of “past due” is finalized.
section_title: The Credit Programs Have Been Facing Three General Challenges to Developing Performance Information
paragraphs: On the basis of agency officials’ responses to questions on developing performance information for the selected programs, we identified three general challenges those programs have been facing: (1) a struggle in reaching consensus among stakeholders on the programs’ intended purposes, performance measures, and target levels of performance; (2) difficulty in separating the effects of external forces from program influences on results; and (3) a lack of relevant program performance and financial baseline data. Agency officials also described some approaches they were taking to address the challenges they have been facing.
subsections: section_title: A Struggle in Reaching Consensus Among Stakeholders on the Programs’ Intended Purposes, Performance Measures, and Target Levels of Performance
paragraphs: As we have noted in a previous report, because the interests of a program’s stakeholders can and often do differ significantly, full agreement among those stakeholders on all aspects of the program’s performance is relatively uncommon. However, our past and current work also has shown that although it is difficult to get stakeholders to reach agreement, stakeholder involvement can help an agency identify results-oriented performance measures and set realistic target levels. For example, VA officials said that they had difficulty reaching consensus with internal program stakeholders at the agency’s field offices. This difficulty concerned how the activities of the field offices would be linked to achieving the intended results of VA’s Loan Guaranty Program as established by headquarters staff. To address this difficulty, VA headquarters and field office staff worked together to develop the program’s performance measures. They said the program then brought together key VA headquarters and field managers to reach agreement on target performance levels for linking field office activities to the intended results of the program.
VA’s Loan Guaranty Program also struggled with trying to reach consensus with OMB. According to a VA official, OMB suggested that the program establish “outcome-oriented” performance measures, where feasible, which could provide data on the extent to which the program is helping veterans achieve a higher rate of homeownership. However, VA officials said that the program is an entitlement program (i.e., veterans receive the benefit regardless of need as a reward for their service); and it is not clear whether increasing homeownership among veterans is a primary intended purpose of the program. According to the officials, the more appropriate performance measures for assessing the program’s performance and holding it accountable are those for monitoring how well VA is delivering the benefit (e.g., satisfying veterans and keeping program costs down).
To address OMB’s suggestion, the program established a homeownership assistance measure in its fiscal year 1998 budget presentation, which is to provide data on the percentage of veterans surveyed who said they would not have been able to purchase any home or would have had to purchase a less expensive home without a VA guaranteed loan. However, the program did not identify the homeownership assistance measure as a performance measure. Instead, the program identified this measure as a measure of workload and other program data, which a VA official said was to provide “contextual program information,” rather than information for gauging the program’s progress in achieving its intended purposes and holding the program accountable. The VA official added that the homeownership assistance measure is an “imperfect measure” because of its reliance on self-reporting by veterans.
Similarly, at the time of our interviews with USDA officials, they said internal stakeholders were grappling with what the appropriate results for their SFH loan programs were. According to those officials, the struggle of trying to reach consensus among those stakeholders contributed to why they rated as a significantly challenging activity “developing measures for assessing the net effect of the program.” One agency official said that the programs’ intended purposes were “putting people [who are unable to get credit from other sources] in homes”; thus, to the extent that they put such borrowers in homes, the programs are having a net impact. However, another agency official suggested that the programs’ intended purposes also should include “improving the quality of life among rural residents” and “improving housing conditions and the economy in a given community or state.” He said that measures for gauging the programs’ progress toward such purposes would attempt to collect data on, for example, the extent to which putting people in homes is improving the quality of life among rural residents.
USDA officials also said it is difficult to balance stakeholders’ interests. They said that they are expected to increase program service while also reducing program costs and minimizing default rates. However, they said this is difficult because the SFH programs were designed to offer credit to a population that the private sector would consider high risk. Specifically, to be eligible for a SFH direct loan, a borrower must have a family income that is “very low” to “low” (i.e., a family income under 80 percent of the median income in the area); and the borrower must be unable to get credit from any other source. Therefore, the program’s target population may be more likely to default. At the time of our review, USDA, VA, and Education were working with stakeholders in and outside of the selected programs, including OMB and Congress, to reach consensus with those stakeholders on the most appropriate goals and measures for the programs.
section_title: Difficulty in Separating the Effects of External Forces From Program Influences on Results
paragraphs: The efforts of federal agencies often are but one factor among many that may influence whether, and the degree to which, their programs achieve their intended results. Our past and current work has found that many agencies have been challenged to separate out the influence that their program activities have had on the achievement of program results when those results also could have been influenced by external forces.
Agency officials from all five of the selected credit programs reported as a significantly challenging activity “separating the impact of the credit programs’ activities from the impact of other factors external to those programs but contributing to the results achieved.” They generally cited economic trends; the role of third parties in helping the programs provide loans; and the existence of other federal financial aid programs (e.g., grants) as examples of forces external to their programs that can affect program results.
For example, the foreclosure rate could be viewed as a measure related to the VA Loan Guaranty Program’s intended purpose of helping veterans and active duty personnel retain homes. However, VA reported that external forces, such as interest rates, unemployment, and the general state of the economy, can influence the foreclosure rate. A VA official said that because of such external forces, it has been difficult to confidently attribute a change in the foreclosure rate to the program’s activities and thus view it as a valid measure of the program’s performance.
VA officials said that the program attempted to address this problem in 1993 but was not successful. Specifically, a VA program official said that at the encouragement of OMB, the program attempted to develop a model to help the program estimate its foreclosure rate and monitor its performance. However, he said when the program implemented the model, it significantly overestimated the number of VA foreclosures and thus was not an adequate model for determining the external forces that could affect the rate. The VA official said that because of the many external forces that could affect the number of foreclosures, it was unclear if the model could be adjusted to help it adequately predict foreclosures and whether—given the cost of potentially making many adjustments to the model—the value of doing this was worth the additional cost. Therefore, the program took another approach. The program established a surrogate measure, the FATS ratio (which, as mentioned earlier, is to provide data on the extent to which foreclosures would have been greater had VA not pursued alternatives to foreclosure), for monitoring its performance in assisting veterans to avoid foreclosures. The program views the FATS ratio as a more valid measure than the foreclosure rate for assessing the program’s performance in helping veterans and active duty personnel retain homes.
USDA officials also said that separating the impact of their SFH guaranteed and direct loan programs’ activities from the impact of other external forces on, for example, the quality of life for rural residents is exceedingly difficult. These officials said that the quality of rural residents’ lives could be affected negatively by other, unrelated events, such as borrowers’ incurring health problems or financial hardship, or the closing of a military base eliminating jobs in the area. They said that although providing single-family housing loans may help to improve the quality of life among those rural residents, such improvement could also be due to other external forces, such as home loans being provided by other federal housing loan programs (e.g., programs administered by VA or the Department of Housing and Urban Development).
Similarly, Education officials told us that loans that are issued or guaranteed through their agency’s direct or guaranteed student loan programs are among several types of financial aid that Education offers to help ensure access to postsecondary education. They said that students’ college participation and completion rates can be affected by borrowers’ eligibility for loans through the direct or guaranteed loan programs as well as by such external forces as the eligibility of borrowers for the other types of financial aid assistance provided by Education (e.g., grants); the extent of parental support for the borrowers attending school; and the borrowers having to financially support a family. Although Education established student participation and completion rates as performance measures, it did so to monitor the combined performance of its financial aid programs instead of the specific performance of either the direct or guaranteed student loan programs on those results.
Education officials said that an approach the agency was taking to better understand the determinants of college enrollment—including financial aid obtained through direct loans, guaranteed loans, grants, or other financial aid programs—was contracting for a study of the effects of financial aid, including aid provided by these programs, and various external forces on this result. According to an Education official, this study is expected to be completed in early 1998. We believe the findings from this study may help to inform any future program evaluations assessing the impact of Education’s financial aid programs on results compared to the impact of external forces. In a prior report, we discussed how impact evaluations can help an agency confidently attribute the achievement of intended results to one or more of its programs by providing information on the extent to which those programs contributed to the results achieved relative to the impact of external forces.
section_title: Lack of Relevant Baseline Data
paragraphs: As we reported in June 1997, our prior work has shown that baseline and trend data on past performance can help agencies set realistic target levels of performance for their programs given the past performance of those programs. However, we also noted that because agencies often did not focus on having results-oriented performance information in the past, they generally have not collected such data. Thus, they have not had all of the baseline and trend data they believed they needed to set goals. Further, credit agencies, including VA, USDA, and Education, generally have had difficulty producing reliable performance data, particularly financial data, which executive and legislative branch decisionmakers need to make well-informed decisions.
Agency officials from all five of the credit programs said that a lack of baseline data was why they rated one or more of the following activities as significantly challenging: “developing objective, quantifiable, and measurable annual program performance goals”; “determining a realistic target level of performance”; “developing unit cost information for the programs’ outputs”; and “developing unit cost information for the programs’ outcomes.” For example, VA officials said that some of their performance measures for their Loan Guaranty Program were new, and baseline data were thus not available on those measures. Consequently, VA did not have data on past performance to use in setting some of the program’s fiscal year 1998 target levels of performance and reported that those targets were “to be determined.”
Education officials also attributed the challenges they had in determining realistic target levels of performance for Education’s direct student loan program to a lack of baseline data. According to those officials, the program had not been in existence long enough to have historical data on many of the program’s measures to use in setting fiscal year 1998 targets. They said that a lack of historical data for the direct loan program was a particular problem in terms of predicting borrower repayment behavior, since few borrowers had yet entered the repayment phase. Thus, to set the target for the program’s default rate measure, Education used historical data for the same measure established for Education’s guaranteed student loan program.
Similarly, USDA officials said the agency’s SFH programs did not have information systems to collect data on some performance measures, such as the number of loans made in targeted geographic areas. Thus, according to those officials, the programs did not have an informed basis on which to set fiscal year 1998 target levels of performance for those measures and did not include them in their fiscal year 1998 budget submission to OMB. Rather, the program included substitute measures for which the programs had information systems in place to collect the data (e.g., the number of rural families with improved or more suitable housing conditions).
subsections: section_title: Lack of Reliable Financial Data
paragraphs: Our prior and ongoing work and that of agencies’ internal or independent auditors have found that some credit agencies still have difficulty, despite numerous years of experience, in producing reliable financial data, such as credit programs’ subsidy rates—the estimated cost to the government from direct loans and loan guarantees. For example, USDA and Education received a disclaimer of an opinion from internal and independent auditors, respectively, on their fiscal year 1996 financial statements. In part, this was due to those agencies not being able to provide the data needed to (1) accurately reflect the cost to the government and (2) permit appropriate cost comparisons between credit and noncredit programs.
VA received an unqualified opinion on its fiscal year 1996 financial statements from VA’s Office of Inspector General (OIG). However, the OIG audit, which included a review of the Loan Guaranty Program, found that the program did not reliably accumulate the financial information needed to comply with federal financial accounting standards, identified significant errors that required financial statement adjustments, and identified other errors where data compiled manually did not always reconcile with original source amounts.
If successfully implemented, the CFO Act will help credit agencies resolve long-standing problems with data reliability. Further, in passing the Results Act, Congress emphasized that the usefulness of agencies’ performance data depends, to a large degree, on the reliability of those data. Therefore, the Results Act requires agencies to describe in their annual performance plans the means to be used to verify and validate performance data. We have suggested in prior reports that such information, including information about the reliability of credit agencies’ performance data, could be equally important for those agencies to disclose in their reports to ensure report users of the quality of that data.
One area in particular need of attention is the development of reliable financial information on the full cost and unit cost of a program, which is an integral part of measuring that program’s efficiency and cost effectiveness. An essential step in developing such information is the identification of individual program costs, such as direct labor. In that regard, unit cost information can be particularly useful in identifying trends and determining key cost drivers of the program. Agency officials from all five of the credit programs we reviewed said they have been challenged to develop unit cost information for the programs’ outputs and outcomes. They generally cited difficulties in allocating basic cost data to specific programs as a reason for this challenge. For example, USDA officials explained that its field offices are involved in administering the SFH direct and guaranteed loan programs as well as other USDA programs. They said difficulties in separating data on labor costs for the various programs, for example, have contributed to the challenge they have faced in developing unit costs for the SFH programs.
Similarly, VA officials said that developing meaningful unit cost information for the Loan Guaranty Program has been a significantly challenging activity due to the lack of an adequate methodology, including VA’s inability to separate the data on the actual costs for the program, such as labor costs, from the costs for the several other programs that VA field offices administer. A VA official said that because of the difficulty it has had in isolating program cost data, the program uses a “very loose” process to calculate unit costs, which involves dividing the various resource levels authorized for the program, such as staffing levels (i.e., authorized full-time equivalents) by the activity level during the year (e.g., number of loans guaranteed).
New accounting standards developed by the Federal Accounting Standards Advisory Board (FASAB) require federal agencies subject to the CFO Act and Results Act to collect relevant and reliable data on the full costs of carrying out a mission or producing products or services. Although these standards were scheduled to be effective for all federal programs beginning with fiscal year 1997, because of serious shortfalls in agencies’ cost accounting systems, FASAB extended the date by 1 year to fiscal year 1998. The standards took effect on October 1, 1997. Credit agency officials said their respective agencies were establishing information systems to collect needed cost data. Specifically, VA officials said their agency was looking at the use of activity-based costing (ABC) to develop more meaningful unit cost information. According to an Education official, his agency also is developing a system that should be able to provide at least some data on unit costs by 1999. USDA reported in its fiscal year 1998 budget presentation that it was working to develop the data for its SFH direct and guaranteed loan programs’ measure on the “cost of housing a family per recipient household.”
section_title: Progress in Developing Common Performance Measures Has Been Limited
paragraphs: According to OMB and the Working Group, comparing results using common measures across credit programs allows program managers and other decisionmakers to identify best practices among those programs that have the potential for improving other credit programs’ performance. In addition, the Working Group anticipated that agencies that administer credit programs could include such measures in their annual performance plans and reports under the Results Act. By September 1996, a Working Group task force had proposed a set of common financial and programmatic performance measures for federal credit programs. Some of the proposed measures—namely, financial measures for meeting the annual budgeting and reporting requirements of DCIA and FCRA—have been adopted by major credit agencies.
However, two general problems have limited the Working Group’s progress in developing common performance measures for credit programs. First, as previously noted, several credit programs lack relevant program performance and financial baseline data. Second, the Working Group has been unable to reach consensus on the appropriateness of decisionmakers using some of the task force’s proposed measures to assess the performance of individual credit programs or to compare that performance against the performance of other credit programs. In addition, OMB does not intend to require credit programs to adopt common performance measures when consensus about the appropriateness of such measures has not been achieved.
subsections: section_title: Common Performance Measures for Credit Programs Have Been Proposed, and Some Have Been Adopted
paragraphs: In July 1995, the Working Group established a task force to develop common performance measures that could help agencies and other decisionmakers make relevant comparisons of the results of credit programs. By September 1996, the task force had proposed a set of common financial and programmatic measures in the following four areas: (1) Financial performance. Measures in this area include total receivables; total delinquent debt; default rates; actual versus projected subsidy rates; and administrative costs, such as the costs of extending credit and servicing loans. (2) Program performance in achieving desired loan characteristics. Measures in this area include the percentage of loans going to borrowers who would otherwise not have access to private credit and the percentage of borrowers pleased with the timeliness and quality of credit program service. (3) Program effects on society. Measures in this area include (1) intended effects, such as “supporting investment important to the economy” as monitored by, for example, the amount and quality of low-income housing financed (home loan programs) and business investment financed (business loan programs); and (2) unintended effects, such as borrowers accumulating excessive debt burden. (4) Program “additionality.” Measures in this area indicate the results achieved by the program by providing financial assistance to borrowers that private markets will not serve. An example of a measure in this area is the net increase in homeownership as a result of the program supplementing versus substituting for private financing.
OMB and the Working Group’s task force also have encouraged credit programs to establish, where appropriate, program-specific measures and explanatory data for helping to ensure a complete assessment of their programs’ performance and to explain the results to users of performance information.
In its June 1997 1997 Federal Financial Management Status Report and Five-Year Plan, OMB and the CFO Council reported that major credit agencies have adopted the proposed common financial measures for assessing their debt collection activities. For example, credit agencies are to collect data on total receivables, total delinquent debt, and total collections for their credit programs. According to an OMB official, these and other common measures that the task force proposed in the area of financial performance are measures that major credit agencies adopted for meeting the annual budgeting and reporting requirements of DCIA and FCRA. However, he said that those agencies have not yet adopted measures that the task force proposed under the other three areas—measures for monitoring and improving a credit program’s (1) performance in achieving desired loan characteristics, (2) effects on society, and (3) additionality.
section_title: Two General Problems Have Limited Progress in Developing Common Performance Measures
paragraphs: Although the common financial measures proposed by the task force have been adopted, two general problems have limited the Working Group’s progress in developing common performance measures. First, as previously noted, several credit programs lack relevant program performance and financial baseline data. Second, the Working Group has been unable to reach consensus on the appropriateness of decisionmakers using some of the task force’s proposed measures to assess the performance of individual credit programs or to compare that performance against the performance of other credit programs. For example, some programs give credit only to persons who are unable to obtain credit from other sources; other programs give credit to anyone who is entitled to the programs’ benefits, regardless of his or her access to credit. Thus, officials in some credit agencies questioned whether it would be appropriate to make comparisons among the default rates of credit programs, because the financial characteristics of borrowers for each program may be different.
subsections: section_title: Relevant Data on Several of the Common Measures Proposed Are Lacking
paragraphs: Agency officials told the task force that for several of the proposed measures, their agencies’ credit programs generally did not collect data relevant to those measures, collected incomplete data, or did not routinely collect such data. Such measures included the percentage of borrowers who were pleased with the timeliness and quality of credit program service and measures for monitoring the effects of credit programs on society and program additionality. For example, common measures proposed by the task force for monitoring the performance of home loan programs and business loan programs in “supporting investment important to the economy”—one of four intended effects on society the task force proposed—included the amount and quality of low-income housing financed and business investment financed, respectively. Officials from VA, the Department of Housing and Urban Development (HUD), and the Small Business Administration told the task force that the credit programs at their respective agencies generally did not collect such data. Moreover, even when data existed, our prior and ongoing work and audits by credit agencies’ inspectors general and others have consistently disclosed serious weaknesses in agencies’ systems, which has affected the reliability of data that are used to account for and manage credit programs.
section_title: The Working Group Has Been Unable to Reach Consensus on the Appropriateness of Some of the Proposed Measures
paragraphs: Consistent with the views of OMB and the Working Group, our work over the last few years has recognized that common performance measures for similar programs can provide important information for permitting comparisons of the results of those programs and the methods used to achieve those results. Such information could help program managers identify credit program performance gaps; set improvement goals; improve credit program processes; and inform other decisionmakers, such as OMB and Congress. However, we found that members of the Working Group have been unable to reach consensus on the appropriateness of some of the common performance measures proposed by the task force.
Specifically, officials in some agencies questioned whether data collected from some of the proposed common measures would be meaningful for assessing the performance of their agencies’ credit programs. For example, VA and Education officials said that their agencies’ Loan Guaranty Program and direct and guaranteed student loan programs, respectively, are entitlement programs in which the government is obligated to give credit to anyone who qualifies for the programs’ benefits, regardless of his or her access to credit. Thus, these officials questioned whether a common programmatic measure proposed by the task force, “percent of loans or guarantees originated going to borrowers who would otherwise not have access to private credit,” was meaningful for their programs. As mentioned earlier, to be eligible for a USDA SFH direct loan, a borrower must have a family income that is “very low” to “low,” and the borrower must be unable to get credit from any other source. A USDA official said that the target and result for this proposed measure would always be 100 percent; thus, he also questioned whether the measure was meaningful for that program.
In addition, for certain measures, officials in some credit agencies questioned whether the data collected would be appropriate to use in making comparisons among different credit programs’ performance results, such as the programs’ default rate, because the financial characteristics of borrowers for each program may be different. For example, USDA officials said that because the borrower must be unable to obtain private credit, the program’s target population may be more likely to default. Thus, according to USDA officials, it may be inappropriate to compare, for example, the actual default rate of the SFH direct loan program with the actual default rate of the VA Loan Guaranty Program, which is an entitlement program.
We agree that common measures need to be carefully explained to help ensure that significant program differences are properly interpreted. However, it is not clear that such differences outweigh the potential usefulness of common measures. Credit agencies have generally agreed with our suggestion that they provide explanatory information where necessary in agency reports to portray program differences more appropriately and help users of performance information understand the reported performance.
Also related to the problem of comparable data, we recently reported that agencies’ use of inconsistent definitions for their programs’ measures could hamper decisionmakers’ use of data collected from those measures in planning, comparing performance, and reporting on performance achieved. Our June 1997 report noted that some credit agencies differ in how they classify previously delinquent debt on which borrowers are currently making payments. Some reclassify such debt as “current,” but others keep it in a delinquent category regardless of the current payment status. Although such classification practices may be suitable within an agency, they make it difficult to compare agency performance or aggregate data for similar programs. For example, VA loans maintain their delinquent status until the delinquency is repaid or written off. Conversely, a home loan program within HUD reclassified single-family delinquent loans to a current repayment status when borrowers complied with forbearance terms, which typically included making partial mortgage payments for up to 3 years. In our June report, we recommended that the Department of the Treasury’s Financial Management Service work with major credit agencies and OMB to help those agencies consistently report on delinquent debt or disclose their inconsistencies. Treasury, OMB, and the major credit agencies generally agreed with that recommendation, and the agencies commented that consistent application of governmentwide debt collection reporting criteria is essential.
Similarly, we observed that when consistent definitions do not exist among credit programs, it is difficult to know whether results reported would be readily comparable. For example, one of the common programmatic measures proposed by the task force for monitoring loan characteristics was the “percent of borrowers who are pleased with the timeliness and quality of credit program service.” However, for this proposed task force measure, it is unclear what is meant by “timeliness” and “quality” of service, because each credit program may have a different interpretation. Education’s guaranteed and direct student loan programs, for example, established a broad measure of “overall borrower satisfaction”; VA’s Loan Guaranty Program established more specific borrower satisfaction measures—the percent of respondents surveyed who will say (1) they are satisfied with their contact with VA; (2) they are satisfied with the time it took to obtain a Certificate of Eligibility; and (3) their loan did not take longer to process than expected as a result of a delay blamed on VA (which is to help the program monitor when program staff may need to work with the lender in identifying reasons for the delay in loan processing time, toward improving performance on this measure). Thus, it is unclear whether the data collected would be comparable if this proposed measure of the task force were used by all credit programs.
The Working Group anticipated that agencies that administer credit programs could include common financial and programmatic measures in their annual performance plans and reports under the Results Act. According to OMB’s Senior Advisor for Cash and Credit Management, at an upcoming meeting of the Working Group OMB intends to ask members whether they want to continue to address their concerns about adopting additional common performance measures for credit programs. However, he said that OMB is not taking a prescriptive role in directing agencies’ performance measurement activities. Rather, he said the administration wants agencies to take the initiative on such activities. According to this official, if members choose not to address those concerns, OMB does not intend to require credit programs to adopt common performance measures when consensus about the appropriateness of such measures has not been achieved. Thus, it is unclear when, or if, agencies that administer credit programs will include common measures for those programs in their annual performance plans under the Results Act.
section_title: Conclusions
paragraphs: The credit programs we reviewed have established goals and performance measures that appeared to be generally related to the programs’ intended purposes and set target levels of performance for most of their respective measures. In addition, our prior work suggests that the development of common measures, where appropriate, can provide important information for permitting comparisons of similar programs’ results and the methods they used to achieve those results. We have suggested that agencies’ annual performance plans and reports under the Results Act can serve as suitable vehicles for developing such measures and providing such information. The Working Group has focused on developing common financial and programmatic performance measures for credit programs and anticipated that agencies that administer credit programs could include such measures in their annual performance plans and reports under the Results Act.
However, two general problems have limited the Working Group’s progress in developing common performance measures for credit programs: (1) the lack of relevant program performance and financial baseline data for several credit programs and (2) the inability among members of the Working Group to reach consensus on the appropriateness of some of the proposed measures.
OMB does not intend to require credit agencies to adopt common performance measures when consensus about the appropriateness of such measures has not been achieved. We agree that OMB should not force the use of common measures when concerns about their appropriateness exist. However, at the time of our review, the Working Group had not resolved those concerns and had yet to decide how and when those concerns would be addressed; thus, it is unclear whether OMB and the credit agencies will maintain their current level of attention to developing common measures. Also unclear is the extent to which agencies that administer credit programs will include common measures for those programs in their annual performance plans that could provide useful information to decisionmakers interested in making performance and cost comparisons.
We recognize the difficulty the Working Group is facing in reaching consensus on common performance measures and that significant differences in program characteristics may limit the usefulness of some measures for broad, cross-program comparisons. However, we believe the potential benefits that could be realized from developing common performance measures, where appropriate, underscore the importance of OMB and the credit agencies continuing their efforts to develop and reach consensus on such measures. In addition, because the development and use of performance measures, especially common measures that can be used across programs and agencies, are in the early stages of implementation, it will be especially useful for decisionmakers to evaluate early experiences to identify successful, as well as unsuccessful, approaches. To that end, documented information on the measures considered, how they are used and should be interpreted, and how they can be improved will be helpful to agencies in further achieving the purposes of the Results Act.
section_title: Recommendations to the Director of OMB
paragraphs: Building on recommendations and suggestions we have made in prior reports, we recommend that the Director of OMB sustain OMB’s efforts to work with major credit agencies to use annual performance planning under the Results Act as a vehicle for developing common performance measures across credit programs, where appropriate. In doing so, we recommend that beginning with those agencies’ fiscal year 2000 annual performance plans, the Director of OMB require each agency that administers credit programs to identify in their plans (1) performance measures the agency is using for its credit program(s) that are the same as those used by other credit programs and the strengths and limitations of using those measures to make performance and cost comparisons among those programs; and (2) what actions, if any, are being taken or could be taken to refine the agency’s performance measurement efforts to address the identified limitations to using existing measures to make performance and cost comparisons across credit programs.
These or some comparable requirements would serve a twofold purpose. First, they would help ensure that the search for common measures continues. And second, they would document the results of those efforts in a way that would (1) permit further analysis directed at the identification of additional common measures and (2) facilitate an understanding of any limitations of using existing common measures to compare results across credit programs.
section_title: Agency Comments and Our Evaluation
paragraphs: On December 11, 1997, we requested comments on a draft of this report from the Secretaries of Education and Agriculture, the Acting Secretary of VA, and the Director of OMB or their designees. On December 17, 1997, the liaison to GAO from USDA’s Rural Development mission area, which administers the SFH direct and guaranteed loan programs, said the Department generally agreed with the draft report’s factual material, conclusions, and recommendations. A rural development official later provided minor technical suggestions, which we included in the report as appropriate. Similarly, on December 19, 1997, the liaison to GAO from the Department of Education also said the Department concurred with the draft report’s general findings and later provided minor technical suggestions, which we included in the report as appropriate.
On January 8, 1998, the VA liaison to GAO said the Department also generally agreed with the draft report’s factual material, conclusions, and recommendations, except for that part of the recommendation requiring credit agencies to include a discussion about common measures in their fiscal year 1999 annual performance plans. This comment was consistent with the comment provided to us on January 6, 1998, by a representative of VA’s Performance Analysis Service within VA’s Office of Budget. Specifically, that representative told us that VA would not likely have time to provide a meaningful discussion about common measures in its fiscal year 1999 plan because VA (1) had yet to achieve consensus among major stakeholders on the common measures, (2) had not done the analysis necessary to provide the discussion about common performance measures that our draft report recommended, and (3) is to submit the agency’s fiscal year 1999 performance plan to Congress within the next few weeks. OMB officials had a similar comment on the feasibility of requiring a discussion about common measures in credit agencies’ fiscal year 1999 performance plans, which we discuss later in this section.
In response to this comment from both VA and OMB officials, we removed from the recommendation our suggestion that such a discussion be included in credit agencies’ fiscal year 1999 annual performance plans. However, we retained the recommendation that the Director of OMB require credit agencies to include a discussion about common measures in those agencies’ fiscal year 2000 annual performance plans and subsequent performance plans. We continue to believe that using annual performance planning under the Results Act as a vehicle for developing and discussing common performance measures across credit programs, where appropriate, could provide decisionmakers with important information and help agencies further achieve the purposes of the Results Act. Such information could be useful to decisionmakers in comparing similar programs’ results and the methods they used to achieve those results and for understanding how such measures are used, should be interpreted, and could be improved.
Further, in a conversation with us on December 17, 1997, a representative of VA’s Loan Guaranty Service commented that developing information systems to collect data from common performance measures may be costly and that before VA developed systems to collect such data, the costs of doing so should be weighed against the benefits. We agree that collecting needed data from common measures is a problem that credit agencies face, as discussed in the draft report. We also have suggested that annual performance plans provide agencies with the opportunity to alert Congress to the problems they have had or anticipate having in collecting needed data, including the cost and data quality trade-offs associated with various collection strategies. The representative of VA’s Loan Guaranty Service also provided minor technical suggestions, which we included in this report where appropriate.
On January 7, 1998, we met with the OMB Senior Advisor for Cash and Credit Management and the Senior Advisor to the Deputy for Management, who said the report would serve as a valuable tool and resource in OMB’s continuing efforts to encourage and work with major credit agencies to effectively implement the Results Act. However, those officials cautioned that based on OMB’s experiences in working with major credit agencies to draft their fiscal year 1999 performance plans and to identify appropriate, results-oriented common performance measures for credit programs, developing and reaching consensus on credit program performance measures will continue to be difficult, time-consuming, and iterative. They said that those experiences and OMB’s review of major credit agencies’ fiscal year 1999 performance plans suggested to them that the priority of those agencies at this point in the implementation of the Results Act needs to be on ensuring the quality of the performance goals and measures for their individual credit programs. According to these OMB officials, in developing program-specific as well as common performance measures for credit programs, major credit agencies need to continue working to reach consensus among the key stakeholders of the agencies’ credit programs and to develop information systems for collecting needed performance data, which will be challenging. Moreover, the OMB officials said that agencies are concluding the preparation of their fiscal year 1999 annual performance plans that will accompany the President’s budget submission to Congress in February 1998. The officials said that those plans reflect budget, policy, and programmatic decisions already made in the course of preparing the budget. Thus, the officials believed it would not be feasible for the Director of OMB to direct credit agencies to include in their fiscal year 1999 plans the discussion about common measures that our draft report recommended.
OMB officials said that given the challenges of developing appropriate, results-oriented common performance measures for credit programs, it is unclear when or if such measures could be adopted by major credit agencies. OMB’s Senior Advisor for Cash and Credit Management suggested that as the Working Group’s efforts advance, two common results-oriented performance measures that may be considered appropriate for agencies that administer credit programs are (1) the number of loans a program made that are repaid successfully and (2) the percentage of customers satisfied with the program. However, he said that developing measures that would help isolate a credit program’s contribution to achieving a particular common result from the contribution of external factors may not be possible for all credit programs. The OMB officials also noted that performance information is just one key factor among many that will go into decisionmaking on management and budget policy issues.
The OMB officials told us that OMB will ensure in calendar year 1998 that developing common results-oriented performance measures across credit programs is a priority agenda item for discussions among the Working Group members. They said that as part of these discussions, major credit agencies could share with one another their experiences in developing their individual fiscal year 1999 performance plans and their congressional committees’ reactions to those plans. On the basis of agencies’ experiences and congressional reactions, the OMB officials said they believe that OMB and the major credit agencies would have a better foundation from which to discuss common performance measures for the agencies’ credit programs. According to the OMB officials, this experience will provide OMB and the agencies with the basis for determining the feasibility of incorporating common measures and a discussion about such measures into future annual performance plans, where appropriate.
We believe that OMB’s planned approach to use major credit agencies’ fiscal year 1999 performance planning efforts as the foundation for discussions among these agencies on common performance measures is responsive to the intent of our recommendation. In this regard, in response to OMB officials’ comments about the feasibility of the Director of OMB requiring credit agencies to include in their fiscal year 1999 annual performance plans the discussion about common measures that our draft report recommended, as mentioned earlier, we removed the reference to those agencies including such a discussion in their fiscal year 1999 plans. However, as also mentioned earlier, we retained the recommendation for those agencies to include such a discussion in their fiscal year 2000 annual performance plans and subsequent performance plans. Such discussions can serve as vehicles for highlighting many of the other cautionary notes that the OMB officials raised, such as the difficulties in developing measures that seek to isolate a credit program’s unique contributions to a particular result.
We are sending copies of this report to the Majority Leader, House of Representatives; the Chairman and Ranking Minority Member, Committee on Government Reform and Oversight, House of Representatives; the Chairman and Ranking Minority Member, Committee on Governmental Affairs, United States Senate; the Secretaries of Education and Agriculture; the Acting Secretary of VA; and the heads of agencies that administer credit programs and are represented on the Federal Credit Policy Working Group. We also will make copies available to others on request.
The major contributors to this report are listed in appendix IV. Please contact me on (202) 512-8676 if you have any questions.
section_title: Overview of the Results Act
paragraphs: The Results Act is the primary legislative framework through which agencies will be required to set strategic goals, measure performance, and report on the degree to which goals were met. It starts by requiring each federal agency to develop a strategic plan that covers a period of at least 5 years and includes the agency’s mission statement; identifies the agency’s long-term strategic goals; and describes how the agency intends to achieve those goals through its activities and through its human, capital, information, and other resources. The first strategic plans that the Act required agencies to develop were to be completed by September 30, 1997.
Also, the act requires each agency to submit to the Office of Management and Budget (OMB), beginning for fiscal year 1999, an annual performance plan. The first annual performance plans were to be submitted to OMB in the fall of 1997. The annual performance plan is to provide the direct linkage between the strategic goals outlined in the agency’s strategic plan and what managers and employees do day to day. In essence, this plan is to contain the annual performance goals the agency will use to gauge its progress toward accomplishing its strategic goals and identify the performance measures the agency will use to assess its progress. Also, OMB will use individual agencies’ performance plans to develop an overall federal government performance plan that OMB is to submit annually to Congress with the president’s budget, beginning for fiscal year 1999.
The Results Act also requires that each agency submit an annual report to the president and to the appropriate authorization and appropriations committees of Congress on program performance for the previous fiscal year (copies are to be provided to other congressional committees and to the public upon request). The first of these reports, on program performance for fiscal year 1999, is due by March 31, 2000; and subsequent reports are due by March 31 for the years that follow. However, for fiscal years 2000 and 2001, agencies’ reports are to include performance data beginning with fiscal year 1999. For each subsequent year, agencies are to include performance data for the year covered by the report and 3 prior years.
Finally, in crafting the Results Act, Congress recognized that managerial accountability for results is linked to managers having sufficient flexibility, discretion, and authority to accomplish desired results. The Act authorizes agencies to apply for managerial flexibility waivers in their annual performance plans beginning with fiscal year 1999. The authority of agencies to request waivers of administrative procedural requirements and controls is intended to provide federal managers with more flexibility to structure agency systems to better support program goals. The nonstatutory requirements that OMB can waive under the Results Act generally involve the allocation and use of resources, such as restrictions on shifting funds among items within a budget account. Agencies must report in their annual performance reports on the use and effectiveness of any managerial flexibility waivers that they receive.
section_title: Objectives, Scope, and Methodology
paragraphs: Our first objective was to identify goals and measures established by the selected credit programs that related to the programs’ intended purposes and determine whether the programs had set target levels of performance for assessing their progress in achieving their desired results. Under the Results Act, target levels of performance are to enable a comparison of planned versus actual results achieved for a given year. Our second objective was to identify the challenges agency officials cited in developing performance information, including goals and measures, for the selected programs and any approaches those programs were taking to address those challenges. Our third objective was to describe the status of the Working Group’s effort to develop common performance measures for federal credit programs.
For our review, we selected a nonrandom, purposive sample of five federal credit programs at three agencies. These programs were the Department of Veterans Affairs’ Loan Guaranty Program; the Department of Education’s William D. Ford Direct Loan Program and its Federal Family Education Loan Program (referred to in this report as Education’s direct and guaranteed student loan programs, respectively); and the Department of Agriculture’s Single-Family Housing (SFH) direct loan program and guaranteed loan program. We selected for our review credit programs that varied in terms of type of program (e.g., housing and education loans); mode of credit delivery (e.g., direct and guaranteed loans); and program size as measured by the amount of outstanding loans. According to data reported by OMB in the fiscal year 1998 budget and agency data, of the total amount of federal credit outstanding in fiscal year 1996 for guaranteed loans ($805 billion) and direct loans ($165 billion), these five programs represented about 32 and 18 percent, respectively. The smallest and largest of the guaranteed loan programs accounted for slightly less than 1/2 percent and 19 percent, respectively, of the fiscal year 1996 credit outstanding in loan guarantees governmentwide. Similarly, the smallest and largest of the direct loan programs held 7 percent and 11 percent, respectively, of the fiscal year 1996 credit outstanding in direct loans governmentwide. Because of the small and nonrandom nature of our sample, our observations and analyses are not generalizable to other federal credit programs.
To address the first part of our first objective (i.e., to identify goals and measures established by the selected credit programs that related to the programs’ intended purposes), we compared goals and measures established by the programs as of May 1, 1997, to the programs’ intended purposes as identified by those programs or their respective agencies.
According to agency officials, these goals and measures were generally the same ones the programs submitted to OMB and Congress with their respective agencies’ fiscal year 1998 budget presentations. The programs also were proposing to include these goals and measures for their agencies’ fiscal year 1999 annual performance plans under the Results Act.
To determine whether a goal or measure was related to a program’s intended purposes, we reviewed available agency and credit program documentation for (1) a description of the program’s intended purposes and (2) a discussion that reasonably related that particular goal or measure to the program’s intended purposes. When agency documentation did not contain such a discussion, we examined the wording and considered the meaning of each program’s goals and measures and compared them to the program’s intended purposes to identify relationships between them. We also interviewed agency officials about the programs’ intended purposes and asked those officials to comment on the relationships we identified between the programs goals and measures and intended purposes.
To address the second part of our first objective (i.e., to determine whether the selected programs set fiscal year 1998 target levels of performance for assessing their progress in achieving desired results), we identified those measures for which the programs had either (1) identified fiscal year 1998 targets or (2) reported prior year baseline data on those measures and indicated how performance on those measures was to change (i.e., increase or decrease) in fiscal year 1998 relative to the baseline. Our review did not address the reasonableness of the processes or methods the programs used to determine how to assess progress or establish target levels toward achieving the programs’ intended purposes; determine whether other, more appropriate, measures existed; or evaluate the feasibility of the targets the programs established.
To address the first part of our second objective (i.e., to identify the challenges agency officials cited facing in developing performance information for the selected programs), we developed a data collection instrument that listed 49 activities that we identified as being associated with developing performance information. Examples of these activities included “determining a realistic target level of performance for annual performance goals” and “developing measures for assessing the net effect of the program compared with what would have occurred in the absence of the program.” To identify these activities, we referred to key steps and practices identified in our Executive Guide: Effectively Implementing the Government Performance and Results Act and other work that we had under way assessing the challenges agencies were facing in implementing performance measurement.
We sent the instrument to agency officials responsible for and involved in the development of goals and measures for the selected programs and asked those officials to rate, using a five-point scale from “little or no” challenge to a “very great” challenge, how great a challenge each of the 49 activities was to perform for those programs. Officials also could indicate that they had not engaged in a particular activity. For purposes of this report, we refer to activities that any of the agency officials rated as a “great” or “very great” challenge as significantly challenging activities. We then interviewed those officials to discuss why they rated certain activities as significantly challenging. We analyzed their responses and related documentation to identify general challenges that led officials to report those activities as significantly challenging for the selected programs.
To address the second part of our second objective (i.e., to identify any approaches that the selected programs were taking to address the challenges they identified to developing performance information), we interviewed the agency officials who responded to our data collection instrument to obtain information on the approaches they were taking to address those activities they had identified as significantly challenging and analyzed agency documentation. We also asked these officials to comment on the way in which we described the approaches the programs were taking to address the challenges in developing performance information that they had identified. We also considered prior and ongoing work we have done on the efforts of VA, Education, USDA, and other credit agencies to implement various credit and financial management reforms.
To address our third objective (i.e., to describe the status of the Working Group’s effort to develop common performance measures for federal credit programs), we reviewed various documentation from OMB and members of the Working Group that described its effort to develop common performance measures for credit programs and the views of agency officials on those measures. We also talked to OMB officials and members of the Working Group, including agency officials at those agencies administering the programs we selected for our review, to obtain their views on the common performance measures proposed. Our review did not address the reasonableness of the processes or methods the Working Group’s task force used in determining how to assess progress for federal credit programs or determine whether other measures existed that may be more appropriate.
section_title: Selected Credit Programs’ Intended Purposes and Examples of Goals and Measures They Presented in Their Fiscal Year 1998 Budgets
paragraphs: Help veterans and active duty personnel purchase and retain homes in recognition of their service to the nation.
Treat all veterans and other participants in the program in a courteous, responsive, and timely manner.
Examples related to the program’s intended General goal(s)
Performance measure(s)
Assist veterans in obtaining home mortgage loans.
Percent of respondents to the Lender Customer Satisfaction Survey who say they are satisfied with their overall interaction with VA.
The general goal defines, in part, the intended purpose of helping veterans purchase homes. VA identified this goal and measure as related on the basis of the following rationale: Lender satisfaction addresses an intermediate result expected to lead to achieving this goal and intended purpose. Specifically, maximizing lenders’ satisfaction with their dealings with VA employees is expected to encourage lenders to participate in the program, which is expected to expand financing opportunities (i.e., available mortgage loans) for helping veterans purchase homes.
Assist veterans in avoiding foreclosures.
Foreclosure Avoidance Through Servicing (FATS) Ratio.
The general goal defines this intended purpose, and VA identified this goal and measure as related. The FATS Ratio is to provide data on the extent to which foreclosures would have been greater had VA not pursued alternatives to foreclosure, such as intervening with the holder of the loan on behalf of the borrower to set up a repayment plan.
Assist veterans in obtaining home mortgage loans.
Percent of respondents to the Veteran Customer Satisfaction Survey who say they are satisfied with their overall contact with VA.
VA identified this goal and measure as related. This measure defines, in part, “assisting” veterans in obtaining homes because it addresses, in summary form, treating veterans in a courteous and responsive manner.
Assist veterans in obtaining home mortgage loans.
Percent of respondents to the Veteran Customer Satisfaction Survey who say they are satisfied with the time it took to obtain a Certificate of Eligibility.
VA identified this goal and measure as related. This measure defines, in part, “assisting” veterans in obtaining homes because it addresses treating veterans in a timely manner. Before a lender issues a VA guaranteed mortgage loan, VA must certify to the lender that the borrower is a veteran who is eligible for a loan guaranteed by VA. (continued)
VA’s Loan Guaranty Program (continued)
Operate in the most efficient manner possible to minimize costs and ensure the best use of the taxpayer’s dollar.
Provide homeownership loans to: — very-low-income and low-income families (i.e., families who have incomes under 80% of median) who do not own adequate housing and cannot obtain credit from other sources; — eligible farm owners for housing for themselves or for farm laborers.
Provide “supervised credit” to many rural borrowers to help them maintain their homes in times of financial crises through credit counseling, workout agreements, and moratoriums.
Provide homeownership opportunities to moderate-income rural residents (i.e., between 80% and 115% of median).
Utilize private lenders to provide mortgages to borrowers who would be unable to obtain credit without the guarantee.
Examples related to the program’s intended General goal(s)
Performance measure(s)
Efficient credit and program management.
Percent of early defaults of all loans originated.
The goal describes the strategy and addresses intermediate results expected to lead to minimizing costs and ensuring the best use of the the taxpayer’s dollar. On the basis of the following rationale, the measure, which VA identified as related to this goal, addresses this intended result: Once a loan is put into default status, collection activities are initiated, which can ultimately include the foreclosure of the borrower’s home. Such activities are costly to the government and, thus, the taxpayer. Early defaults (i.e., defaults within 6 months of origination) are more likely than a later default to be due to a deficiency in the underwriting of the loans. Thus, efficient credit and program management is expected to reduce early defaults and, therefore, minimize costs and ensure the best use of the taxpayer’s dollar.
No general goal for this intended purpose was identified by the program or agency.
Number of rural families with improved or more suitable housing conditions.
This measure includes the number of direct loans provided, or issued, to eligible borrowers. Thus, the rationale for the relationship is that the program provides loans, or “housing opportunities,” for improved or more suitable housing to very-low- to low-income families and farm owners who do not own adequate housing and who cannot obtain credit from other sources.
No general goal for this intended purpose was identified by the program or agency.
Percentage of borrowers current.
The rationale for the relationship between this measure and helping rural borrowers maintain their homes follows: Rural borrowers encountering financial crises are likely to miss scheduled payments and thus are not likely to be counted as “current” in making scheduled loan payments. When a borrower is viewed as not current, the borrower’s loan is delinquent and put into default status. Once a loan is put into default status, collection activities are initiated, which can ultimately include the foreclosure of the borrower’s home.
No general goal for this intended purpose was identified by the program or agency.
Number of rural families with improved or more suitable housing conditions.
This measure includes the number of guaranteed loans provided to eligible borrowers. Thus, the rationale for this relationship is that the program utilizes private lenders to provide mortgages, or homeownership opportunities, for improved or more suitable housing to moderate-income rural residents who would be unable to obtain credit without the guarantee. (continued)
Provide students and their families with federally sponsored loans—using a streamlined student loan system that simplifies loan access and allows for flexible repayment—to help borrowers meet increasing postsecondary education costs and to reduce taxpayer costs.
Examples related to the program’s intended General goal(s)
Performance measure(s)
Maintain a high level of borrower satisfaction.
Rate of borrowers’ overall satisfaction with the program during the first year.
The rationale for the relationship between this goal and measure and helping borrowers meet increasing postsecondary education costs is that:  — providing loans using a streamlined system will simplify borrowers’ access to postsecondary education loans, thereby helping to satisfy those borrowers in simplifying their ability to meet increasing postsecondary education costs; and  — using a streamlined system that allows for flexible repayment will help ease borrowers’ debt burden, thereby also helping to satisfy those borrowers in helping them meet increasing postsecondary education costs.
Provide flexible repayment options so that debt burden is eased and defaults are minimized.
Cohort default rate.
The general goal defines the program’s strategy and intended purpose, and Education identified this goal and measure as related. The rationale for the relationship between the measure and intended purpose is that borrowers whose loans are placed into a default status are likely encountering debt burden and thus having difficulty meeting postsecondary costs. Once a loan is put into default status, debt collection activities are initiated, which are costly to the government and thus the taxpayer. In addition, by using a streamlined system that allows for flexible repayment, the program will help ease borrowers’ debt burden, thereby helping those borrowers meet increasing postsecondary education costs. (continued)
Education’s direct student loan program (continued)
Ensure access to capital for postsecondary education.
Successfully implement and manage the direct student loan program.
Provide students and their families with federally sponsored loans to help meet increasing postsecondary education costs.
Ensure access to capital for postsecondary education.
Examples related to the program’s intended General goal(s)
Performance measure(s)
Continue to provide a streamlined loan delivery system to attract schools to participate.
Institutional direct loan program participation rate.
This goal describes the strategy for achieving this intended purpose. In addition, this goal and measure describe an intermediate result the program expects to lead to achieving this intended purpose. Specifically, the program uses schools as the vehicles for providing loans to students and their families. By providing a streamlined loan delivery system, the program expects to increase schools’ participation, which is an intermediate result the program expects will lead to broader student access to capital for postsecondary education.
Maintain a high level of school satisfaction.
Rate of overall  satisfaction with the direct student loan program.
Institutional retention rate.
This goal and these measures describe intermediate results the program also expects to lead it to achieving this intended purpose. As previously noted, the program uses schools as the vehicles for providing loans to students and their families. Increasing participating schools’ satisfaction is an intermediate result that the program expects will lead to encouraging those schools to stay as participants in the program; and increasing schools’ retention in the program is an intermedate result that the program expects will lead to broader student access to capital for postsecondary education.
Continue to provide strong fiscal management of the program.
Number of internal control program weaknesses identified in Education’s financial statement audit.
The goal defines the intended purpose, and Education identified this goal and measure as related.
Maintain a high level of borrower satisfaction from the time of loan origination through the end of the repayment period.
Overall rate of borrower satisfaction with the guaranteed student loan program.
The rationale for the relationship between this goal and measure and the intended purpose is that by providing students and their families with loans (through the guaranteeing of loans made by private lenders), the program will be helping to satisfy those borrowers in their ability to meet increasing postsecondary education costs.
Ensure access to guaranteed loans in a changing marketplace.
Number of borrower complaints.
The goal defines the intended purpose, and Education identified the goal and measure as related. The number of borrower complaints is to include data on the number of borrowers who complain to Education about being denied access to a guaranteed postsecondary education loan. (continued)
Education’s guaranteed student loan program (continued)
Successfully deliver and manage the guaranteed student loan program in an efficient and cost-effective manner to help students and their parents meet postsecondary education costs.
Examples related to the program’s intended General goal(s)
Performance measure(s)
Provide a program that is cost-effective for the taxpayer.
Annual delinquency rate.
Annual collection rate.
Per unit administrative costs.
The goal defines the intended purpose, and Education identified this goal and these measures as related. Monitoring the program’s performance in, for example, managing the program’s loan portfolio debt and the program’s administrative costs addresses providing a program that is cost-effective for the taxpayer, or successfully delivering and managing the program in an efficient and cost-effective manner.
section_title: Major Contributors
subsections: section_title: General Government Division
section_title: Acknowledgements
paragraphs: In addition to those named above, the following individuals made notable contributions to this report: From the Accounting and Information Management Division: Jeff Steinhoff, Director of Planning and Reporting; Dan Blair, Mary Ellen Chervenic, Michael J. Curro, Julie S. Tessauro, and McCoy Williams, Assistant Directors; and Rita A. Grieco and Carolyn Litsinger, Senior Evaluators. From the General Government Division: Joseph S. Wholey, Senior Advisor for Evaluation Methodology; and Stephanie Shipman, Assistant Director. From the Health, Education and Human Services Division: Joseph J. Eglin, Jr., Assistant Director; Paula N. Denman, Senior Evaluator; and Sara E. Edmondson, Senior Social Science Analyst. From the Office of General Counsel: Alan Belkin, Assistant General Counsel; and James M. Rebbe, Attorney. From the Resources, Community, and Economic Development Division: Robert S. Procaccini, Assistant Director.
section_title: Related GAO Products
paragraphs: Managing for Results: Agencies’ Annual Performance Plans Can Help Address Strategic Planning Challenges (GAO/GGD-98-44, Jan. 30, 1998).
Managing For Results: Using the Results Act to Address Mission Fragmentation and Program Overlap (GAO/AIMD-97-146, Aug. 29, 1997).
The Government Performance and Results Act: 1997 Governmentwide Implementation Will Be Uneven (GAO/GGD-97-109, June 2, 1997).
Debt Collection: Improved Reporting Needed on Billions of Dollars in Delinquent Debt and Agency Collection Performance (GAO/AIMD-97-48, June 2, 1997).
Managing For Results: Analytic Challenges in Measuring Performance (GAO/HEHS/GGD-97-138, May 30, 1997).
Executive Guide: Effectively Implementing the Government Performance and Results Act (GAO/GGD-96-118, June 1996).
GPRA Performance Reports (GAO/GGD-96-66R, Feb. 14, 1996).
Financial Management: Continued Momentum Essential to Achieve CFO Act Goals (GAO/T-AIMD-96-10, Dec. 14, 1995).
Executive Guide: Improving Mission Performance Through Strategic Information Management and Technology—Learning From Leading Organizations (GAO/AIMD-94-115, May 1994).
Credit Reform: Case-by-Case Assessment Advisable in Evaluating Coverage and Compliance (GAO/AIMD-94-57, July 28, 1994).
Federal Credit Programs: Agencies Had Serious Problems Meeting Credit Reform Accounting Requirements (GAO/AFMD-93-17, Jan. 6, 1993).
The first copy of each GAO report and testimony is free. Additional copies are $2 each. Orders should be sent to the following address, accompanied by a check or money order made out to the Superintendent of Documents, when necessary. VISA and MasterCard credit cards are accepted, also. Orders for 100 or more copies to be mailed to a single address are discounted 25 percent.
U.S. General Accounting Office P.O. Box 37050 Washington, DC  20013 Room 1100 700 4th St. NW (corner of 4th and G Sts. NW) U.S. General Accounting Office Washington, DC Orders may also be placed by calling (202) 512-6000  or by using fax number (202) 512-6061, or TDD (202) 512-2537.
Each day, GAO issues a list of newly available reports and testimony.  To receive facsimile copies of the daily list or any list from the past 30 days, please call (202) 512-6000 using a touchtone phone.  A recorded menu will provide information on how to obtain these lists.