United States General Accounting Office

GAO Report to the Director, Office of

Management and Budget

February 1998 MANAGING FOR

RESULTS

Experiences of

Selected Credit

Programs

GAO/GGD-98-41

GAO United States

General Accounting Office

Washington, D.C. 20548

General Government Division

B-275125 

February 19, 1998

The Honorable Franklin D. Raines

Director, Office of Management

 and Budget

Dear Mr. Raines:

The Government Performance and Results Act of 1993, also referred to as

“GPRA” or “the Results Act,” seeks to improve the efficiency, effectiveness,

and accountability of federal programs by establishing a system to set

goals for programs based on their intended purposes, measure the

performance of those programs, and use performance information to

improve results. As you know, if successfully implemented, the Results

Act will help agencies focus on how to improve their programs’

performance in achieving desired results.

The Act requires agencies to establish goals, performance measures, and

target levels of performance for fiscal year 1999. However, since the

passage of the Act, many agencies have been preparing for its

governmentwide implementation, beginning in the fall of 1997. In

June 1997, under the Results Act requirement that we report on agencies’

implementation of the Act, we concluded that governmentwide

implementation would be uneven.1

 That report included a brief discussion

of federal credit programs’ experiences in developing goals and measures

under the Results Act.

This report provides additional information and context on major credit

agencies’ efforts to implement the Act. It also describes the efforts of the

Federal Credit Policy Working Group—an interagency advisory group on

credit management policy—to develop common performance measures

for credit programs.2

 In July 1995, the Working Group established a task

force to develop and enhance credit program performance measures. The

report is addressed to you because of the Office of Management and

Budget’s (OMB) responsibility for overseeing agencies’ implementation of

the Act and because the Deputy Director of Management at OMB chairs the

Working Group.

1

The Government Performance and Results Act: 1997 Governmentwide Implementation Will Be

Uneven (GAO/GGD-97-109, June 2, 1997).

2

The Federal Credit Policy Working Group is a forum of OMB and federal credit agency officials that

was established in 1986 to provide advice in the formulation and implementation of credit

management policy.

Page 1 GAO/GGD-98-41 Experiences of Selected Credit Programs
B-275125 

Federal credit, or loan, programs are generally intended to provide

financial assistance to borrowers that the private sector will not serve. In

doing so, these programs are typically expected to help achieve certain

social and economic results, such as reaching underserved populations

and neighborhoods and supporting investment that is important to the

economy. As reported in the fiscal year 1998 budget, at the end of fiscal

year 1996, the outstanding face value of credit programs had reached

almost $1 trillion in direct loans and loan guarantees.3

For this report, our first objective was to identify goals and measures

established by the selected credit programs that related to the programs’

intended purposes and to determine whether the programs had set target

levels of performance for assessing their progress in achieving their

desired results. Under the Results Act, target levels of performance are to

enable decisionmakers to compare planned versus actual results achieved

for a given year. Our second objective was to identify the challenges

agency officials cited in developing performance information, including

goals and measures, for the selected programs and any approaches those

programs were taking to address those challenges. Our third objective was

to describe the status of the Working Group’s effort to develop common

performance measures for federal credit programs.

We selected for our review credit programs that varied in terms of type of

program (e.g., housing and education loans); mode of credit delivery (e.g.,

direct and guaranteed loans); and program size as measured by the

amount of outstanding loans. These programs were

• the Department of Veterans Affairs’ (VA) Loan Guaranty Program, which

provides home loans for veterans and active duty personnel;

• the Department of Education’s William D. Ford Direct Loan Program and

its Federal Family Education Loan Program (referred to in this report as

Education’s direct and guaranteed student loan programs, respectively),

which issue or guarantee loans, respectively, for students enrolled in

postsecondary education; and

• the Department of Agriculture’s (USDA) Single-Family Housing (SFH) direct

loan program and guaranteed loan program, which make or guarantee

loans to low- and moderate-income rural families, respectively, who are

unable to obtain credit from other sources.

3

Under direct loan programs, a federal agency generally makes a direct disbursement to an approved

borrower and then services and collects on the loan. Under guaranteed loan programs, federal

agencies rely on private sector lenders to originate and service loans within federal guidelines. In the

event of a default on a guaranteed loan, the government is liable for all or part of the interest and loan

principal.

Page 2 GAO/GGD-98-41 Experiences of Selected Credit Programs
B-275125 

According to data reported by OMB in the fiscal year 1998 budget and

agency data, of the total amount of federal credit outstanding in fiscal year

1996 for guaranteed loans ($805 billion) and direct loans ($165 billion),

these five programs represented about 32 and 18 percent, respectively.

Results in Brief In their efforts to implement the Results Act, the five credit programs

established goals and performance measures that appeared to be generally

related to the programs’ intended purposes. For example, to monitor its

performance in achieving one of its intended purposes, namely helping

veterans retain their homes, VA’s Loan Guaranty Program established the

“Foreclosure Avoidance Through Servicing (FATS) Ratio.” According to VA,

the FATS ratio is to measure the extent to which foreclosures would have

been greater had VA not pursued alternatives to foreclosure, such as

intervening with the holder of the loan on behalf of the borrower to set up

a repayment plan. The selected programs also set fiscal year 1998 targets

for most of their respective measures. Thus, if the selected programs

collect accurate corresponding data on their actual performance, they

should be able to monitor their progress in achieving desired results on

those measures and have fiscal year 1998 baseline data to use in setting

future targets for those measures.

Although the selected programs have established goals, measures, and

targets in their efforts to implement the Results Act, on the basis of our

discussions with agency officials, we identified three general challenges

the programs have been facing in developing performance information:

(1) a struggle in reaching consensus among stakeholders on the programs’

intended purposes, performance measures, and target levels of

performance; (2) difficulty in separating the effects of external forces from

program influences on results; and (3) a lack of relevant program

performance and financial baseline data. Agency officials also described

some approaches they were taking to address the challenges they have

been facing. For example, VA officials told us that they had difficulty

linking field office activities to the intended results of the program as

established by headquarters staff. To address this difficulty, VA

headquarters and field office staff worked together to develop the

program’s performance measures, and VA managers reached agreement on

target levels of performance.

According to OMB and the Working Group, comparing results using

common measures across credit programs allows program managers and

Page 3 GAO/GGD-98-41 Experiences of Selected Credit Programs
B-275125 

other decisionmakers to identify best practices among those programs

that have the potential for improving other credit programs’ performance.

By September 1996, a Working Group task force had proposed a set of

common performance measures for federal credit programs. Examples of

common financial and programmatic measures proposed by the task force

included total delinquent debt and the percentage of borrowers who were

pleased with the timeliness and quality of credit program service,

respectively. In June 1997, OMB reported that major credit agencies have

adopted some of the proposed measures.4

 Specifically, those agencies

have adopted some of the proposed common measures, in particular,

financial measures for meeting the annual budgeting and reporting

requirements of the Debt Collection Improvement Act and Federal Credit

Reform Act—budget and credit management reform legislation enacted to

address long-standing concerns about the program and financial

management of credit programs.

However, two general problems have limited the Working Group’s

progress in developing common performance measures for credit

programs. First, several credit programs lack relevant program

performance and financial baseline data. Second, the Working Group has

been unable to reach consensus on the appropriateness of decisionmakers

using some of the task force’s proposed measures to assess the

performance of individual credit programs or to compare that

performance against the performance of other credit programs. For

example, some programs give credit only to persons who are unable to

obtain credit from other sources; other programs give credit to anyone

who is entitled to the programs’ benefits, regardless of his or her access to

credit. Thus, officials in some credit agencies questioned whether it would

be appropriate to make comparisons among credit programs’ default rates,

because the financial characteristics of borrowers for each program may

be different.

The Working Group anticipated that agencies that administer credit

programs could include common financial and programmatic measures in

their annual performance plans and reports under the Results Act.

However, OMB does not intend to require credit agencies to adopt common

performance measures when consensus about the appropriateness of such

measures has not been achieved. We agree that OMB should not force the

use of common measures when concerns about their appropriateness

exist; but at the time of our review, the Working Group had not resolved

those concerns and had not decided how and when those concerns would

4

1997 Federal Financial Management Status Report and Five-Year Plan, OMB, June 25, 1997.

Page 4 GAO/GGD-98-41 Experiences of Selected Credit Programs
B-275125 

be addressed. Thus, it is unclear whether OMB and the credit agencies will

maintain their current level of attention to developing common measures.

Also unclear is the extent to which agencies that administer credit

programs will include common measures for those programs in their

annual performance plans that could provide useful information to

decisionmakers interested in making performance and cost comparisons.

We recognize the difficulty the Working Group is facing in reaching

consensus on common performance measures and that significant

differences in program characteristics may limit the usefulness of some

measures for broad, cross-program comparisons. However, we believe the

potential benefits that could be realized from developing common

performance measures, where appropriate, underscore the importance of

OMB and the credit agencies continuing their efforts to develop and reach

consensus on such measures. We recommend ways that the Director of

OMB can work with those agencies to achieve progress in this area. Our

recommendations are intended to serve a twofold purpose. First, they

would help ensure that the search for common measures continues. And

second, they would document the results of those efforts in a way that

would (1) permit further analysis directed at the identification of

additional common measures and (2) facilitate an understanding of any

limitations of using existing common measures to compare results across

credit programs.

Background In the 1990s, Congress put in place a statutory framework to address the

long-standing weaknesses in federal operations, improve federal

management practices, and provide greater accountability for achieving

results. This framework included as its essential elements the Results Act

and key financial management and information technology reform

legislation: the Chief Financial Officers Act of 1990 (CFO)—as expanded by

the Government Management Reform Act of 1994 (GMRA)—and the

Paperwork Reduction Act of 1995 and the Clinger-Cohen Act of 1996,

respectively. Taken together, these legislative initiatives seek to respond

to a need for accurate, reliable, and integrated budget, financial, and

program information for congressional and executive branch

decisionmaking, information that much of our prior work has shown to be

badly lacking.5

The goal-setting and performance measurement and improvement system

envisioned by the Results Act is the centerpiece of this framework and

5

GAO/GGD-97-109, June 2, 1997.

Page 5 GAO/GGD-98-41 Experiences of Selected Credit Programs
B-275125 

starts with the requirement that each executive agency develop and

periodically update a strategic plan to lay out its mission, long-term goals

and objectives, and strategies for achieving those goals and objectives.

Under the Results Act, the first of these plans were due by September 30,

1997. Next, each agency is to develop an annual performance plan,

beginning with the agency’s plan for fiscal year 1999, which covers each

program activity set forth in the agency’s budget. Agencies were to submit

their fiscal year 1999 annual performance plans to OMB in the fall of 1997

with their fiscal year 1999 budget requests and are to submit those plans to

Congress after the President’s fiscal year 1999 budget is provided to

Congress in February 1998. Among other things, agencies’ annual

performance plans are to contain their programs’ goals and measures for

fiscal year 1999. Finally, each agency is to report publicly on its programs’

performance, specifically on the degree to which the goals that are laid out

in the agency’s annual performance plan are being met and on actions it

plans to take to achieve unmet goals. The first of these reports, on

programs’ performance for fiscal year 1999, is due by March 31, 2000;

subsequent reports are due by March 31 for the years that follow. (For a

detailed description of the Results Act’s requirements, see app. I.)

The CFO Act and GMRA are intended to strengthen the reliability of agencies’

financial and programmatic performance information and the reporting of

such information by, among other things, having agencies develop better

performance measures and cost information and design results-oriented

reports that integrate budget, accounting, and program information.6

Finally, the Paperwork Reduction Act and Clinger-Cohen Act seek to help

agencies address long-standing weaknesses in their use of information

technology. Under this legislation, each agency is to better link its

technology plan and information technology use to achieving the agency’s

desired results.7

In addition, long-standing concerns about the program and financial

management of credit programs have prompted Congress to enact

important budget and credit management reform initiatives over the last

15 years. These initiatives include the Debt Collection Act of 1982 (DCA)

and amendments, the Debt Collection Improvement Act of 1996 (DCIA), and

6

For more information on the CFO Act, see Financial Management: Continued Momentum Essential to

Achieve CFO Act Goals (GAO/T-AIMD-96-10, Dec. 14, 1995).

7

For more information on the best practices that leading public and private organizations use to

effectively manage information technology, which served as a basis for the Paperwork Reduction Act

and Clinger-Cohen Act, see Executive Guide: Improving Mission Performance Through Strategic

Information Management and Technology—Learning From Leading Organizations (GAO/AIMD-94-115,

May 1994).

Page 6 GAO/GGD-98-41 Experiences of Selected Credit Programs
B-275125 

the Federal Credit Reform Act of 1990 (FCRA). DCA and DCIA are significant

pieces of credit management legislation designed to, among other things,

facilitate federal efforts to decrease delinquencies and increase

collections. Under DCA, agencies were to annually report to the Director of

OMB and the Secretary of the Treasury on the status of their debt collection

activities. Under DCIA, agencies are now to annually report on those

activities only to the Secretary of the Treasury, who in turn is to annually

report to Congress, beginning no later than 1999, on such activities

governmentwide.8 FCRA changed the budget treatment of direct loans and

loan guarantees made on or after October 1, 1991, to (1) facilitate more

accurate reporting by credit agencies on the full cost to the government in

the budget for the year in which the programs made or guaranteed the

loans so that executive branch and congressional decisionmakers might

consider such costs when making budget decisions and (2) permit

appropriate cost comparisons between direct and guaranteed credit

programs and between credit and noncredit programs intended to achieve

similar purposes.9

Guided by these legislative initiatives, since 1992, OMB has encouraged

federal credit agencies to improve their credit programs’ financial and

programmatic performance measures and to adopt a set of common

performance measures for those programs. According to OMB and the

Federal Credit Policy Working Group, common measures should help

credit program managers and other decisionmakers assess how similar

functions are performed and promote an atmosphere of cooperation and a

sharing of ideas among agency officials on how to improve the

performance of credit programs.

In response to the increasing importance being placed on agencies’

integration of performance measurement with budgeting, management

improvement, and overall agency accountability, the Working Group has

focused since July 1995 on developing common performance measures for

credit programs. To do so, the Working Group established a task force to

develop measures for credit programs consistent with the Results Act and

credit and financial management reform initiatives. In August 1997 we

reported that agencies’ annual planning under the Results Act could be

8

For more information on DCA (P.L. 97-365) and DCIA (P.L. 104-134), see Debt Collection: Improved

Reporting Needed on Billions of Dollars in Delinquent Debt and Agency Collection Performance

(GAO/AIMD-97-48, June 2, 1997).

9

Guidance for reporting of direct and guaranteed loans is provided in the Statement of Federal

Financial Accounting Standards No. 2, Accounting for Direct Loans and Loan Guarantees, which

essentially mirrors budgetary reporting under FCRA. For more information on the FCRA (P.L.

101-508), see Credit Reform: Case-by-Case Assessment Advisable in Evaluating Coverage and

Compliance (GAO/AIMD-94-57, July 28, 1994).

Page 7 GAO/GGD-98-41 Experiences of Selected Credit Programs
B-275125 

used as a vehicle for developing, where appropriate, common performance

measures for permitting future comparisons of similar programs’ results

and the methods those programs used to achieve those results.10

Scope and

Methodology

To meet our first objective—to identify goals and measures established by

the selected credit programs that related to the programs’ intended

purposes and determine whether the programs had set target levels of

performance for assessing their progress in achieving their desired

results—we compared the programs’ goals and measures to their

respective intended purposes, as identified by the programs or their

respective agencies. We interviewed agency officials about the programs’

intended purposes and asked those officials to comment on the

relationships we identified between the programs’ goals and measures and

intended purposes. To determine whether the programs had set target

levels of performance for assessing their progress in achieving their

desired results, we identified those measures for which the programs had

either (1) identified fiscal year 1998 targets or (2) reported prior year

baseline data for those measures and indicated how performance on those

measures was to change (i.e., increase or decrease) in fiscal year 1998

relative to the baseline.

The programs’ goals and performance measures that we used for our

assessment were those established by the programs as of May 1, 1997,

which, according to agency officials, were generally the same ones the

programs submitted with their respective agencies’ fiscal year 1998 budget

presentations to OMB and Congress. The programs also were proposing to

include these goals and measures for their agencies’ fiscal year 1999

annual performance plans under the Results Act. Although we make

observations about differences in goals and measures among the

programs, our review did not address the reasonableness of the processes

or methods the programs used to determine how to assess progress or

establish target levels toward achieving the programs’ intended purposes;

determine whether other, more appropriate measures existed; or evaluate

the feasibility of the targets the programs established.

To meet our second objective—to identify the challenges agency officials

cited in developing performance information, including goals and

measures, for the selected programs and any approaches those programs

were taking to address those challenges—we asked agency officials

10Managing For Results: Using the Results Act to Address Mission Fragmentation and Program Overlap

(GAO/AIMD-97-146, Aug. 29, 1997).

Page 8 GAO/GGD-98-41 Experiences of Selected Credit Programs
B-275125 

responsible for and involved in the development of goals and measures for

those programs to rate how great a challenge it was to perform each of 49

activities we identified as associated with developing performance

information. Examples of these activities included “determining a realistic

target level of performance for annual performance goals” and “developing

measures for assessing the net effect of the program compared with what

would have occurred in the absence of the program.” To identify these

activities, we referred to key steps and practices identified in our

Executive Guide: Effectively Implementing the Government Performance

and Results Act and other work that we had under way assessing the

challenges agencies were facing in implementing performance

measurement.11

We asked officials to use a five-point scale to rate the 49 activities, ranging

from “little to no” challenge to a “very great” challenge for the selected

credit programs in their agencies. For purposes of this report, we refer to

activities that any of the agency officials rated as a “great” or “very great”

challenge as significantly challenging activities. We then interviewed

agency officials about why they rated certain activities associated with

developing performance information as significantly challenging. We

analyzed their responses and related documentation to identify general

challenges that led agency officials to report those activities as

significantly challenging for the selected programs. To identify any

approaches those programs were taking to address the challenges they

have been facing, we talked with agency officials about such approaches

and analyzed agency documentation. We also considered prior and

ongoing work we have done on the efforts of VA, Education, USDA, and

other credit agencies to implement various credit and financial

management reforms.

To meet our third objective—to describe the status of the Working

Group’s effort to develop common performance measures for federal

credit programs—we reviewed various documentation from OMB and

members of the Working Group that described this effort and agency

officials’ views about those measures. We also talked to OMB officials and

members of the Working Group, including agency officials at those

agencies administering the programs we selected for our review, to obtain

their views of the common performance measures proposed. Our review

did not address the reasonableness of the processes or methods the

Working Group’s task force used in determining how to assess progress

11Executive Guide: Effectively Implementing the Government Performance and Results Act

(GAO/GGD-96-118, June 1996); and Managing For Results: Analytic Challenges in Measuring

Performance (GAO/HEHS/GGD-97-138, May 30, 1997).

Page 9 GAO/GGD-98-41 Experiences of Selected Credit Programs
B-275125 

for federal credit programs or determine whether other measures existed

that may be more appropriate. (For a more detailed discussion on our

objectives, scope, and methodology, see app. II.)

We did our work from September 1996 to December 1997, in Washington,

D.C., in accordance with generally accepted government auditing

standards. We requested comments on a draft of this report from the

Director of OMB, the Secretaries of Education and Agriculture, and the

Acting Secretary of VA or their designees. Comments were provided orally

by designees of the four agencies, and those comments are discussed at

the end of this report.

Selected Credit

Programs Established

Goals, Measures, and

Targets to Monitor

Their Progress

In their efforts to implement the Results Act, the five credit programs

established goals and performance measures that appeared to be generally

related to the programs’ intended purposes. In some cases, the programs

established goals or measures that addressed intermediate results that the

programs expected to lead to their intended purposes. Also, some of the

selected programs established a range of measures that should provide a

more complete picture of particular aspects of their performance related

to the programs’ intended purposes. Finally, the selected programs also set

fiscal year 1998 targets for most of their measures.

Goals and Performance

Measures Generally

Related to the Programs’

Intended Purposes

In their fiscal year 1998 budget presentations, each of the five credit

programs established goals and performance measures that appeared to

be generally related to the programs’ intended purposes. For example, to

monitor its performance in achieving one of its intended purposes, namely

helping veterans retain their homes, VA’s Loan Guaranty Program

established the Foreclosure Avoidance Through Servicing (FATS) Ratio

measure. According to VA, the FATS ratio is to provide data on the extent to

which foreclosures would have been greater had VA not pursued

alternatives to foreclosure, such as intervening with the holder of the loan

on behalf of the borrower to set up a repayment plan. For each of the

selected credit programs, appendix III shows the intended purposes

identified by those programs and their respective agencies and examples

of related goals and performance measures the programs established in

their fiscal year 1998 budget presentations.

In some cases, the programs’ goals or performance measures addressed

intermediate results that the programs expected to lead to an intended

purpose. For example, the performance measure that VA’s Loan Guaranty

Page 10 GAO/GGD-98-41 Experiences of Selected Credit Programs
B-275125 

Program established to monitor its performance in achieving its intended

purpose of helping veterans purchase homes was the percentage of

respondents to VA’s Lender Customer Satisfaction Survey who say they are

satisfied with their overall interaction with VA. According to VA,

“maximizing lender satisfaction with their dealings with VA employees . . .

will encourage lenders to participate in the program, expanding financing

opportunities for veterans.” In this way, the intermediate result of

increasing lender satisfaction could be expected to contribute to helping

veterans and active duty personnel purchase homes.

Similarly, Education’s direct student loan program established as

performance measures the rate of (1) institutional (i.e., school)

participation; (2) overall satisfaction of schools with the direct student

loan program; and (3) institutional retention in the program. The program

uses schools as the vehicles for providing loans to students and their

families. By providing a streamlined loan delivery system, the program

expects to attract schools to participate in the program. Further, by

satisfying participating schools, the program expects to encourage those

schools to stay as participants in the program. Thus, increases in schools’

participation, satisfaction, and retention in the program are intermediate

results that the program expects will lead to broader student access to

capital for postsecondary education.

Some of the selected programs established a range of measures to provide

a more complete picture of particular aspects of their performance related

to their intended purposes. For example, VA’s Loan Guaranty Program

established two performance measures for monitoring the timeliness of

issuing a certificate of eligibility, which is related to its intended purpose

of treating all veterans in a timely manner. These measures were (1) the

percentage of veterans responding to a VA veteran survey who say they are

satisfied with the time it takes VA to certify veterans’ eligibility for a home

loan; and (2) the average time VA calculated it took to issue a certificate of

eligibility, which is to supplement the survey data. Similarly, Education

established several performance measures for assessing its guaranteed

and direct student loan programs’ performance toward successfully

managing the programs in a cost-effective manner. These measures

included the programs’ lifetime gross dollar default rates; lifetime net

default rates (i.e., loss rate); annual delinquency rates; per unit

administrative costs; and the annual collection rates.

Page 11 GAO/GGD-98-41 Experiences of Selected Credit Programs
B-275125 

The Credit Programs Set

Fiscal Year 1998 Targets

for Most of Their Measures

The Results Act defines a performance goal as the target level of

performance expressed as a tangible, measurable objective against which

achievement is to be compared. Thus, annual performance goals should

consist of two parts: (1) the performance measure that represents the

specific characteristic of the program that the program uses to gauge its

performance and (2) the annual target level of performance to be achieved

during a given fiscal year for the measure. The Results Act also requires

each agency to report to the President and Congress annually, beginning

for fiscal year 1999, on the degree to which the agency is meeting its

annual performance goals. Thus, under the Act, an agency is to monitor

and report on its actual performance during the year compared to the

targets it had established for its performance measures for that year.

As shown in table 1, we found that the selected programs set fiscal year

1998 targets for most of their respective measures. These included

measures for which the programs either (1) set fiscal year 1998 target

levels of performance; or (2) reported prior year baseline data for those

measures and indicated how performance on those measures was to

change (i.e., increase or decrease) in fiscal year 1998 relative to their

baselines. Thus, if the selected programs collect accurate corresponding

data on their actual performance, they should be able to monitor their

progress in achieving desired results on those measures and have fiscal

year 1998 baseline data to use in setting future targets for those measures.

For example, one of the fiscal year 1998 performance goals set by

Education’s guaranteed student loan program is that the “Level of [overall

school] satisfaction will meet or exceed the level of school satisfaction

measured last year, [where] 67 percent of the schools reported

satisfaction.” However, for other measures, the program did not set targets

that could be used to monitor their progress on those measures. For

example, Education’s guaranteed student loan program did not establish a

fiscal year 1998 target for its annual delinquency rate measure. The

program reported that this measure will provide information on the dollar

amount of loans “past due” as a percentage of dollars in repayment and

that baseline data for the measure will be developed as the definition of

“past due” is finalized.

Page 12 GAO/GGD-98-41 Experiences of Selected Credit Programs
B-275125 

Table 1: Number of Performance

Measures With Fiscal Year 1998

Targets

Number of performance measures

Program Total

With FY 1998

targetsa

VA’s Loan Guaranty Program (guaranteed

home loans) 17 13

USDA’s SFH direct and guaranteed loan

programsb 10 9

Education’s direct student loan program 13 9

Education’s guaranteed student

loan program 14 8

aThe number of measures with fiscal year 1998 targets include those measures where the

programs either (1) set fiscal year 1998 target levels of performance; or (2) reported prior year

baseline data for those measures and indicated how performance on those measures was to

change (i.e., increase or decrease) in fiscal year 1998 relative to their baselines.

bAccording to a USDA official, USDA established one set of performance measures and targets to

monitor the combined performance of its direct and guaranteed SFH loan programs as well as its

Housing Repair Loan Program and its Very-Low-Income Repair Grant Program.

Source: Selected programs’ measures and targets as of May 1, 1997, which, according to

agency officials, were generally the same ones the programs submitted with their respective

agencies’ fiscal year 1998 budget presentations to OMB and Congress.

The Credit Programs

Have Been Facing

Three General

Challenges to

Developing

Performance

Information

On the basis of agency officials’ responses to questions on developing

performance information for the selected programs, we identified three

general challenges those programs have been facing: (1) a struggle in

reaching consensus among stakeholders on the programs’ intended

purposes, performance measures, and target levels of performance;

(2) difficulty in separating the effects of external forces from program

influences on results; and (3) a lack of relevant program performance and

financial baseline data. Agency officials also described some approaches

they were taking to address the challenges they have been facing.

A Struggle in Reaching

Consensus Among

Stakeholders on the

Programs’ Intended

Purposes, Performance

Measures, and Target

Levels of Performance

As we have noted in a previous report, because the interests of a

program’s stakeholders can and often do differ significantly, full

agreement among those stakeholders on all aspects of the program’s

performance is relatively uncommon.12 However, our past and current

work also has shown that although it is difficult to get stakeholders to

reach agreement, stakeholder involvement can help an agency identify

results-oriented performance measures and set realistic target levels.13 For

example, VA officials said that they had difficulty reaching consensus with

12GAO/GGD-96-118, June 1996.

13GAO/GGD-97-109, June 2, 1997.

Page 13 GAO/GGD-98-41 Experiences of Selected Credit Programs
B-275125 

internal program stakeholders at the agency’s field offices. This difficulty

concerned how the activities of the field offices would be linked to

achieving the intended results of VA’s Loan Guaranty Program as

established by headquarters staff. To address this difficulty, VA

headquarters and field office staff worked together to develop the

program’s performance measures. They said the program then brought

together key VA headquarters and field managers to reach agreement on

target performance levels for linking field office activities to the intended

results of the program.

VA’s Loan Guaranty Program also struggled with trying to reach consensus

with OMB. According to a VA official, OMB suggested that the program

establish “outcome-oriented” performance measures, where feasible,

which could provide data on the extent to which the program is helping

veterans achieve a higher rate of homeownership. However, VA officials

said that the program is an entitlement program (i.e., veterans receive the

benefit regardless of need as a reward for their service); and it is not clear

whether increasing homeownership among veterans is a primary intended

purpose of the program. According to the officials, the more appropriate

performance measures for assessing the program’s performance and

holding it accountable are those for monitoring how well VA is delivering

the benefit (e.g., satisfying veterans and keeping program costs down).

To address OMB’s suggestion, the program established a homeownership

assistance measure in its fiscal year 1998 budget presentation, which is to

provide data on the percentage of veterans surveyed who said they would

not have been able to purchase any home or would have had to purchase a

less expensive home without a VA guaranteed loan. However, the program

did not identify the homeownership assistance measure as a performance

measure. Instead, the program identified this measure as a measure of

workload and other program data, which a VA official said was to provide

“contextual program information,” rather than information for gauging the

program’s progress in achieving its intended purposes and holding the

program accountable. The VA official added that the homeownership

assistance measure is an “imperfect measure” because of its reliance on

self-reporting by veterans.

Similarly, at the time of our interviews with USDA officials, they said

internal stakeholders were grappling with what the appropriate results for

their SFH loan programs were. According to those officials, the struggle of

trying to reach consensus among those stakeholders contributed to why

they rated as a significantly challenging activity “developing measures for

Page 14 GAO/GGD-98-41 Experiences of Selected Credit Programs
B-275125 

assessing the net effect of the program.” One agency official said that the

programs’ intended purposes were “putting people [who are unable to get

credit from other sources] in homes”; thus, to the extent that they put such

borrowers in homes, the programs are having a net impact. However,

another agency official suggested that the programs’ intended purposes

also should include “improving the quality of life among rural residents”

and “improving housing conditions and the economy in a given community

or state.” He said that measures for gauging the programs’ progress toward

such purposes would attempt to collect data on, for example, the extent to

which putting people in homes is improving the quality of life among rural

residents.

USDA officials also said it is difficult to balance stakeholders’ interests.

They said that they are expected to increase program service while also

reducing program costs and minimizing default rates. However, they said

this is difficult because the SFH programs were designed to offer credit to a

population that the private sector would consider high risk. Specifically, to

be eligible for a SFH direct loan, a borrower must have a family income that

is “very low” to “low” (i.e., a family income under 80 percent of the median

income in the area); and the borrower must be unable to get credit from

any other source. Therefore, the program’s target population may be more

likely to default. At the time of our review, USDA, VA, and Education were

working with stakeholders in and outside of the selected programs,

including OMB and Congress, to reach consensus with those stakeholders

on the most appropriate goals and measures for the programs.

Difficulty in Separating the

Effects of External Forces

From Program Influences

on Results

The efforts of federal agencies often are but one factor among many that

may influence whether, and the degree to which, their programs achieve

their intended results. Our past and current work has found that many

agencies have been challenged to separate out the influence that their

program activities have had on the achievement of program results when

those results also could have been influenced by external forces.14

Agency officials from all five of the selected credit programs reported as a

significantly challenging activity “separating the impact of the credit

programs’ activities from the impact of other factors external to those

programs but contributing to the results achieved.” They generally cited

economic trends; the role of third parties in helping the programs provide

loans; and the existence of other federal financial aid programs (e.g.,

14GAO/GGD-97-109, June 2, 1997.

Page 15 GAO/GGD-98-41 Experiences of Selected Credit Programs
B-275125 

grants) as examples of forces external to their programs that can affect

program results.

For example, the foreclosure rate could be viewed as a measure related to

the VA Loan Guaranty Program’s intended purpose of helping veterans and

active duty personnel retain homes. However, VA reported that external

forces, such as interest rates, unemployment, and the general state of the

economy, can influence the foreclosure rate. A VA official said that because

of such external forces, it has been difficult to confidently attribute a

change in the foreclosure rate to the program’s activities and thus view it

as a valid measure of the program’s performance.

VA officials said that the program attempted to address this problem in

1993 but was not successful. Specifically, a VA program official said that at

the encouragement of OMB, the program attempted to develop a model to

help the program estimate its foreclosure rate and monitor its

performance. However, he said when the program implemented the

model, it significantly overestimated the number of VA foreclosures and

thus was not an adequate model for determining the external forces that

could affect the rate. The VA official said that because of the many external

forces that could affect the number of foreclosures, it was unclear if the

model could be adjusted to help it adequately predict foreclosures and

whether—given the cost of potentially making many adjustments to the

model—the value of doing this was worth the additional cost. Therefore,

the program took another approach. The program established a surrogate

measure, the FATS ratio (which, as mentioned earlier, is to provide data on

the extent to which foreclosures would have been greater had VA not

pursued alternatives to foreclosure), for monitoring its performance in

assisting veterans to avoid foreclosures. The program views the FATS ratio

as a more valid measure than the foreclosure rate for assessing the

program’s performance in helping veterans and active duty personnel

retain homes.

USDA officials also said that separating the impact of their SFH guaranteed

and direct loan programs’ activities from the impact of other external

forces on, for example, the quality of life for rural residents is exceedingly

difficult. These officials said that the quality of rural residents’ lives could

be affected negatively by other, unrelated events, such as borrowers’

incurring health problems or financial hardship, or the closing of a military

base eliminating jobs in the area. They said that although providing

single-family housing loans may help to improve the quality of life among

those rural residents, such improvement could also be due to other

Page 16 GAO/GGD-98-41 Experiences of Selected Credit Programs
B-275125 

external forces, such as home loans being provided by other federal

housing loan programs (e.g., programs administered by VA or the

Department of Housing and Urban Development).

Similarly, Education officials told us that loans that are issued or

guaranteed through their agency’s direct or guaranteed student loan

programs are among several types of financial aid that Education offers to

help ensure access to postsecondary education. They said that students’

college participation and completion rates can be affected by borrowers’

eligibility for loans through the direct or guaranteed loan programs as well

as by such external forces as the eligibility of borrowers for the other

types of financial aid assistance provided by Education (e.g., grants); the

extent of parental support for the borrowers attending school; and the

borrowers having to financially support a family. Although Education

established student participation and completion rates as performance

measures, it did so to monitor the combined performance of its financial

aid programs instead of the specific performance of either the direct or

guaranteed student loan programs on those results.

Education officials said that an approach the agency was taking to better

understand the determinants of college enrollment—including financial

aid obtained through direct loans, guaranteed loans, grants, or other

financial aid programs—was contracting for a study of the effects of

financial aid, including aid provided by these programs, and various

external forces on this result. According to an Education official, this

study is expected to be completed in early 1998. We believe the findings

from this study may help to inform any future program evaluations

assessing the impact of Education’s financial aid programs on results

compared to the impact of external forces. In a prior report, we discussed

how impact evaluations can help an agency confidently attribute the

achievement of intended results to one or more of its programs by

providing information on the extent to which those programs contributed

to the results achieved relative to the impact of external forces.15

Lack of Relevant Baseline

Data

As we reported in June 1997, our prior work has shown that baseline and

trend data on past performance can help agencies set realistic target levels

of performance for their programs given the past performance of those

programs.16 However, we also noted that because agencies often did not

focus on having results-oriented performance information in the past, they

15GAO/GGD-97-109, June 2, 1997.

16GAO/GGD-97-109, June 2, 1997.

Page 17 GAO/GGD-98-41 Experiences of Selected Credit Programs
B-275125 

generally have not collected such data. Thus, they have not had all of the

baseline and trend data they believed they needed to set goals. Further,

credit agencies, including VA, USDA, and Education, generally have had

difficulty producing reliable performance data, particularly financial data,

which executive and legislative branch decisionmakers need to make

well-informed decisions.

Agency officials from all five of the credit programs said that a lack of

baseline data was why they rated one or more of the following activities as

significantly challenging: “developing objective, quantifiable, and

measurable annual program performance goals”; “determining a realistic

target level of performance”; “developing unit cost information for the

programs’ outputs”; and “developing unit cost information for the

programs’ outcomes.” For example, VA officials said that some of their

performance measures for their Loan Guaranty Program were new, and

baseline data were thus not available on those measures. Consequently, VA

did not have data on past performance to use in setting some of the

program’s fiscal year 1998 target levels of performance and reported that

those targets were “to be determined.”

Education officials also attributed the challenges they had in determining

realistic target levels of performance for Education’s direct student loan

program to a lack of baseline data. According to those officials, the

program had not been in existence long enough to have historical data on

many of the program’s measures to use in setting fiscal year 1998 targets.

They said that a lack of historical data for the direct loan program was a

particular problem in terms of predicting borrower repayment behavior,

since few borrowers had yet entered the repayment phase. Thus, to set the

target for the program’s default rate measure, Education used historical

data for the same measure established for Education’s guaranteed student

loan program.

Similarly, USDA officials said the agency’s SFH programs did not have

information systems to collect data on some performance measures, such

as the number of loans made in targeted geographic areas. Thus, according

to those officials, the programs did not have an informed basis on which to

set fiscal year 1998 target levels of performance for those measures and

did not include them in their fiscal year 1998 budget submission to OMB.

Rather, the program included substitute measures for which the programs

had information systems in place to collect the data (e.g., the number of

rural families with improved or more suitable housing conditions).

Page 18 GAO/GGD-98-41 Experiences of Selected Credit Programs
B-275125 

Lack of Reliable Financial Data Our prior and ongoing work and that of agencies’ internal or independent

auditors have found that some credit agencies still have difficulty, despite

numerous years of experience, in producing reliable financial data, such as

credit programs’ subsidy rates—the estimated cost to the government

from direct loans and loan guarantees.17 For example, USDA and Education

received a disclaimer of an opinion from internal and independent

auditors, respectively, on their fiscal year 1996 financial statements. In

part, this was due to those agencies not being able to provide the data

needed to (1) accurately reflect the cost to the government and (2) permit

appropriate cost comparisons between credit and noncredit programs.

VA received an unqualified opinion on its fiscal year 1996 financial

statements from VA’s Office of Inspector General (OIG). However, the OIG

audit, which included a review of the Loan Guaranty Program, found that

the program did not reliably accumulate the financial information needed

to comply with federal financial accounting standards, identified

significant errors that required financial statement adjustments, and

identified other errors where data compiled manually did not always

reconcile with original source amounts.

If successfully implemented, the CFO Act will help credit agencies resolve

long-standing problems with data reliability. Further, in passing the

Results Act, Congress emphasized that the usefulness of agencies’

performance data depends, to a large degree, on the reliability of those

data. Therefore, the Results Act requires agencies to describe in their

annual performance plans the means to be used to verify and validate

performance data. We have suggested in prior reports that such

information, including information about the reliability of credit agencies’

performance data, could be equally important for those agencies to

disclose in their reports to ensure report users of the quality of that data.18

One area in particular need of attention is the development of reliable

financial information on the full cost and unit cost of a program, which is

an integral part of measuring that program’s efficiency and cost

effectiveness.19 An essential step in developing such information is the

17Federal Credit Programs: Agencies Had Serious Problems Meeting Credit Reform Requirements

(GAO/AFMD-93-17, Jan. 6, 1993); GAO/T-AIMD-96-10, Dec. 14, 1995; and GAO/AIMD-97-48, June 2,

1997.

18GPRA Performance Reports (GAO/GGD-96-66R, Feb. 14, 1996); and GAO/AIMD-97-48, June 2, 1997.

19The full cost of a program is calculated as the sum of all resources used by the program that

indirectly or directly contributed to producing the program’s products and services, including

resources provided by entities outside of the program. A credit program’s unit costs are the total costs

per unit of loan issued or guaranteed, or of service provided by that program.

Page 19 GAO/GGD-98-41 Experiences of Selected Credit Programs
B-275125 

identification of individual program costs, such as direct labor. In that

regard, unit cost information can be particularly useful in identifying

trends and determining key cost drivers of the program. Agency officials

from all five of the credit programs we reviewed said they have been

challenged to develop unit cost information for the programs’ outputs and

outcomes. They generally cited difficulties in allocating basic cost data to

specific programs as a reason for this challenge. For example, USDA

officials explained that its field offices are involved in administering the

SFH direct and guaranteed loan programs as well as other USDA programs.

They said difficulties in separating data on labor costs for the various

programs, for example, have contributed to the challenge they have faced

in developing unit costs for the SFH programs.

Similarly, VA officials said that developing meaningful unit cost

information for the Loan Guaranty Program has been a significantly

challenging activity due to the lack of an adequate methodology, including

VA’s inability to separate the data on the actual costs for the program, such

as labor costs, from the costs for the several other programs that VA field

offices administer. A VA official said that because of the difficulty it has

had in isolating program cost data, the program uses a “very loose”

process to calculate unit costs, which involves dividing the various

resource levels authorized for the program, such as staffing levels (i.e.,

authorized full-time equivalents)20 by the activity level during the year

(e.g., number of loans guaranteed).

New accounting standards developed by the Federal Accounting

Standards Advisory Board (FASAB) require federal agencies subject to the

CFO Act and Results Act to collect relevant and reliable data on the full

costs of carrying out a mission or producing products or services.

Although these standards were scheduled to be effective for all federal

programs beginning with fiscal year 1997, because of serious shortfalls in

agencies’ cost accounting systems, FASAB extended the date by 1 year to

fiscal year 1998. The standards took effect on October 1, 1997. Credit

agency officials said their respective agencies were establishing

information systems to collect needed cost data. Specifically, VA officials

said their agency was looking at the use of activity-based costing (ABC) to

develop more meaningful unit cost information.21 According to an

20A full-time equivalent consists of one or more employed individuals who collectively complete 2,080

work hours in a given year. Therefore, both one full-time employee and two half-time employees equal

one full-time equivalent.

21ABC is a methodology that assigns costs to products or services on the basis of the resources they

consume. ABC gives visibility to how effectively resources are being used and how all relevant

activities contribute to the cost of a product or service.

Page 20 GAO/GGD-98-41 Experiences of Selected Credit Programs
B-275125 

Education official, his agency also is developing a system that should be

able to provide at least some data on unit costs by 1999. USDA reported in

its fiscal year 1998 budget presentation that it was working to develop the

data for its SFH direct and guaranteed loan programs’ measure on the “cost

of housing a family per recipient household.”

Progress in

Developing Common

Performance

Measures Has Been

Limited

According to OMB and the Working Group, comparing results using

common measures across credit programs allows program managers and

other decisionmakers to identify best practices among those programs

that have the potential for improving other credit programs’ performance.

In addition, the Working Group anticipated that agencies that administer

credit programs could include such measures in their annual performance

plans and reports under the Results Act. By September 1996, a Working

Group task force had proposed a set of common financial and

programmatic performance measures for federal credit programs. Some of

the proposed measures—namely, financial measures for meeting the

annual budgeting and reporting requirements of DCIA and FCRA—have been

adopted by major credit agencies.

However, two general problems have limited the Working Group’s

progress in developing common performance measures for credit

programs. First, as previously noted, several credit programs lack relevant

program performance and financial baseline data. Second, the Working

Group has been unable to reach consensus on the appropriateness of

decisionmakers using some of the task force’s proposed measures to

assess the performance of individual credit programs or to compare that

performance against the performance of other credit programs. In

addition, OMB does not intend to require credit programs to adopt common

performance measures when consensus about the appropriateness of such

measures has not been achieved.

Common Performance

Measures for Credit

Programs Have Been

Proposed, and Some Have

Been Adopted

In July 1995, the Working Group established a task force to develop

common performance measures that could help agencies and other

decisionmakers make relevant comparisons of the results of credit

programs. By September 1996, the task force had proposed a set of

common financial and programmatic measures in the following four areas:

(1) Financial performance. Measures in this area include total receivables;

total delinquent debt; default rates; actual versus projected subsidy rates;

Page 21 GAO/GGD-98-41 Experiences of Selected Credit Programs
B-275125 

and administrative costs, such as the costs of extending credit and

servicing loans.

(2) Program performance in achieving desired loan characteristics.

Measures in this area include the percentage of loans going to borrowers

who would otherwise not have access to private credit and the percentage

of borrowers pleased with the timeliness and quality of credit program

service.

(3) Program effects on society. Measures in this area include (1) intended

effects, such as “supporting investment important to the economy” as

monitored by, for example, the amount and quality of low-income housing

financed (home loan programs) and business investment financed

(business loan programs); and (2) unintended effects, such as borrowers

accumulating excessive debt burden.

(4) Program “additionality.” Measures in this area indicate the results

achieved by the program by providing financial assistance to borrowers

that private markets will not serve. An example of a measure in this area is

the net increase in homeownership as a result of the program

supplementing versus substituting for private financing.

OMB and the Working Group’s task force also have encouraged credit

programs to establish, where appropriate, program-specific measures and

explanatory data for helping to ensure a complete assessment of their

programs’ performance and to explain the results to users of performance

information.

In its June 1997 1997 Federal Financial Management Status Report and

Five-Year Plan, OMB and the CFO Council reported that major credit

agencies have adopted the proposed common financial measures for

assessing their debt collection activities.22 For example, credit agencies

are to collect data on total receivables, total delinquent debt, and total

collections for their credit programs. According to an OMB official, these

and other common measures that the task force proposed in the area of

financial performance are measures that major credit agencies adopted for

meeting the annual budgeting and reporting requirements of DCIA and FCRA.

However, he said that those agencies have not yet adopted measures that

the task force proposed under the other three areas—measures for

22Authorized by the CFO Act, the CFO Council is an interagency, governmentwide body that comprises

the CFOs and Deputy CFOs of the 24 largest federal agencies and senior OMB and Department of

Treasury officials. The Council was established to address critical crosscutting financial issues.

Page 22 GAO/GGD-98-41 Experiences of Selected Credit Programs
B-275125 

monitoring and improving a credit program’s (1) performance in achieving

desired loan characteristics, (2) effects on society, and (3) additionality.

Two General Problems

Have Limited Progress in

Developing Common

Performance Measures

Although the common financial measures proposed by the task force have

been adopted, two general problems have limited the Working Group’s

progress in developing common performance measures. First, as

previously noted, several credit programs lack relevant program

performance and financial baseline data. Second, the Working Group has

been unable to reach consensus on the appropriateness of decisionmakers

using some of the task force’s proposed measures to assess the

performance of individual credit programs or to compare that

performance against the performance of other credit programs. For

example, some programs give credit only to persons who are unable to

obtain credit from other sources; other programs give credit to anyone

who is entitled to the programs’ benefits, regardless of his or her access to

credit. Thus, officials in some credit agencies questioned whether it would

be appropriate to make comparisons among the default rates of credit

programs, because the financial characteristics of borrowers for each

program may be different.

Relevant Data on Several of the

Common Measures Proposed

Are Lacking

Agency officials told the task force that for several of the proposed

measures, their agencies’ credit programs generally did not collect data

relevant to those measures, collected incomplete data, or did not routinely

collect such data. Such measures included the percentage of borrowers

who were pleased with the timeliness and quality of credit program

service and measures for monitoring the effects of credit programs on

society and program additionality. For example, common measures

proposed by the task force for monitoring the performance of home loan

programs and business loan programs in “supporting investment

important to the economy”—one of four intended effects on society the

task force proposed—included the amount and quality of low-income

housing financed and business investment financed, respectively. Officials

from VA, the Department of Housing and Urban Development (HUD), and

the Small Business Administration told the task force that the credit

programs at their respective agencies generally did not collect such data.

Moreover, even when data existed, our prior and ongoing work and audits

by credit agencies’ inspectors general and others have consistently

disclosed serious weaknesses in agencies’ systems, which has affected the

reliability of data that are used to account for and manage credit

programs.23

23GAO/AFMD-93-17, Jan. 6, 1993; GAO/T-AIMD-96-10, Dec. 14, 1995; and GAO/AIMD-97-48, June 2, 1997.

Page 23 GAO/GGD-98-41 Experiences of Selected Credit Programs
B-275125 

The Working Group Has Been

Unable to Reach Consensus on

the Appropriateness of Some of

the Proposed Measures

Consistent with the views of OMB and the Working Group, our work over

the last few years has recognized that common performance measures for

similar programs can provide important information for permitting

comparisons of the results of those programs and the methods used to

achieve those results.24 Such information could help program managers

identify credit program performance gaps; set improvement goals; improve

credit program processes; and inform other decisionmakers, such as OMB

and Congress. However, we found that members of the Working Group

have been unable to reach consensus on the appropriateness of some of

the common performance measures proposed by the task force.

Specifically, officials in some agencies questioned whether data collected

from some of the proposed common measures would be meaningful for

assessing the performance of their agencies’ credit programs. For

example, VA and Education officials said that their agencies’ Loan

Guaranty Program and direct and guaranteed student loan programs,

respectively, are entitlement programs in which the government is

obligated to give credit to anyone who qualifies for the programs’ benefits,

regardless of his or her access to credit.25 Thus, these officials questioned

whether a common programmatic measure proposed by the task force,

“percent of loans or guarantees originated going to borrowers who would

otherwise not have access to private credit,” was meaningful for their

programs. As mentioned earlier, to be eligible for a USDA SFH direct loan, a

borrower must have a family income that is “very low” to “low,” and the

borrower must be unable to get credit from any other source. A USDA

official said that the target and result for this proposed measure would

always be 100 percent; thus, he also questioned whether the measure was

meaningful for that program.

In addition, for certain measures, officials in some credit agencies

questioned whether the data collected would be appropriate to use in

making comparisons among different credit programs’ performance

results, such as the programs’ default rate, because the financial

characteristics of borrowers for each program may be different. For

example, USDA officials said that because the borrower must be unable to

obtain private credit, the program’s target population may be more likely

to default. Thus, according to USDA officials, it may be inappropriate to

compare, for example, the actual default rate of the SFH direct loan

24GAO/AIMD-97-146, Aug. 29, 1997; GAO/AIMD-97-48, June 2, 1997; and GAO/GGD-96-118, June 1996.

25Under entitlement or mandatory programs, the government is obligated by provisions of law to make

a loan directly or guarantee a loan made by a private lender to an eligible borrower.

Page 24 GAO/GGD-98-41 Experiences of Selected Credit Programs
B-275125 

program with the actual default rate of the VA Loan Guaranty Program,

which is an entitlement program.

We agree that common measures need to be carefully explained to help

ensure that significant program differences are properly interpreted.

However, it is not clear that such differences outweigh the potential

usefulness of common measures. Credit agencies have generally agreed

with our suggestion that they provide explanatory information where

necessary in agency reports to portray program differences more

appropriately and help users of performance information understand the

reported performance.26

Also related to the problem of comparable data, we recently reported that

agencies’ use of inconsistent definitions for their programs’ measures

could hamper decisionmakers’ use of data collected from those measures

in planning, comparing performance, and reporting on performance

achieved.27 Our June 1997 report noted that some credit agencies differ in

how they classify previously delinquent debt on which borrowers are

currently making payments.28 Some reclassify such debt as “current,” but

others keep it in a delinquent category regardless of the current payment

status. Although such classification practices may be suitable within an

agency, they make it difficult to compare agency performance or aggregate

data for similar programs. For example, VA loans maintain their delinquent

status until the delinquency is repaid or written off. Conversely, a home

loan program within HUD reclassified single-family delinquent loans to a

current repayment status when borrowers complied with forbearance

terms, which typically included making partial mortgage payments for up

to 3 years. In our June report, we recommended that the Department of

the Treasury’s Financial Management Service work with major credit

agencies and OMB to help those agencies consistently report on delinquent

debt or disclose their inconsistencies. Treasury, OMB, and the major credit

agencies generally agreed with that recommendation, and the agencies

commented that consistent application of governmentwide debt collection

reporting criteria is essential.

Similarly, we observed that when consistent definitions do not exist

among credit programs, it is difficult to know whether results reported

would be readily comparable. For example, one of the common

26GAO/AIMD-97-48, June 2, 1997.

27GAO/GGD-97-109, June 2, 1997; and GAO/AIMD-97-48, June 2, 1997.

28GAO/AIMD-97-48, June 2, 1997.

Page 25 GAO/GGD-98-41 Experiences of Selected Credit Programs
B-275125 

programmatic measures proposed by the task force for monitoring loan

characteristics was the “percent of borrowers who are pleased with the

timeliness and quality of credit program service.” However, for this

proposed task force measure, it is unclear what is meant by “timeliness”

and “quality” of service, because each credit program may have a different

interpretation. Education’s guaranteed and direct student loan programs,

for example, established a broad measure of “overall borrower

satisfaction”; VA’s Loan Guaranty Program established more specific

borrower satisfaction measures—the percent of respondents surveyed

who will say (1) they are satisfied with their contact with VA; (2) they are

satisfied with the time it took to obtain a Certificate of Eligibility; and

(3) their loan did not take longer to process than expected as a result of a

delay blamed on VA (which is to help the program monitor when program

staff may need to work with the lender in identifying reasons for the delay

in loan processing time, toward improving performance on this measure).

Thus, it is unclear whether the data collected would be comparable if this

proposed measure of the task force were used by all credit programs.

The Working Group anticipated that agencies that administer credit

programs could include common financial and programmatic measures in

their annual performance plans and reports under the Results Act.

According to OMB’s Senior Advisor for Cash and Credit Management, at an

upcoming meeting of the Working Group OMB intends to ask members

whether they want to continue to address their concerns about adopting

additional common performance measures for credit programs. However,

he said that OMB is not taking a prescriptive role in directing agencies’

performance measurement activities. Rather, he said the administration

wants agencies to take the initiative on such activities. According to this

official, if members choose not to address those concerns, OMB does not

intend to require credit programs to adopt common performance

measures when consensus about the appropriateness of such measures

has not been achieved. Thus, it is unclear when, or if, agencies that

administer credit programs will include common measures for those

programs in their annual performance plans under the Results Act.

Conclusions The credit programs we reviewed have established goals and performance

measures that appeared to be generally related to the programs’ intended

purposes and set target levels of performance for most of their respective

measures. In addition, our prior work suggests that the development of

common measures, where appropriate, can provide important information

for permitting comparisons of similar programs’ results and the methods

Page 26 GAO/GGD-98-41 Experiences of Selected Credit Programs
B-275125 

they used to achieve those results. We have suggested that agencies’

annual performance plans and reports under the Results Act can serve as

suitable vehicles for developing such measures and providing such

information. The Working Group has focused on developing common

financial and programmatic performance measures for credit programs

and anticipated that agencies that administer credit programs could

include such measures in their annual performance plans and reports

under the Results Act.

However, two general problems have limited the Working Group’s

progress in developing common performance measures for credit

programs: (1) the lack of relevant program performance and financial

baseline data for several credit programs and (2) the inability among

members of the Working Group to reach consensus on the

appropriateness of some of the proposed measures.

OMB does not intend to require credit agencies to adopt common

performance measures when consensus about the appropriateness of such

measures has not been achieved. We agree that OMB should not force the

use of common measures when concerns about their appropriateness

exist. However, at the time of our review, the Working Group had not

resolved those concerns and had yet to decide how and when those

concerns would be addressed; thus, it is unclear whether OMB and the

credit agencies will maintain their current level of attention to developing

common measures. Also unclear is the extent to which agencies that

administer credit programs will include common measures for those

programs in their annual performance plans that could provide useful

information to decisionmakers interested in making performance and cost

comparisons.

We recognize the difficulty the Working Group is facing in reaching

consensus on common performance measures and that significant

differences in program characteristics may limit the usefulness of some

measures for broad, cross-program comparisons. However, we believe the

potential benefits that could be realized from developing common

performance measures, where appropriate, underscore the importance of

OMB and the credit agencies continuing their efforts to develop and reach

consensus on such measures. In addition, because the development and

use of performance measures, especially common measures that can be

used across programs and agencies, are in the early stages of

implementation, it will be especially useful for decisionmakers to evaluate

early experiences to identify successful, as well as unsuccessful,

Page 27 GAO/GGD-98-41 Experiences of Selected Credit Programs
B-275125 

approaches. To that end, documented information on the measures

considered, how they are used and should be interpreted, and how they

can be improved will be helpful to agencies in further achieving the

purposes of the Results Act.

Recommendations to

the Director of OMB

Building on recommendations and suggestions we have made in prior

reports, we recommend that the Director of OMB sustain OMB’s efforts to

work with major credit agencies to use annual performance planning

under the Results Act as a vehicle for developing common performance

measures across credit programs, where appropriate. In doing so, we

recommend that beginning with those agencies’ fiscal year 2000 annual

performance plans, the Director of OMB require each agency that

administers credit programs to identify in their plans

(1) performance measures the agency is using for its credit program(s)

that are the same as those used by other credit programs and the strengths

and limitations of using those measures to make performance and cost

comparisons among those programs; and

(2) what actions, if any, are being taken or could be taken to refine the

agency’s performance measurement efforts to address the identified

limitations to using existing measures to make performance and cost

comparisons across credit programs.

These or some comparable requirements would serve a twofold purpose.

First, they would help ensure that the search for common measures

continues. And second, they would document the results of those efforts

in a way that would (1) permit further analysis directed at the

identification of additional common measures and (2) facilitate an

understanding of any limitations of using existing common measures to

compare results across credit programs.

Agency Comments

and Our Evaluation

On December 11, 1997, we requested comments on a draft of this report

from the Secretaries of Education and Agriculture, the Acting Secretary of

VA, and the Director of OMB or their designees. On December 17, 1997, the

liaison to GAO from USDA’s Rural Development mission area, which

administers the SFH direct and guaranteed loan programs, said the

Department generally agreed with the draft report’s factual material,

conclusions, and recommendations. A rural development official later

provided minor technical suggestions, which we included in the report as

Page 28 GAO/GGD-98-41 Experiences of Selected Credit Programs
B-275125 

appropriate. Similarly, on December 19, 1997, the liaison to GAO from the

Department of Education also said the Department concurred with the

draft report’s general findings and later provided minor technical

suggestions, which we included in the report as appropriate.

On January 8, 1998, the VA liaison to GAO said the Department also

generally agreed with the draft report’s factual material, conclusions, and

recommendations, except for that part of the recommendation requiring

credit agencies to include a discussion about common measures in their

fiscal year 1999 annual performance plans. This comment was consistent

with the comment provided to us on January 6, 1998, by a representative

of VA’s Performance Analysis Service within VA’s Office of Budget.

Specifically, that representative told us that VA would not likely have time

to provide a meaningful discussion about common measures in its fiscal

year 1999 plan because VA (1) had yet to achieve consensus among major

stakeholders on the common measures, (2) had not done the analysis

necessary to provide the discussion about common performance measures

that our draft report recommended, and (3) is to submit the agency’s fiscal

year 1999 performance plan to Congress within the next few weeks. OMB

officials had a similar comment on the feasibility of requiring a discussion

about common measures in credit agencies’ fiscal year 1999 performance

plans, which we discuss later in this section.

In response to this comment from both VA and OMB officials, we removed

from the recommendation our suggestion that such a discussion be

included in credit agencies’ fiscal year 1999 annual performance plans.

However, we retained the recommendation that the Director of OMB

require credit agencies to include a discussion about common measures in

those agencies’ fiscal year 2000 annual performance plans and subsequent

performance plans. We continue to believe that using annual performance

planning under the Results Act as a vehicle for developing and discussing

common performance measures across credit programs, where

appropriate, could provide decisionmakers with important information

and help agencies further achieve the purposes of the Results Act. Such

information could be useful to decisionmakers in comparing similar

programs’ results and the methods they used to achieve those results and

for understanding how such measures are used, should be interpreted, and

could be improved.

Further, in a conversation with us on December 17, 1997, a representative

of VA’s Loan Guaranty Service commented that developing information

systems to collect data from common performance measures may be

Page 29 GAO/GGD-98-41 Experiences of Selected Credit Programs
B-275125 

costly and that before VA developed systems to collect such data, the costs

of doing so should be weighed against the benefits. We agree that

collecting needed data from common measures is a problem that credit

agencies face, as discussed in the draft report. We also have suggested that

annual performance plans provide agencies with the opportunity to alert

Congress to the problems they have had or anticipate having in collecting

needed data, including the cost and data quality trade-offs associated with

various collection strategies.29 The representative of VA’s Loan Guaranty

Service also provided minor technical suggestions, which we included in

this report where appropriate.

On January 7, 1998, we met with the OMB Senior Advisor for Cash and

Credit Management and the Senior Advisor to the Deputy for Management,

who said the report would serve as a valuable tool and resource in OMB’s

continuing efforts to encourage and work with major credit agencies to

effectively implement the Results Act. However, those officials cautioned

that based on OMB’s experiences in working with major credit agencies to

draft their fiscal year 1999 performance plans and to identify appropriate,

results-oriented common performance measures for credit programs,

developing and reaching consensus on credit program performance

measures will continue to be difficult, time-consuming, and iterative. They

said that those experiences and OMB’s review of major credit agencies’

fiscal year 1999 performance plans suggested to them that the priority of

those agencies at this point in the implementation of the Results Act needs

to be on ensuring the quality of the performance goals and measures for

their individual credit programs. According to these OMB officials, in

developing program-specific as well as common performance measures

for credit programs, major credit agencies need to continue working to

reach consensus among the key stakeholders of the agencies’ credit

programs and to develop information systems for collecting needed

performance data, which will be challenging. Moreover, the OMB officials

said that agencies are concluding the preparation of their fiscal year 1999

annual performance plans that will accompany the President’s budget

submission to Congress in February 1998. The officials said that those

plans reflect budget, policy, and programmatic decisions already made in

the course of preparing the budget. Thus, the officials believed it would

not be feasible for the Director of OMB to direct credit agencies to include

in their fiscal year 1999 plans the discussion about common measures that

our draft report recommended.

29Managing for Results: Agencies’ Annual Performance Plans Can Help Address Strategic Planning

Challenges (GAO/GGD-98-44, Jan. 30, 1998).

Page 30 GAO/GGD-98-41 Experiences of Selected Credit Programs
B-275125 

OMB officials said that given the challenges of developing appropriate,

results-oriented common performance measures for credit programs, it is

unclear when or if such measures could be adopted by major credit

agencies. OMB’s Senior Advisor for Cash and Credit Management suggested

that as the Working Group’s efforts advance, two common results-oriented

performance measures that may be considered appropriate for agencies

that administer credit programs are (1) the number of loans a program

made that are repaid successfully and (2) the percentage of customers

satisfied with the program. However, he said that developing measures

that would help isolate a credit program’s contribution to achieving a

particular common result from the contribution of external factors may

not be possible for all credit programs. The OMB officials also noted that

performance information is just one key factor among many that will go

into decisionmaking on management and budget policy issues.

The OMB officials told us that OMB will ensure in calendar year 1998 that

developing common results-oriented performance measures across credit

programs is a priority agenda item for discussions among the Working

Group members. They said that as part of these discussions, major credit

agencies could share with one another their experiences in developing

their individual fiscal year 1999 performance plans and their congressional

committees’ reactions to those plans. On the basis of agencies’

experiences and congressional reactions, the OMB officials said they

believe that OMB and the major credit agencies would have a better

foundation from which to discuss common performance measures for the

agencies’ credit programs. According to the OMB officials, this experience

will provide OMB and the agencies with the basis for determining the

feasibility of incorporating common measures and a discussion about such

measures into future annual performance plans, where appropriate.

We believe that OMB’s planned approach to use major credit agencies’

fiscal year 1999 performance planning efforts as the foundation for

discussions among these agencies on common performance measures is

responsive to the intent of our recommendation. In this regard, in

response to OMB officials’ comments about the feasibility of the Director of

OMB requiring credit agencies to include in their fiscal year 1999 annual

performance plans the discussion about common measures that our draft

report recommended, as mentioned earlier, we removed the reference to

those agencies including such a discussion in their fiscal year 1999 plans.

However, as also mentioned earlier, we retained the recommendation for

those agencies to include such a discussion in their fiscal year 2000 annual

performance plans and subsequent performance plans. Such discussions

Page 31 GAO/GGD-98-41 Experiences of Selected Credit Programs
B-275125 

can serve as vehicles for highlighting many of the other cautionary notes

that the OMB officials raised, such as the difficulties in developing

measures that seek to isolate a credit program’s unique contributions to a

particular result.

We are sending copies of this report to the Majority Leader, House of

Representatives; the Chairman and Ranking Minority Member, Committee

on Government Reform and Oversight, House of Representatives; the

Chairman and Ranking Minority Member, Committee on Governmental

Affairs, United States Senate; the Secretaries of Education and Agriculture;

the Acting Secretary of VA; and the heads of agencies that administer credit

programs and are represented on the Federal Credit Policy Working

Group. We also will make copies available to others on request.

The major contributors to this report are listed in appendix IV. Please

contact me on (202) 512-8676 if you have any questions.

Sincerely yours,

J. Christopher Mihm

Associate Director, Federal Management

 and Workforce Issues

Page 32 GAO/GGD-98-41 Experiences of Selected Credit Programs
Page 33 GAO/GGD-98-41 Experiences of Selected Credit Programs
Contents

Letter 1

Appendix I 

Overview of the

Results Act

36

Appendix II 

Objectives, Scope,

and Methodology

38

Appendix III 

Selected Credit

Programs’ Intended

Purposes and

Examples of Goals

and Measures They

Presented in Their

Fiscal Year 1998

Budgets

42

Appendix IV 

Major Contributors

52

Related GAO Products 56

Table Table 1: Number of Performance Measures With Fiscal Year 1998

Targets

13

Page 34 GAO/GGD-98-41 Experiences of Selected Credit Programs
Contents

Abbreviations

ABC activity-based costing

CFO Chief Financial Officers Act

DCA Debt Collection Act

DCIA Debt Collection Improvement Act

FASAB Federal Accounting Standards Advisory Board

FATS Foreclosure Avoidance Through Servicing

FCRA Federal Credit Reform Act

GMRA Government Management and Reform Act

GPRA General Performance and Results Act

HUD Department of Housing and Urban Development

OIG Office of Inspector General

OMB Office of Management and Budget

SFH Single-Family Housing

USDA U.S. Department of Agriculture

VA Department of Veterans Affairs

Page 35 GAO/GGD-98-41 Experiences of Selected Credit Programs
Appendix I 

Overview of the Results Act

The Results Act is the primary legislative framework through which

agencies will be required to set strategic goals, measure performance, and

report on the degree to which goals were met. It starts by requiring each

federal agency to develop a strategic plan that covers a period of at least 5

years and includes the agency’s mission statement; identifies the agency’s

long-term strategic goals; and describes how the agency intends to achieve

those goals through its activities and through its human, capital,

information, and other resources. The first strategic plans that the Act

required agencies to develop were to be completed by September 30, 1997.

Also, the act requires each agency to submit to the Office of Management

and Budget (OMB), beginning for fiscal year 1999, an annual performance

plan. The first annual performance plans were to be submitted to OMB in

the fall of 1997. The annual performance plan is to provide the direct

linkage between the strategic goals outlined in the agency’s strategic plan

and what managers and employees do day to day. In essence, this plan is

to contain the annual performance goals the agency will use to gauge its

progress toward accomplishing its strategic goals and identify the

performance measures the agency will use to assess its progress. Also, OMB

will use individual agencies’ performance plans to develop an overall

federal government performance plan that OMB is to submit annually to

Congress with the president’s budget, beginning for fiscal year 1999.

The Results Act also requires that each agency submit an annual report to

the president and to the appropriate authorization and appropriations

committees of Congress on program performance for the previous fiscal

year (copies are to be provided to other congressional committees and to

the public upon request). The first of these reports, on program

performance for fiscal year 1999, is due by March 31, 2000; and subsequent

reports are due by March 31 for the years that follow. However, for fiscal

years 2000 and 2001, agencies’ reports are to include performance data

beginning with fiscal year 1999. For each subsequent year, agencies are to

include performance data for the year covered by the report and 3 prior

years.

Finally, in crafting the Results Act, Congress recognized that managerial

accountability for results is linked to managers having sufficient flexibility,

discretion, and authority to accomplish desired results. The Act authorizes

agencies to apply for managerial flexibility waivers in their annual

performance plans beginning with fiscal year 1999. The authority of

agencies to request waivers of administrative procedural requirements and

controls is intended to provide federal managers with more flexibility to

Page 36 GAO/GGD-98-41 Experiences of Selected Credit Programs
Appendix I 

Overview of the Results Act

structure agency systems to better support program goals. The

nonstatutory requirements that OMB can waive under the Results Act

generally involve the allocation and use of resources, such as restrictions

on shifting funds among items within a budget account. Agencies must

report in their annual performance reports on the use and effectiveness of

any managerial flexibility waivers that they receive.

Page 37 GAO/GGD-98-41 Experiences of Selected Credit Programs
Appendix II 

Objectives, Scope, and Methodology

Our first objective was to identify goals and measures established by the

selected credit programs that related to the programs’ intended purposes

and determine whether the programs had set target levels of performance

for assessing their progress in achieving their desired results. Under the

Results Act, target levels of performance are to enable a comparison of

planned versus actual results achieved for a given year. Our second

objective was to identify the challenges agency officials cited in

developing performance information, including goals and measures, for

the selected programs and any approaches those programs were taking to

address those challenges. Our third objective was to describe the status of

the Working Group’s effort to develop common performance measures for

federal credit programs.

For our review, we selected a nonrandom, purposive sample of five federal

credit programs at three agencies. These programs were the Department

of Veterans Affairs’ Loan Guaranty Program; the Department of

Education’s William D. Ford Direct Loan Program and its Federal Family

Education Loan Program (referred to in this report as Education’s direct

and guaranteed student loan programs, respectively); and the Department

of Agriculture’s Single-Family Housing (SFH) direct loan program and

guaranteed loan program. We selected for our review credit programs that

varied in terms of type of program (e.g., housing and education loans);

mode of credit delivery (e.g., direct and guaranteed loans); and program

size as measured by the amount of outstanding loans. According to data

reported by OMB in the fiscal year 1998 budget and agency data, of the total

amount of federal credit outstanding in fiscal year 1996 for guaranteed

loans ($805 billion) and direct loans ($165 billion), these five programs

represented about 32 and 18 percent, respectively. The smallest and

largest of the guaranteed loan programs accounted for slightly less than

1/2 percent and 19 percent, respectively, of the fiscal year 1996 credit

outstanding in loan guarantees governmentwide. Similarly, the smallest

and largest of the direct loan programs held 7 percent and 11 percent,

respectively, of the fiscal year 1996 credit outstanding in direct loans

governmentwide. Because of the small and nonrandom nature of our

sample, our observations and analyses are not generalizable to other

federal credit programs.

To address the first part of our first objective (i.e., to identify goals and

measures established by the selected credit programs that related to the

programs’ intended purposes), we compared goals and measures

established by the programs as of May 1, 1997, to the programs’ intended

purposes as identified by those programs or their respective agencies.

Page 38 GAO/GGD-98-41 Experiences of Selected Credit Programs
Appendix II 

Objectives, Scope, and Methodology

According to agency officials, these goals and measures were generally the

same ones the programs submitted to OMB and Congress with their

respective agencies’ fiscal year 1998 budget presentations. The programs

also were proposing to include these goals and measures for their

agencies’ fiscal year 1999 annual performance plans under the Results Act.

To determine whether a goal or measure was related to a program’s

intended purposes, we reviewed available agency and credit program

documentation for (1) a description of the program’s intended purposes

and (2) a discussion that reasonably related that particular goal or

measure to the program’s intended purposes. When agency documentation

did not contain such a discussion, we examined the wording and

considered the meaning of each program’s goals and measures and

compared them to the program’s intended purposes to identify

relationships between them. We also interviewed agency officials about

the programs’ intended purposes and asked those officials to comment on

the relationships we identified between the programs goals and measures

and intended purposes.

To address the second part of our first objective (i.e., to determine

whether the selected programs set fiscal year 1998 target levels of

performance for assessing their progress in achieving desired results), we

identified those measures for which the programs had either (1) identified

fiscal year 1998 targets or (2) reported prior year baseline data on those

measures and indicated how performance on those measures was to

change (i.e., increase or decrease) in fiscal year 1998 relative to the

baseline. Our review did not address the reasonableness of the processes

or methods the programs used to determine how to assess progress or

establish target levels toward achieving the programs’ intended purposes;

determine whether other, more appropriate, measures existed; or evaluate

the feasibility of the targets the programs established.

To address the first part of our second objective (i.e., to identify the

challenges agency officials cited facing in developing performance

information for the selected programs), we developed a data collection

instrument that listed 49 activities that we identified as being associated

with developing performance information. Examples of these activities

included “determining a realistic target level of performance for annual

performance goals” and “developing measures for assessing the net effect

of the program compared with what would have occurred in the absence

of the program.” To identify these activities, we referred to key steps and

practices identified in our Executive Guide: Effectively Implementing the

Page 39 GAO/GGD-98-41 Experiences of Selected Credit Programs
Appendix II 

Objectives, Scope, and Methodology

Government Performance and Results Act and other work that we had

under way assessing the challenges agencies were facing in implementing

performance measurement.1

We sent the instrument to agency officials responsible for and involved in

the development of goals and measures for the selected programs and

asked those officials to rate, using a five-point scale from “little or no”

challenge to a “very great” challenge, how great a challenge each of the 49

activities was to perform for those programs. Officials also could indicate

that they had not engaged in a particular activity. For purposes of this

report, we refer to activities that any of the agency officials rated as a

“great” or “very great” challenge as significantly challenging activities. We

then interviewed those officials to discuss why they rated certain activities

as significantly challenging. We analyzed their responses and related

documentation to identify general challenges that led officials to report

those activities as significantly challenging for the selected programs.

To address the second part of our second objective (i.e., to identify any

approaches that the selected programs were taking to address the

challenges they identified to developing performance information), we

interviewed the agency officials who responded to our data collection

instrument to obtain information on the approaches they were taking to

address those activities they had identified as significantly challenging and

analyzed agency documentation. We also asked these officials to comment

on the way in which we described the approaches the programs were

taking to address the challenges in developing performance information

that they had identified. We also considered prior and ongoing work we

have done on the efforts of VA, Education, USDA, and other credit agencies

to implement various credit and financial management reforms.

To address our third objective (i.e., to describe the status of the Working

Group’s effort to develop common performance measures for federal

credit programs), we reviewed various documentation from OMB and

members of the Working Group that described its effort to develop

common performance measures for credit programs and the views of

agency officials on those measures. We also talked to OMB officials and

members of the Working Group, including agency officials at those

agencies administering the programs we selected for our review, to obtain

their views on the common performance measures proposed. Our review

did not address the reasonableness of the processes or methods the

Working Group’s task force used in determining how to assess progress

1

GAO/GGD-96-118, June 1996; and GAO/HEHS/GGD-97-138, May 30, 1997.

Page 40 GAO/GGD-98-41 Experiences of Selected Credit Programs
Appendix II 

Objectives, Scope, and Methodology

for federal credit programs or determine whether other measures existed

that may be more appropriate.

Page 41 GAO/GGD-98-41 Experiences of Selected Credit Programs
Appendix III 

Selected Credit Programs’ Intended

Purposes and Examples of Goals and

Measures They Presented in Their Fiscal

Year 1998 Budgets

Program

Type of loans

provided

Intended purpose(s) as identified by

the selected program and its

respective agency

VA’s Loan Guaranty

Program

Guaranteed home

loans

Help veterans and active duty

personnel purchase and retain homes

in recognition of their service to the

nation.

Treat all veterans and other participants

in the program in a courteous,

responsive, and timely manner.

Page 42 GAO/GGD-98-41 Experiences of Selected Credit Programs
Appendix III 

Selected Credit Programs’ Intended

Purposes and Examples of Goals and

Measures They Presented in Their Fiscal

Year 1998 Budgets

Examples related to the program’s intended

purpose Link between the intended

purpose(s) and goal and

measure in example General goal(s)

Performance

measure(s) Explanation of relationship

Help veterans obtain loans

to purchase homes.

Assist veterans in

obtaining home

mortgage loans.

Percent of respondents

to the Lender Customer

Satisfaction Survey who

say they are satisfied

with their overall

interaction with VA.

The general goal defines, in part, the intended

purpose of helping veterans purchase homes. VA

identified this goal and measure as related on the

basis of the following rationale: Lender satisfaction

addresses an intermediate result expected to lead

to achieving this goal and intended purpose.

Specifically, maximizing lenders’ satisfaction with

their dealings with VA employees is expected to

encourage lenders to participate in the program,

which is expected to expand financing

opportunities (i.e., available mortgage loans) for

helping veterans purchase homes.

Help veterans retain their

homes.

Assist veterans in

avoiding foreclosures.

Foreclosure Avoidance

Through Servicing

(FATS) Ratio.

The general goal defines this intended purpose,

and VA identified this goal and measure as

related. The FATS Ratio is to provide data on the

extent to which foreclosures would have been

greater had VA not pursued alternatives to

foreclosure, such as intervening with the holder of

the loan on behalf of the borrower to set up a

repayment plan.

Treat all veterans in a

courteous and responsive

manner.

Assist veterans in

obtaining home

mortgage loans.

Percent of respondents

to the Veteran Customer

Satisfaction Survey who

say they are satisfied

with their overall contact

with VA.

VA identified this goal and measure as related.

This measure defines, in part, “assisting” veterans

in obtaining homes because it addresses, in

summary form, treating veterans in a courteous

and responsive manner.

Treat all veterans in a timely

manner.

Assist veterans in

obtaining home

mortgage loans.

Percent of respondents

to the Veteran Customer

Satisfaction Survey who

say they are satisfied

with the time it took to

obtain a Certificate of

Eligibility.

VA identified this goal and measure as related.

This measure defines, in part, “assisting” veterans

in obtaining homes because it addresses treating

veterans in a timely manner. Before a lender

issues a VA guaranteed mortgage loan, VA must

certify to the lender that the borrower is a veteran

who is eligible for a loan guaranteed by VA.

(continued)

Page 43 GAO/GGD-98-41 Experiences of Selected Credit Programs
Appendix III 

Selected Credit Programs’ Intended

Purposes and Examples of Goals and

Measures They Presented in Their Fiscal

Year 1998 Budgets

Program

Type of loans

provided

Intended purpose(s) as identified by

the selected program and its

respective agency

VA’s Loan Guaranty

Program (continued)

Guaranteed home

loans

Operate in the most efficient manner

possible to minimize costs and ensure

the best use of the taxpayer’s dollar.

USDA’s SFH direct

loan program

Direct home loans Provide homeownership loans to:

— very-low-income and low-income

families (i.e., families who have incomes

under 80% of median) who do not own

adequate housing and cannot obtain

credit from other sources;

— eligible farm owners for housing for

themselves or for farm laborers.

Provide “supervised credit” to many

rural borrowers to help them maintain

their homes in times of financial crises

through credit counseling, workout

agreements, and moratoriums.

USDA’s SFH

guaranteed loan

program

Guaranteed home

loans

Provide homeownership opportunities

to moderate-income rural residents (i.e.,

between 80% and 115% of median).

Utilize private lenders to provide

mortgages to borrowers who would be

unable to obtain credit without the

guarantee.

Page 44 GAO/GGD-98-41 Experiences of Selected Credit Programs
Appendix III 

Selected Credit Programs’ Intended

Purposes and Examples of Goals and

Measures They Presented in Their Fiscal

Year 1998 Budgets

Examples related to the program’s intended

purpose Link between the intended

purpose(s) and goal and

measure in example General goal(s)

Performance

measure(s) Explanation of relationship

Operate in the most efficient

manner possible to minimize

costs and ensure the best

use of the taxpayer’s dollar.

Efficient credit and

program management.

Percent of early defaults

of all loans originated.

The goal describes the strategy and addresses

intermediate results expected to lead to

minimizing costs and ensuring the best use of the

the taxpayer’s dollar. On the basis of the following

rationale, the measure, which VA identified as

related to this goal, addresses this intended result:

Once a loan is put into default status, collection

activities are initiated, which can ultimately include

the foreclosure of the borrower’s home. Such

activities are costly to the government and, thus,

the taxpayer. Early defaults (i.e., defaults within 6

months of origination) are more likely than a later

default to be due to a deficiency in the

underwriting of the loans.a Thus, efficient credit

and program management is expected to reduce

early defaults and, therefore, minimize costs and

ensure the best use of the taxpayer’s dollar.

Provide homeownership

loans to eligible rural

borrowers.

No general goal for this

intended purpose was

identified by the program

or agency.b

Number of rural families

with improved or more

suitable housing

conditions.

This measure includes the number of direct loans

provided, or issued, to eligible borrowers. Thus,

the rationale for the relationship is that the

program provides loans, or “housing

opportunities,” for improved or more suitable

housing to very-low- to low-income families and

farm owners who do not own adequate housing

and who cannot obtain credit from other sources.

Provide “supervised credit”

to help them (i.e., rural

borrowers) maintain their

homes.

No general goal for this

intended purpose was

identified by the program

or agency.b

Percentage of borrowers

current.

The rationale for the relationship between this

measure and helping rural borrowers maintain

their homes follows: Rural borrowers encountering

financial crises are likely to miss scheduled

payments and thus are not likely to be counted as

“current” in making scheduled loan payments.

When a borrower is viewed as not current, the

borrower’s loan is delinquent and put into default

status. Once a loan is put into default status,

collection activities are initiated, which can

ultimately include the foreclosure of the borrower’s

home.

Utilize private lenders to

provide mortgages to

eligible, rural resident

borrowers and thus provide

homeownership opportunities

to eligible rural residents.

No general goal for this

intended purpose was

identified by the program

or agency.b

Number of rural families

with improved or more

suitable housing

conditions.

This measure includes the number of guaranteed

loans provided to eligible borrowers. Thus, the

rationale for this relationship is that the program

utilizes private lenders to provide mortgages, or

homeownership opportunities, for improved or

more suitable housing to moderate-income rural

residents who would be unable to obtain credit

without the guarantee.

(continued)

Page 45 GAO/GGD-98-41 Experiences of Selected Credit Programs
Appendix III 

Selected Credit Programs’ Intended

Purposes and Examples of Goals and

Measures They Presented in Their Fiscal

Year 1998 Budgets

Program

Type of loans

provided

Intended purpose(s) as identified by

the selected program and its

respective agency

Education’s direct

student loan program

Direct postsecondary

education loans

Provide students and their families with

federally sponsored loans—using a

streamlined student loan system that

simplifies loan access and allows for

flexible repayment—to help borrowers

meet increasing postsecondary

education costs and to reduce taxpayer

costs.

Page 46 GAO/GGD-98-41 Experiences of Selected Credit Programs
Appendix III 

Selected Credit Programs’ Intended

Purposes and Examples of Goals and

Measures They Presented in Their Fiscal

Year 1998 Budgets

Examples related to the program’s intended

purpose Link between the intended

purpose(s) and goal and

measure in example General goal(s)

Performance

measure(s) Explanation of relationship

Provide students and their

families with federally

sponsored loans to help

borrowers meet increasing

postsecondary education

costs.

Maintain a high level of

borrower satisfaction.

Rate of borrowers’

overall satisfaction with

the program during the

first year.c

The rationale for the relationship between this goal

and measure and helping borrowers meet

increasing postsecondary education costs is that: 

— providing loans using a streamlined system will

simplify borrowers’ access to postsecondary

education loans, thereby helping to satisfy those

borrowers in simplifying their ability to meet

increasing postsecondary education costs; and 

— using a streamlined system that allows for

flexible repayment will help ease borrowers’ debt

burden, thereby also helping to satisfy those

borrowers in helping them meet increasing

postsecondary education costs.

Help eligible borrowers meet

increasing postsecondary

education costs and reduce

taxpayer costs.

Provide flexible

repayment options so

that debt burden is

eased and defaults are

minimized.

Cohort default rate. The general goal defines the program’s strategy

and intended purpose, and Education identified

this goal and measure as related. The rationale for

the relationship between the measure and

intended purpose is that borrowers whose loans

are placed into a default status are likely

encountering debt burden and thus having

difficulty meeting postsecondary costs. Once a

loan is put into default status, debt collection

activities are initiated, which are costly to the

government and thus the taxpayer. In addition, by

using a streamlined system that allows for flexible

repayment, the program will help ease borrowers’

debt burden, thereby helping those borrowers

meet increasing postsecondary education costs.

(continued)

Page 47 GAO/GGD-98-41 Experiences of Selected Credit Programs
Appendix III 

Selected Credit Programs’ Intended

Purposes and Examples of Goals and

Measures They Presented in Their Fiscal

Year 1998 Budgets

Program

Type of loans

provided

Intended purpose(s) as identified by

the selected program and its

respective agency

Education’s direct

student loan program

(continued)

Direct postsecondary

education loans

Ensure access to capital for

postsecondary education.

Successfully implement and manage

the direct student loan program.

Education’s

guaranteed student

loan program

Guaranteed

postsecondary

education loans

Provide students and their families with

federally sponsored loans to help meet

increasing postsecondary education

costs.

Ensure access to capital for

postsecondary education.

Page 48 GAO/GGD-98-41 Experiences of Selected Credit Programs
Appendix III 

Selected Credit Programs’ Intended

Purposes and Examples of Goals and

Measures They Presented in Their Fiscal

Year 1998 Budgets

Examples related to the program’s intended

purpose Link between the intended

purpose(s) and goal and

measure in example General goal(s)

Performance

measure(s) Explanation of relationship

Ensure access to capital for

postsecondary education.

Continue to provide a

streamlined loan delivery

system to attract schools

to participate.

Maintain a high level of

school satisfaction.

Institutional direct loan

program participation

rate.

Rate of overall [school]

satisfaction with the

direct student loan

program.

Institutional retention

rate.

This goal describes the strategy for achieving this

intended purpose. In addition, this goal and

measure describe an intermediate result the

program expects to lead to achieving this

intended purpose. Specifically, the program uses

schools as the vehicles for providing loans to

students and their families. By providing a

streamlined loan delivery system, the program

expects to increase schools’ participation, which

is an intermediate result the program expects will

lead to broader student access to capital for

postsecondary education.

This goal and these measures describe

intermediate results the program also expects to

lead it to achieving this intended purpose. As

previously noted, the program uses schools as the

vehicles for providing loans to students and their

families. Increasing participating schools’

satisfaction is an intermediate result that the

program expects will lead to encouraging those

schools to stay as participants in the program;

and increasing schools’ retention in the program is

an intermedate result that the program expects

will lead to broader student access to capital for

postsecondary education.

Successfully implement and

manage the direct student

loan program.

Continue to provide

strong fiscal

management of the

program.

Number of internal

control program

weaknesses identified in

Education’s financial

statement audit.

The goal defines the intended purpose, and

Education identified this goal and measure as

related.

Provide students and their

families with federally

sponsored loans to help

them meet increasing

postsecondary education

costs.

Maintain a high level of

borrower satisfaction

from the time of loan

origination through the

end of the repayment

period. 

Overall rate of borrower

satisfaction with the

guaranteed student loan

program.

The rationale for the relationship between this goal

and measure and the intended purpose is that by

providing students and their families with loans

(through the guaranteeing of loans made by

private lenders), the program will be helping to

satisfy those borrowers in their ability to meet

increasing postsecondary education costs.

Ensure access to capital for

postsecondary education.

Ensure access to

guaranteed loans in a

changing marketplace.

Number of borrower

complaints.

The goal defines the intended purpose, and

Education identified the goal and measure as

related. The number of borrower complaints is to

include data on the number of borrowers who

complain to Education about being denied access

to a guaranteed postsecondary education loan.

(continued)

Page 49 GAO/GGD-98-41 Experiences of Selected Credit Programs
Appendix III 

Selected Credit Programs’ Intended

Purposes and Examples of Goals and

Measures They Presented in Their Fiscal

Year 1998 Budgets

Program

Type of loans

provided

Intended purpose(s) as identified by

the selected program and its

respective agency

Education’s

guaranteed student

loan program

(continued)

Guaranteed

postsecondary

education loans

Successfully deliver and manage the

guaranteed student loan program in an

efficient and cost-effective manner to

help students and their parents meet

postsecondary education costs.

Page 50 GAO/GGD-98-41 Experiences of Selected Credit Programs
Appendix III 

Selected Credit Programs’ Intended

Purposes and Examples of Goals and

Measures They Presented in Their Fiscal

Year 1998 Budgets

Examples related to the program’s intended

purpose Link between the intended

purpose(s) and goal and

measure in example General goal(s)

Performance

measure(s) Explanation of relationship

Successfully deliver and

manage the guaranteed

student loan program in an

efficient and cost-effective

manner.

Provide a program that

is cost-effective for the

taxpayer.

Annual delinquency rate.

Annual collection rate.

Per unit administrative

costs.

The goal defines the intended purpose, and

Education identified this goal and these measures

as related. Monitoring the program’s performance

in, for example, managing the program’s loan

portfolio debt and the program’s administrative

costs addresses providing a program that is

cost-effective for the taxpayer, or successfully

delivering and managing the program in an

efficient and cost-effective manner.

aUnderwriting involves a detailed credit analysis, based on credit information furnished by the

borrower, such as employment history, salary, and financial statements, before a loan is granted.

bIn commenting on a draft of this report, a USDA Rural Development official said that in the

agency’s September 30, 1997, strategic plan, Rural Development established a general goal for

the SFH direct and guaranteed loan programs that related to the programs’ intended purposes

and examples of performance measures. Specifically, according to this official, the general goal

established was “Rural Development will improve the quality of life of rural residents by providing

access to technical assistance, capital and credit for quality housing, and modern, essential

community facilities.” He explained that the number of rural families with improved or more

suitable housing conditions performance measure is related to that part of the goal of “improving

the quality of life of rural residents by providing access to capital and credit for quality

housing”; and the percentage of borrowers current performance measure is related to that part of

the goal of “improving the quality of life of rural residents by providing access to technical

assistance.”

cIn commenting on a draft of this report, the Education liaison to GAO told us that through the

agency’s Direct Loan evaluation, the agency is annually measuring borrower satisfaction (i.e.,

in-school borrowers, out-of-school borrowers, and parent borrowers) with both the direct and

guaranteed student loan programs. Therefore, according to this official, the part of the measure,

“rate of borrowers’ overall satisfaction with the [direct student loan] program during the first year,”

that refers to monitoring that satisfaction “during the first year” should be dropped.

Source: Selected credit programs and their respective agencies.

Page 51 GAO/GGD-98-41 Experiences of Selected Credit Programs
Appendix IV 

Major Contributors

General Government

Division

Al Stapleton, Assistant Director, Federal Management and Workforce

 Issues, (202) 512-3418

Victoria Miller O’Dea, Evaluator-in-Charge, (202) 512-6397

Thomas M. Beall, Senior Social Science Analyst

Kiki Theodoropoulos, Senior Evaluator and Communications Analyst

Acknowledgements In addition to those named above, the following individuals made notable

contributions to this report: From the Accounting and Information

Management Division: Jeff Steinhoff, Director of Planning and Reporting;

Dan Blair, Mary Ellen Chervenic, Michael J. Curro, Julie S. Tessauro, and

McCoy Williams, Assistant Directors; and Rita A. Grieco and Carolyn

Litsinger, Senior Evaluators. From the General Government Division:

Joseph S. Wholey, Senior Advisor for Evaluation Methodology; and

Stephanie Shipman, Assistant Director. From the Health, Education and

Human Services Division: Joseph J. Eglin, Jr., Assistant Director; Paula N.

Denman, Senior Evaluator; and Sara E. Edmondson, Senior Social Science

Analyst. From the Office of General Counsel: Alan Belkin, Assistant

General Counsel; and James M. Rebbe, Attorney. From the Resources,

Community, and Economic Development Division: Robert S. Procaccini,

Assistant Director.

Page 52 GAO/GGD-98-41 Experiences of Selected Credit Programs
Page 53 GAO/GGD-98-41 Experiences of Selected Credit Programs
Page 54 GAO/GGD-98-41 Experiences of Selected Credit Programs
Page 55 GAO/GGD-98-41 Experiences of Selected Credit Programs
Related GAO Products

Managing for Results: Agencies’ Annual Performance Plans Can Help

Address Strategic Planning Challenges (GAO/GGD-98-44, Jan. 30, 1998).

Managing For Results: Using the Results Act to Address Mission

Fragmentation and Program Overlap (GAO/AIMD-97-146, Aug. 29, 1997).

The Government Performance and Results Act: 1997 Governmentwide

Implementation Will Be Uneven (GAO/GGD-97-109, June 2, 1997).

Debt Collection: Improved Reporting Needed on Billions of Dollars in

Delinquent Debt and Agency Collection Performance (GAO/AIMD-97-48,

June 2, 1997).

Managing For Results: Analytic Challenges in Measuring Performance

(GAO/HEHS/GGD-97-138, May 30, 1997).

Executive Guide: Effectively Implementing the Government Performance

and Results Act (GAO/GGD-96-118, June 1996).

GPRA Performance Reports (GAO/GGD-96-66R, Feb. 14, 1996).

Financial Management: Continued Momentum Essential to Achieve CFO

Act Goals (GAO/T-AIMD-96-10, Dec. 14, 1995).

Executive Guide: Improving Mission Performance Through Strategic

Information Management and Technology—Learning From Leading

Organizations (GAO/AIMD-94-115, May 1994).

Credit Reform: Case-by-Case Assessment Advisable in Evaluating

Coverage and Compliance (GAO/AIMD-94-57, July 28, 1994).

Federal Credit Programs: Agencies Had Serious Problems Meeting Credit

Reform Accounting Requirements (GAO/AFMD-93-17, Jan. 6, 1993).

(410058) Page 56 GAO/GGD-98-41 Experiences of Selected Credit Programs
Ordering Information

The first copy of each GAO report and testimony is free.

Additional copies are $2 each. Orders should be sent to the

following address, accompanied by a check or money order

made out to the Superintendent of Documents, when

necessary. VISA and MasterCard credit cards are accepted, also.

Orders for 100 or more copies to be mailed to a single address

are discounted 25 percent.

Orders by mail:

U.S. General Accounting Office

P.O. Box 37050

Washington, DC 20013

or visit:

Room 1100

700 4th St. NW (corner of 4th and G Sts. NW)

U.S. General Accounting Office

Washington, DC

Orders may also be placed by calling (202) 512-6000 

or by using fax number (202) 512-6061, or TDD (202) 512-2537.

Each day, GAO issues a list of newly available reports and

testimony. To receive facsimile copies of the daily list or any

list from the past 30 days, please call (202) 512-6000 using a

touchtone phone. A recorded menu will provide information on

how to obtain these lists.

For information on how to access GAO reports on the INTERNET,

send an e-mail message with "info" in the body to:

info@www.gao.gov

or visit GAO’s World Wide Web Home Page at:

http://www.gao.gov

PRINTED ON RECYCLED PAPER
United States

General Accounting Office

Washington, D.C. 20548-0001

Official Business

Penalty for Private Use $300

Address Correction Requested

Bulk Rate

Postage & Fees Paid

GAO

Permit No. G100