Community Development: CDFI Fund Can Improve Its Systems to Measure, Monitor, and Evaluate Awardees' Performance
Published: 1998-07-15T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Pursuant to a legislative requirement, GAO reviewed the performance of the Community Development Financial Institutions (CDFI) Fund, focusing on: (1) the first round of awards in the CDFI and Bank Enterprise Award (BEA) programs and the strategic plan that the Fund developed under the Government Performance and Results Act of 1993; (2) the Fund's development of systems to measure, monitor, and evaluate the awardees' performance; and (3) BEA's impact on banks' lending and investment in CDFIs and distressed communities.
section_title: What GAO Found
paragraphs: GAO noted that: (1) for fiscal year 1996, the Fund complied with the CDFI Act's requirements for negotiating performance goals and measures based on the awardees' business plans; (2) moreover, these goals and measures are consistent with the CDFI program's mission of promoting economic revitalization and community development; (3) however, because the CDFI Act provides no specific guidance for evaluating performance measures, GAO applied the Results Act's standards to the goals and measures that the Fund negotiated in assistance agreements with the 1996 awardees; (4) GAO found that the Fund could improve the nature, completeness, and specificity of these goals and measures; (5) GAO's evaluation of the awardees' assistance agreements revealed an emphasis on measures of activity, rather than on measures of accomplishment; (6) as a result, the assistance agreements focus primarily on what the awardees will do, rather than on how their activities will affect the distressed communities; (7) GAO's evaluation also revealed occasional omissions of measures for key aspects of goals and widespread omissions of baseline data and information on target markets; (8) primarily because of staffing limits, the Fund has just begun to develop mandated monitoring and evaluation systems; (9) given that most awardees have just recently signed their assistance agreements, it is still too early to assess their progress; (10) the impact of the BEA program on banks' lending and investment in CDFIs and distressed communities is difficult to isolate from the impact of regulatory and other economic incentives; (11) moreover, because the Fund did not require banks to report material changes in rewarded investments, it did not have a systematic way of learning about any such changes; (12) the Fund is not authorized to address what banks do with their award funds; however, most awardees have reported that they have reinvested at least a portion of their awards in community development activities; (13) the Fund's current strategic plan contains all of the elements required by the Results Act and suggested by the Office of Management and Budget's implementing guidance; (14) however, these elements generally lack the clarity, specificity, and linkage with one another that the act envisioned; (15) in addition, the plan does not describe the relationship of its activities to similar ones in other government agencies, and it does not indicate whether or how the Fund coordinated with other agencies in developing the plan; and (16) these difficulties are similar to those experienced by other federal agencies in implementing the Results Act's requirements.

section_title: Letter
paragraphs: This report responds to a mandate in the Riegle Community Development and Regulatory Improvement Act of 1994 (P.L. 103-325) that GAO report on the Community Development Financial Institutions (CDFI) Fund, including its structure, governance, and performance. However, as agreed with your offices, the report does not discuss the structure and governance of the Fund because the Department of the Treasury’s Inspector General is conducting an audit to address these issues. The report evaluates how well the Fund is meeting its performance goals and objectives through its two primary programs—the CDFI program and the Bank Enterprise Award program—and considers opportunities for improving their management. In addition, the report provides an assessment of the Fund’s strategic plan. Finally, the report makes recommendations designed to improve the Fund’s performance.
We are sending copies of this report to the appropriate congressional committees and to the Secretary of the Treasury. Copies are available to others upon request.
If you have any questions, please contact me at (202) 512-7631. Major contributors to this report are listed in appendix II.
subsections: section_title: Introduction
paragraphs: Access to credit and investment capital is essential for creating and retaining jobs, developing affordable housing, revitalizing neighborhoods, and promoting the development and growth of small businesses. Over the past three decades, community-based financial institutions have demonstrated that strategic lending and investment activities tailored to the unique characteristics of underserved communities can be effective in improving the economic well-being of these communities and of the people who live in them.
To help create new and expand existing financial institutions that specialize in serving distressed communities, the Congress, in the Community Development Banking and Financial Institutions Act of 1994, established the Community Development Financial Institutions (CDFI) Fund. In creating the Fund, the Congress recognized that the number of CDFIs was limited, most were small, and many were having difficulty raising capital needed to meet the demands for their products and services. The CDFI Fund currently provides needed capital primarily through two programs—the CDFI program, which makes awards directly to qualifying CDFIs, and the Bank Enterprise Award (BEA) program, which provides awards to insured depository institutions for investing in CDFIs and/or distressed communities.
This report responds to a requirement in the 1994 legislation that we report on the Fund’s structure, governance, and performance 30 months after the appointment of an Administrator for the Fund. However, as agreed with your offices, the report does not discuss the structure and governance of the Fund because the Department of the Treasury’s Inspector General is conducting an audit addressing these issues. Our report evaluates how well the Fund is meeting its performance goals and objectives through the CDFI and BEA programs and considers opportunities for improving their implementation. In addition, because this report focuses on the Fund’s performance, it reviews the strategic plan developed under the Government Performance and Results Act of 1993 (Results Act) to guide the Fund’s other activities.
subsections: section_title: The CDFI Fund: Organization and Funding History
paragraphs: The 1994 act established the CDFI Fund as a wholly owned government corporation. Subsequent legislation placed the Fund within the Department of the Treasury and gave the Secretary of the Treasury all of the powers and rights to manage the Fund, as set forth in the authorizing legislation. Figure 1.1 displays the Fund’s organization.
The 1994 legislation also created an Advisory Board to advise the Fund on policy issues. The Fund’s Advisory Board consists of 15 members, including representatives from the departments of Agriculture, Commerce, Housing and Urban Development, the Interior, and the Treasury and from the Small Business Administration, as well as nine private citizens appointed by the President.
From fiscal year 1995 through fiscal year 1998, the Fund received appropriations totaling $225 million. The Fund’s appropriations can be obligated over 2 years; therefore, the total budget authority available to the Fund in any given year is the current year’s appropriation plus any unspent funding from the previous year’s appropriation. Figure 1.2 shows the Fund’s appropriations by fiscal year.
Of the amounts appropriated to the Fund, not more than $5.5 million may be used in a single fiscal year to pay the Fund’s administrative costs and expenses. The remaining appropriations have been committed primarily to the CDFI and BEA programs. Under the act, one-third of the amounts appropriated for programs in any fiscal year must be made available to the BEA program.
The fiscal year 1995 budget proposed an estimated 30 full-time-equivalent staff positions for the Fund. However, the 1995 Rescissions Act limited the Fund to 10 full-time-equivalent positions in fiscal year 1995. During fiscal year 1997, the Fund’s staffing increased to 14 full-time-equivalent positions. For fiscal year 1998, the Fund is authorized 35 full-time-equivalent positions, and the Fund’s management intends to fill all of the vacant positions by the end of the fiscal year. The Fund’s administrative budget remains capped at $5.5 million.
section_title: Community Development Financial Institutions Program
paragraphs: Under the CDFI program, the Fund invests in CDFIs to promote their long-term ability to serve economically distressed communities. Specifically, the Fund provides financial and technical assistance to CDFIs to enhance their ability to make loans and investments and to provide services for economically distressed communities, targeted populations, or both. CDFIs are private profit-making and nonprofit financial institutions that focus on providing financial services to distressed geographic areas and populations that are underserved by conventional lenders and investors. The following are among the types of organizations that qualify for funding under the program:
Community Development Bank: A community development organization centered around a bank or savings and loan that combines the structure and expertise of a profit-making financial institution with a commitment to a distressed place or population.
Community Development Credit Union: A financial cooperative owned and operated by lower-income persons. It provides financial services to its members, including savings and checking accounts and loans for homes, cars, or other personal needs.
Nonprofit Community Development Loan Fund: A financial intermediary that raises capital from individuals and institutional investors, churches, businesses, and foundations, at below-market rates, and relends these funds primarily to community-based organizations and businesses and nonprofit developers in low-income urban or rural communities.
Microenterprise Loan Fund: An entity that receives funding from a private or nonprofit foundation, government agency, or private bank and generally provides technical assistance and loans, ranging from as little as $500 to $10,000, to start up or expand self-help business opportunities for low-income individuals.
Community Development Venture Capital Fund: An entity that provides managerial support, along with equity and debt with equity features, to businesses (typically manufacturing-based) located in low-income communities.
Although two types of CDFIs—community development banks and credit unions—are regulated and insured by the Federal Deposit Insurance Corporation and the National Credit Union Share Insurance Fund, respectively, the remaining types of CDFIs are generally unregulated.
Each year, the Fund solicits applications for awards from CDFIs seeking assistance. An organization’s application must include a comprehensive business plan covering 5 years or more and describing how the organization plans to meet the needs of its investment area, its targeted population, or both. While an applicant is given considerable flexibility in designating an investment area, the area must meet objective criteria for economic distress developed by the Fund. An investment area may include a variety of geographic units reflecting the neighborhoods, areas, or markets that the applicant serves or proposes to serve.
The Fund awards assistance competitively, using selection criteria that include the quality of an applicant’s business plan, the extent and nature of the plan’s impact on community development, and the experience and background of the applicant’s management team. This assistance may be in the form of, for instance, an equity investment, a grant, a loan, and/or technical assistance. In the CDFI program, there were 31 awardees in fiscal year 1996 and 48 awardees in fiscal year 1997.
To receive any financial assistance through the CDFI program, a CDFI must be certified by the Fund, obtain matching funding from nonfederal sources, and enter into an assistance agreement with the Fund. The Fund certifies a CDFI after determining, among other things, that it has a primary mission of promoting community development, its predominant business activity is lending or investing in development, and it serves (an) economically distressed investment area(s) or targeted population(s). The Fund’s certification signifies that a CDFI is eligible to participate in the CDFI program, but it does not constitute an opinion on the CDFI’s financial viability or indicate that the CDFI will receive an award. While the program’s regulations allow an uncertified CDFI to apply and be selected for an award, the CDFI will not receive financial assistance until it has been certified. The Fund requires CDFIs to be recertified every 2 years. As of March 1998, the Fund had certified 205 CDFIs. While the total number of CDFIs nationwide that could be certified by the Fund is unknown, GAO surveyed 925 community development organizations that described their organization as a type of CDFI (see app. I).
To meet the requirement for matching funds, an awardee must obtain assistance from nonfederal sources that is at least equal in form and value to the Fund’s federal assistance. The awardee must provide the Fund with written documentation that it has received either a firm commitment from the provider of the matching funds or the matching funds themselves. This provision is intended to ensure that no federal funds are released until other resources have been leveraged.
After selecting the awardees for a given funding round, the Fund begins negotiating an assistance agreement with each awardee. When both parties have agreed upon all of the elements, the Fund and the awardee enter into an assistance agreement by signing the negotiated document. Each agreement is tailored to the nature of the CDFI—regulated or unregulated—and the type of assistance given to the awardee. The Fund generally disburses funds after an agreement has been signed and the awardee has met the requirements for matching funds and certification.
A key negotiated element of the assistance agreement is a performance schedule for the Fund to use in evaluating the awardee’s performance. This schedule includes performance goals and measures, benchmarks, and an expected evaluation date. Each agreement also requires the CDFI to provide quarterly and annual reports on its financial condition and progress toward meeting its performance goals. In addition, the agreement allows the Fund to apply sanctions, or remedies, that range from changing the goals to requiring the awardee to repay the award if it does not achieve at least a satisfactory level of performance by the evaluation date.
Each year, the Fund must evaluate each awardee’s performance using the performance goals, measures, and benchmarks in the awardee’s assistance agreement. The Fund describes performance goals as qualitative goals and measures as quantitative indicators of the extent to which the awardee has achieved the goals. Benchmarks define levels of performance, ranging from outstanding to unacceptable, which are assessed at specific dates. Awardees often attach “assumptions” to their goals and measures to identify factors beyond their control, such as the continuation of external funding, that may affect the achievement of their goals.
section_title: Bank Enterprise Award Program
paragraphs: The purpose of the BEA program is to encourage insured depository institutions to increase (1) their investments in certified CDFIs and (2) their lending and provision of other financial services in economically distressed communities. According to the Department of the Treasury, there are significant gaps in these communities’ access to capital and the market potential in them is one that banks often do not recognize or understand. By creating an incentive for banks to increase activities in these communities, the BEA program seeks to make it easier for mainstream sources of capital to serve low-income people.
To encourage increased investment, the BEA program rewards banks on the basis of the amount by which they increase their investments and other financial services over a 6-month assessment period (compared with the preceding 6-month, or baseline, period). A bank applying for an award must demonstrate that it plans to invest in or otherwise support a CDFI that is certified by the Fund and/or that the lending and other financial services it plans to undertake are otherwise eligible under the BEA program’s guidelines and regulations. These guidelines and regulations spell out (1) the kinds of services for which the program will grant an award, such as business, consumer, agricultural, or single-family mortgage lending, and (2) the economic and other characteristics that define an area as a “distressed” community, which all of the banks’ rewarded increases in activity must serve. Investments in certified CDFIs are eligible for awards of up to 33 percent of the amount by which banks increase their investments, whereas lending and other services in distressed communities are eligible for awards of 5 percent of the increased investment. In its application for an award, a bank may include not only its plans for specific investments in CDFIs, loans, or services that it knows it will undertake but also good-faith estimates of eligible activities that it expects to undertake.
As long as the Fund (1) has determined that banks have met all of its criteria for awards from the BEA program and (2) has sufficient BEA funds available, it announces that the banks that have applied successfully have won awards. However, the Fund does not disburse a bank’s award immediately. Instead, it disburses the award funds to the extent that the bank provides adequate documentation and other assurances that it has completed the increased investments and/or activities for which it received the award. The Fund disburses award funds in increments when a bank completes its increased activities in increments. For example, if a bank increased its investment in a certified CDFI in two equal installments, it could seek and the Fund would disburse half of its award after the first installment; the Fund would retain the second half until the bank had completed the second installment. For the 1997 awardees, the Fund gives an award on the basis of a bank’s commitment to increase an activity over a period of up to 3 years so long as the bank makes that commitment and begins to fund it during the 6-month assessment period. As a result, the Fund may take up to 3 years after approving an award to disburse all of the award.
Once a bank has received its award funds, it may do anything it wants with them because the legislation creating the program places no restrictions on their use. The Fund has no authority, for example, to require banks to invest their award funds for specific purposes or to report on the uses to which they have put the funds.
section_title: The Results Act
paragraphs: In the 1990s, the Congress established a statutory framework to address long-standing weaknesses in federal operations, improve federal management practices, and provide greater accountability for achieving results. This framework included as its essential elements the Results Act and key financial management and information technology reform legislation: the Chief Financial Officers Act of 1990—as expanded by the Government Management Reform Act of 1994—and the Paperwork Reduction Act of 1995 and the Clinger-Cohen Act of 1996, respectively. Taken together, these legislative initiatives seek to respond to a need for accurate, reliable, and integrated budget, financial, and program information for congressional and executive branch decision-making.
The goal-setting and performance measurement and improvement system envisioned by the Results Act is the centerpiece of this framework and starts with a requirement that each executive agency develop and periodically update a strategic plan covering a period of at least 5 years. This strategic plan is to include elements such as the agency’s mission statement, long-term goals and objectives, and strategies for achieving these goals and objectives. Under the Results Act, the first of these plans was due by September 30, 1997. With the submission of the fiscal year 1999 budget, agencies are also required to prepare annual performance plans that establish connections between the long-term strategic goals outlined in the strategic plans. Finally, the act requires that each agency report annually on the extent to which it is meeting its annual performance goals and the actions needed to achieve or modify any goals that have not been met. The first of these reports, on programs’ performance for fiscal year 1999, is due by March 31, 2000.
The Fund is required by the Department of the Treasury to provide the Secretary with a strategic plan that complies with the provisions of the Results Act. The Fund submitted its plan on September 23, 1997.
section_title: Objectives, Scope, and Methodology
paragraphs: Our objectives for this assignment were to evaluate (1) the progress of the CDFI Fund in developing performance measures for awardees in the CDFI program and systems to monitor and evaluate their progress in meeting their performance goals, as well as the accomplishments they have reported to date; (2) the performance of banks under the BEA program, the impact of the program on banks’ investment activities and on economically distressed communities, and the uses to which banks have put their award funds; and (3) the Fund’s progress in meeting the Results Act’s requirements for strategic planning and the steps the Fund could take to improve its management. Our report focuses on the first round of awards, which the Fund made in 1996.
To accomplish our first objective, we reviewed the Fund’s process for setting goals and developing performance measures with the 1996 awardees and discussed the Fund’s performance measurement system with responsible officials at the Fund’s headquarters in Washington, D.C. We also randomly selected six awardees as case studies to gain these awardees’ perspectives on the process of developing performance measures. In addition, we conducted a national survey of 925 CDFI organizations to obtain information on the performance measurement and monitoring systems used in the CDFI field. We reviewed the Fund’s statutory, regulatory, and other reporting and monitoring requirements and spoke with officials at the Fund and at our case studies to assess the Fund’s progress in developing systems for monitoring and evaluating awardees’ progress. Finally, we reviewed quarterly progress reports submitted by 19 of the 1996 awardees and held discussions with case study officials to assess the awardees’ progress.
To accomplish our second objective, we reviewed the Fund’s guidance, policies, procedures, and other materials on the BEA awards process and discussed these and other issues related to the program with responsible Fund officials. We obtained data on the banks’ performance from the Fund’s status report on activities completed as of January 1998. In addition, we conducted case studies of five awardees that, collectively, provided the full range of activities for which banks can receive awards.
To accomplish our third objective, we reviewed the Fund’s most recent strategic plan, developed in September 1997 in accordance with the Results Act’s requirements and OMB’s guidance. We judged the quality of the plan as a whole and of its primary components, using our knowledge of the program and using guidance developed by GAO for evaluating agencies’ strategic plans.
We provided a draft of this report to the CDFI Fund for its review and comment. The Fund’s comments are addressed at the end of each applicable chapter.
We performed our review from July 1997 through June 1998 in accordance with generally accepted government auditing standards. We relied on data provided to us by the Fund, by awardees in both the CDFI and the BEA programs, and by CDFIs responding to our survey. A more detailed discussion of our methodology appears in appendix I.
section_title: Better Performance Measures Are Needed to Monitor and Evaluate the CDFI Program’s Accomplishments
paragraphs: In entering into assistance agreements with the 1996 awardees in the CDFI program, the Fund complied with the CDFI Act’s requirements for negotiating performance measures based on the awardees’ business plans. However, opportunities exist for the Fund to improve the nature, completeness, and specificity of the performance measures that it negotiates with future awardees. Our evaluation of the 1996 assistance agreements revealed (1) an emphasis on measures of activity, such as the number of loans made, rather than on measures of accomplishment, such as the net number of jobs created or retained; (2) the occasional omission of measures for key aspects of goals; and (3) the widespread omission of baseline data and information on target markets, needed to track progress over time. Including more requirements in the assistance agreements for reporting accomplishments could help to keep both the awardees and the Fund focused on the primary purposes of the CDFI program, and using more complete and specific goals and measures could facilitate the Fund’s monitoring and evaluation of awardees’ performance.
The Fund is just beginning to develop mandated monitoring and evaluation systems. It has established reporting requirements for awardees to collect information for monitoring their performance, and it is developing postaward monitoring procedures for using this information to assess their compliance with their assistance agreements. Although the Fund has published some “success stories,” it has not established a system for evaluating the impact of awardees’ activities.
Most CDFIs only recently signed their assistance agreements. Therefore, any reports of progress are limited and preliminary. Furthermore, the many different types of CDFIs support different types of activities and use different performance measures, making any general assessment of their progress difficult.
subsections: section_title: Progress in Developing Performance Measures Is Mixed
paragraphs: The CDFI Fund’s progress in developing performance goals and measures for awardees in the CDFI program is mixed. On one hand, as of January 1998, the Fund had entered into assistance agreements with 26 of the 31 CDFIs that received awards in 1996 and had disbursed approximately $31 million of the $37 million set aside for this first round of awards under the program. As the CDFI Act requires, these agreements include performance measures that (1) were negotiated with the awardees and (2) are generally based on the awardees’ business plans. On the other hand, the performance goals and measures that the Fund negotiated with the 1996 awardees fall somewhat short of the standards for performance measures established in the Results Act. Because the CDFI Act provides no further guidance on developing performance measures, we drew on the Results Act’s standards for our evaluation, even though the performance measures in the assistance agreements are not subject to the Results Act.
In part, the CDFIs’ widespread use of activity measures, rather than accomplishment measures, is attributable to concerns about isolating the results of community development initiatives from the influences of other factors, which may be beyond the awardees’ control. In addition, related concerns about the Fund’s possible imposition of sanctions appear to further deter the use of accomplishment measures. However, accomplishment measures are important to focus attention on the desired results of the Fund’s investments and can, in our view, be negotiated so as to address these concerns. Lack of written guidance on developing performance measures for the 1996 awardees may be partly responsible for the occasional omission of measures for key aspects of some goals. Finally, baseline data and data on target markets generally do not appear in the assistance agreements. Although we found that these data were generally available in other documents—and, according to Fund officials, were used in setting performance levels—their absence from the agreements could hinder the efficient evaluation of awardees’ performance.
subsections: section_title: The Fund Met the CDFI Act’s Requirements
paragraphs: In total, the Fund negotiated 87 performance goals and 165 performance measures in the 26 assistance agreements that it signed with awardees through January 1998. These goals and measures were consistent with the CDFI program’s mission of promoting economic revitalization and community development and were generally based on the awardees’ business plans.
subsections: section_title: The Fund Devised a Flexible Schedule for Negotiating Performance Measures
paragraphs: To provide a framework for negotiation, the Fund developed a performance schedule that proved flexible enough for the many different types of CDFIs to tailor their performance measures to their particular activities. This schedule appears in each of the 1996 assistance agreements and is intended to be a complete description of each awardee’s planned performance for the period of the award (generally 5 years or more). Besides performance goals and measures, the schedule includes benchmarks (expressed in ranges) for evaluating the awardee’s level of performance as of the evaluation date and, optionally, at one or more interim dates. Finally, the schedule includes assumptions about external factors (i.e., factors outside the awardee’s control) that could affect the awardee’s performance. Table 2.1 illustrates the schedule, using one of the more comprehensive schedules prepared by a 1996 awardee.
The performance schedule documents the results of the negotiations between the Fund and the awardee and sets forth the standards that the Fund will use to hold the awardee accountable for its use of the Fund’s money. If the awardee does not achieve at least a satisfactory level of performance for each measure, the Fund has the authority to impose sanctions (referred to as remedies), set forth elsewhere in the assistance agreement. Sanctions vary in severity. While some, such as requiring a change in the awardee’s performance goals, are relatively minor, others, such as requiring the repayment of any assistance that has been distributed to the awardee and/or barring the awardee from applying for any future assistance from the Fund, are much more severe.
section_title: Performance Goals Were Generally Based on Awardees’ Business Plans
paragraphs: Our analysis of six 1996 awardees’ assistance agreements shows that their performance goals were generally based on the awardees’ business plans, as the CDFI Act requires. In some instances, the performance goals in the agreements also incorporated updates or adjustments reflecting changes that occurred after the business plans were submitted with the awardees’ application packages. For example, the Fund instructed the awardees to incorporate in their performance goals any changes that had taken place in their funding, operations, and/or target markets. In addition, Fund staff told us that they considered the performance projected in some of the business plans to be overly optimistic, and they said they used their expertise in CDFI subfields (e.g., loan funds, credit unions, venture capital funds) to help the awardees set more realistic, reachable benchmarks.
Both the business plans and the performance goals in the assistance agreements supported the CDFI program’s mission of promoting economic revitalization and community development. The business plans, which formed the most significant component of each awardee’s application package, were required to include information about the anticipated impact of the awardee’s planned activities on the target community. These plans served, in large measure, as the basis for selecting awardees to participate in the program. The plans also provided the basis for negotiating the performance goals and measures in the assistance agreements. According to our analysis, 98 percent of the performance goals in the assistance agreements were consistent with the CDFI program’s mission of promoting economic revitalization and community development. However, as discussed below, the business plans we reviewed incorporated more accomplishment measures than the awardees’ assistance agreements.
section_title: Results Act Sets Standards for Improving Future Performance Measurement
paragraphs: The Results Act, supplemented by Circular A-11, the Office of Management and Budget’s (OMB) implementing guidance, and a GAO document entitled The Results Act: An Evaluator’s Guide to Assessing Agency Annual Performance Plans (GAO/GGD-10.1.20, Apr. 1998) provides more explicit guidance for developing performance goals and measures than the CDFI legislation. Although the Results Act does not apply to awardees, it establishes the federal government’s standards for performance measurement, and its provisions apply to all federal agencies, including the Department of the Treasury, of which the CDFI Fund is a part. Thus, consistency between the awardees’ performance goals and measures and the provisions of the Results Act will facilitate the Fund’s compliance with the Results Act.
In broad terms, the Results Act and its guidance (1) consider measures of accomplishments (outcomes) preferable to measures of activities (outputs); (2) recommend the use of objective measures that adequately indicate progress towards the performance goals; and (3) say that the goals and measures should be measurable and quantifiable.
subsections: section_title: Measures in Agreements Focus More on Activities Than on Accomplishments
paragraphs: According to the Results Act and its guidance, both activity and accomplishment measures can be useful. However, the act and guidance regard accomplishment measures as better indicators of a program’s results because they describe the effects of an organization’s activities on the populations that are expected to be served. The 26 assistance agreements that we reviewed contain relatively few accomplishment measures. Of the 165 measures, we identified 25, or about 15 percent, that were accomplishment measures. As a result, the assistance agreements focus primarily on what the awardees will do, rather than on how their activities will affect the distressed communities. This limited use of accomplishment measures contrasts with the widespread use of such measures that the CDFI field reported to us and that we observed in the business plans of our six case study awardees.
To determine how the CDFI field assesses progress toward meeting its goals, we surveyed 925 CDFIs nationwide and received responses from 623 of them. Among the respondents were 24 of the 26 CDFIs that had signed (closed) assistance agreements as of January 1998. According to the responses we received, the CDFIs commonly use both accomplishment and activity measures to assess their progress. For example, 91 percent of the CDFIs providing lending services identified at least one quantifiable lending accomplishment measure, and nearly 60 percent identified three or more such measures. The responses of the 24 CDFIs with closed assistance agreements were consistent with the responses of the CDFIs who responded to our survey and indicated a wider use of accomplishment measures than we observed in the assistance agreements of these 24 CDFIs. Table 2.2 compares the use of accomplishment measures as reported to us by the 24 awardees and as shown in their assistance agreements.
When we compared the types of performance measures that our six case study awardees used in their business plans and in their assistance agreements, we found that the awardees generally included both activity and accomplishment measures in their business plans but were less likely to include accomplishment measures in their assistance agreements. For example, one awardee, a community development loan fund, included several activity measures (e.g., the number of loans made and the dollar volume of the loans) and accomplishment measures (e.g., the number of jobs created, stabilized, and upgraded) in its business plan. By contrast, the awardee included only activity measures (e.g., the number and dollar volume of loans made) in its assistance agreement. Another awardee, a microenterprise loan fund, included both activity measures (e.g., the number and dollar volume of loans disbursed) and accomplishment measures reflecting the potential effect of these loans (e.g., increases in business assets and in the number of new jobs) in its business plan. In its assistance agreement, however, the awardee included only activity measures (e.g., the number and dollar volume of loans disbursed). Only one of the six case study awardees included an accomplishment measure in its assistance agreement that did not appear in its business plan.
According to most of the case study awardees and Fund officials who negotiated with the awardees, the 1996 awardees were reluctant to incorporate accomplishment measures into their assistance agreements for two major reasons. First, they were concerned about being able to demonstrate that their activities had produced specific effects in distressed communities. Second, they were concerned that the Fund might impose sanctions if they could not achieve results that were, to some extent, beyond their control.
Both the case study awardees and Fund officials recognized that it is difficult to isolate the effects of a CDFI’s community development activities from the effects of external factors, such as economic, social, and political conditions. Because they could not demonstrate a direct causal link between these activities and results in the community, they tended to negotiate measures of activities—what the awardees were doing—rather than measures of accomplishments—how their activities were improving economic conditions in distressed communities. While this approach is understandable, it goes beyond the requirements of performance measurement. Specifically, performance measurement is intended to identify and track the changes associated with a program but does not require the establishment of an exclusive causal link between a program and an observed change. Establishing a direct causal link is a more complex task, often requiring controlled studies or statistical analyses, and is the purpose of impact analysis, discussed later in this chapter. Nevertheless, tracking changes associated with a program is important because it focuses attention on, and provides an indication of, the program’s results.
The 1996 awardees were also reluctant to incorporate accomplishment measures into their assistance agreements because they were concerned about the Fund’s possible imposition of sanctions. As several awardees pointed out, accomplishments may be subject to outside factors and can, therefore, be harder to control than activities. Consequently, most of these awardees did not want to be held contractually accountable for meeting benchmarks for accomplishment measures over which they had limited control. As is clear from the array of assumptions set forth in the performance schedule illustrated in figure 2.1, an awardee’s performance may be subject to many factors beyond the awardee’s control—including the continuation of funding from other sources, interest rates, economic conditions, property taxes, the reputation of the local public schools, and the rate of inflation.
The application of sanctions is at the Fund’s discretion, and the Fund has not yet demonstrated how it will apply sanctions for poor performance. The 1996 awardees were, therefore, uncertain about how the Fund would use sanctions. Even though the Fund stipulated in the 1996 assistance agreements that it would generally not impose the more severe sanctions—that is, those requiring the repayment of assistance or reducing or terminating assistance—for failure to meet the benchmarks, the perceived threat of sanctions limited the types of measures considered during the negotiation of performance measures and benchmarks, according to three of the six case study awardees.
For the 1997 awardees, the Fund has revised both the performance schedule and the assistance agreement. As table 2.3 illustrates, the revised schedule reduces the duration of the benchmarks from 5 years to 1 year. The revised agreement reintroduces the potential use of all the sanctions for unacceptable performance. Thus, the Fund could require the 1997 awardees to repay the assistance they have received if they do not meet their benchmarks. According to a senior Fund official, the Fund changed the duration of the benchmarks to facilitate compliance with the statutory requirement that it submit an annual report on awardees’ performance and to more effectively promote awardees’ accomplishment of their business plans. He also indicated that the Fund reintroduced the potential use of all sanctions so that it would have a full range of options available to address instances of noncompliance. However, with less time to reach the benchmarks and potentially stiffer sanctions for not reaching them, the awardees may be further deterred from using accomplishment measures. Alternatively, limiting the application of sanctions to measures over which awardees have control could allow the Fund to address instances of noncompliance without discouraging the use of accomplishment measures.
According to the Results Act and its guidance, performance measures should adequately indicate progress toward the performance goal. To determine, for the 1996 agreements, whether the 165 performance measures indicated progress towards meeting the 87 goals, we systematically assessed whether the measures had a clear, apparent, or commonly accepted relationship to the intended goals and whether the measures appropriately represented the goals without showing any biasin measuring progress toward them. We found that about 96 percent of the measures were related to the goals and about 84 percent were unbiased. For example, one community development credit union’s goal was to significantly expand the availability of financial services to its members. The measures that accompanied this goal were the number of new checking accounts opened, the number of ATM cards issued, and the number of credit cards issued. These measures were related to and appropriately represented the goal because they were limited to financial services that the credit union currently offered to its members.
The Results Act and its guidance also indicate that performance measures should address all key aspects of goals. Although we did not attempt to quantify the degree to which groups of measures addressed all key aspects of goals, we observed that there were differences in the extent to which goals were adequately addressed. In some instances, a group of two or three measures did not address certain key aspects of a stated goal:
One awardee’s goal was to “alleviate poverty by providing substantial loans and technical assistance to promote entrepreneurship and sustainable businesses.” The awardee’s assistance agreement included three measures for providing loans—the dollar amount of loans disbursed to nonprofit organizations, the number of borrowers receiving loans from the awardee, and the number of borrowers remaining in operation since the loans were made to them. However, the assistance agreement included no measures pertaining to entrepreneurship or technical assistance.
In other instances, the measures as a group seemed to address all key aspects of the goal:
One awardee’s goal was to “increase job opportunities by encouraging the start-up of new businesses and the expansion of existing businesses.” The associated measures were “the net increase in the number of full-and part-time jobs in businesses financed by the awardee as of April 1997,” and “the net increase in salaries and wage expenses of businesses financed by the awardee as of April 1997.” If we assume that the awardee will be funding only business start-ups or expansions, then the measures address the key aspects of increasing job opportunities—increasing the number of full- and part-time jobs and increasing wage expenses.
section_title: Units and Terms Were Specific but Lacked Baseline Data and Did Not Refer to the Target Market
paragraphs: Under the Results Act and its guidance, performance goals should be measurable and quantifiable. These characteristics, which are important to facilitate monitoring and evaluation, are best achieved through the use of specific units, well-defined terms, and baseline and target (end) values and dates. Identifying the geographic areas or populations to be served in the goal statements also makes monitoring and evaluation easier. Our analysis showed that the performance goals and measures in the 1996 assistance agreements had most of these features:
All of the goals had at least one measure with a defined unit, such as a “loan,” and most (95 percent) of the units were at least minimally specific (e.g., “housing loans”). In nearly all (95 percent) of the goals and measures, the terms were generally known or, if not known, defined.
Essentially all (99 percent) of the goals and measures had target values and dates that, together, comprise the benchmark (see fig. 2.1).
While the performance measures in the 1996 performance schedules had several measurable or quantifiable features, they generally lacked baseline information. Specifically, 95 percent of the goals and measures lacked baseline values and 93 percent lacked baseline dates. Including such values and dates in the performance schedule is important because it establishes a context for understanding the significance of the benchmark ranges and ensures that the Fund and the awardee have an agreed-upon basis for measuring performance.
Fund officials told us that they used specific baseline dates and values in negotiating the benchmarks for awardees’ performance measures. For some case study awardees, we were able to identify this information in other documents. However, having to refer to other documents to obtain this information undermines the assistance agreement as a ready reference for understanding and assessing progress toward an awardee’s stated goals.
We also analyzed whether the goal statements identified the populations served by the awardees. The majority (83 percent) of the goals and measures identified the geographic areas or populations to be served; however, the level of detail varied widely. For instance, some specified “the investment area” (e.g., a particular geographic area that can help define an awardee’s target market), while others simply identified a particular city or portion of a state (e.g., southwest Pennsylvania). Still others provided no information about the geographic areas or populations to be served. Moreover, it was often unclear how the area or group identified in a measure was related to the target market defined in the awardee’s certification documentation. If information about the target market does not appear in the performance schedule, the schedule creates no context for understanding why specific measures are targeted to particular groups or areas. Like the omission of baseline information, the omission of information about the target market is inconsistent with the usefulness of the performance schedule as a ready reference for understanding and assessing progress toward an awardee’s goal.
section_title: The Fund Is Taking Steps to Improve Performance Measures
paragraphs: Limitations in the 1996 awardees’ goals and measures can be attributed to several factors, including the recent implementation of the Results Act at all federal agencies, understaffing at the Fund and lack of training in the Results Act’s provisions for its staff, and lack of written guidance and training for the 1996 awardees on developing goals and measures. Over time, the Fund, like other federal agencies, has become more familiar with the Results Act, and it conducted training in April 1998 for its staff on the act’s provisions. The Fund is also hiring additional management and program staff to help implement the CDFI program. Finally, the Fund conducted a workshop for the 1997 awardees on the closing process for the assistance agreements. This workshop included more substantial guidance on drafting performance goals and measures.
section_title: Mandatory Monitoring and Evaluation Systems Are Not Yet in Place
paragraphs: Although the Fund has developed reporting requirements for awardees to collect information for monitoring their performance, it lacks documented postaward monitoring procedures for assessing their compliance with their assistance agreements, determining the need for corrective actions, and verifying the accuracy of the information collected. In addition, the Fund has not yet established procedures for reporting and evaluating the impact of awardees’ activities. The effectiveness of the Fund’s monitoring and evaluation systems will depend, in large part, on the accuracy of the information being collected. Primarily because of statutorily imposed staffing restrictions in fiscal year 1995 and subsequent departmental hiring restrictions, the Fund has had a limited number of staff to develop and implement its monitoring and evaluation systems. In fiscal year 1998, it began to hire management and professional staff to develop monitoring and evaluation policies and procedures.
subsections: section_title: The Fund Has Developed Reporting Requirements but Lacks Postaward Monitoring Procedures
paragraphs: The CDFI Act stipulates that the Fund shall require each CDFI receiving assistance to submit an annual report on its activities, financial condition, success in meeting its performance goals, and success in satisfying the terms and conditions of its assistance agreement. Before the Fund could review an awardee’s progress, it had to establish financial and performance reporting requirements based on the awardee’s assistance agreement. It then had to develop monitoring procedures to ensure that the required reports were received, reviewed, and acted upon as necessary.
To develop reporting requirements, the Treasury’s Office of Inspector General identified for the Fund the types of information that it should collect. The May 1996 Inspector General’s report identified the information that awardees needed to report quarterly and annually so that the Fund could assess their financial condition, performance, and compliance with the terms and conditions of their assistance agreements. The report also suggested that the information furnished by the awardees be supplemented with audited financial statements and reviews of the awardees’ organizations. Such reviews typically include desk reviews and site visits.
The Fund’s reporting requirements meet the CDFI program’s statutory review requirements, and the 1996 assistance agreements incorporate the Inspector General’s suggested reporting requirements. Awardees are therefore required to provide both quarterly and annual reports on their progress in achieving their performance goals, submit financial statements certifying their financial viability, and report any changes in their operations that could materially affect their ability to meet the terms and conditions of their assistance agreements.
Although the Fund has established reporting requirements for awardees, it lacks documented monitoring procedures for tracking and reviewing the data submitted in the quarterly and annual reports, according to an independent audit recently completed by KPMG Peat Marwick. To develop these procedures, the audit recommended, among other things, that the Fund develop written monitoring procedures, continue to design and develop a portfolio-monitoring database system, and track the receipt of all required reports for all awardees.
Fund staff told us that they had not previously developed postaward monitoring procedures because of the CDFI Fund’s initial staffing restrictions and shortages. Now that the Fund has hired additional staff, it has turned its attention to correcting the weaknesses identified by the auditors. The Fund has established a system for tracking the receipt of quarterly and annual reports and the financial statements submitted by awardees. In addition, the Fund has hired two key staff—(1) a chief financial officer for management to oversee all aspects of the Fund’s administrative operations, accounting, reporting, management controls, budget, portfolio monitoring, and compliance with laws and regulations and (2) an awards manager to conduct an initial assessment of the Fund’s monitoring requirements, recommend a monitoring and assessment process for the Fund’s awardees, and integrate these monitoring recommendations into the design of the Fund’s awards database.
In addition to the weaknesses identified by KPMG Peat Marwick, we found that the Fund has not yet developed written guidance for verifying the information provided by awardees when they are reporting quarterly or annually on their performance. Furthermore, the Fund has not established plans or procedures for conducting periodic desk audits or site visits or for otherwise verifying the accuracy of the reported performance information. Staffing restrictions go a long way toward explaining why the Fund has not yet developed such written guidance or procedures.
The Fund may be able to take advantage of the expertise of the independent auditors that conduct annual audits of the awardees to verify some of the information on performance that the awardees report to the Fund. Since the Fund is required in the assistance agreements to provide guidance to the independent auditors, this guidance could require that the audits, at a minimum, verify the results of activity-based performance measures and financial data. Providing this guidance would enable the Fund to leverage the expertise of the independent auditors in verifying both financial and performance information. To date, the Fund has provided the auditors only with verbal guidance, at their request. This guidance focused on the financial audit and did not address the verification of reported performance data. While providing verbal guidance may meet the provision of the assistance agreement, it does not ensure the same degree of consistency that would be derived from written guidelines. Fund staff have indicated that they intend to develop written guidance for the Fund’s own auditors and independent auditors during fiscal year 1998.
Monitoring the compliance of awardees will require reviews by staff who understand CDFIs and can systematically assess their progress and evaluate their compliance with their assistance agreements. For example, each year, for the terms of their agreements, the 26 1996 awardees whose agreements we reviewed will be required to submit a total of 104 quarterly reports and 26 annual reports. For each report, the Fund will need to assess the awardee’s compliance with (1) agreed-upon performance benchmarks, (2) financial soundness covenants, and (3) requirements for serving target markets. The Fund will also need to decide how to respond when an awardee’s compliance is less than satisfactory. Such a decision will likely involve analyzing the awardee’s operations, the target market, and the influence of the external factors identified in the performance schedule.
The Fund is developing procedures for reviewing awardees’ quarterly and annual reports and following up, as necessary, on any preliminary indications of noncompliance with provisions of the assistance agreements. When all 31 of the 1996 agreements and all 48 of the 1997 agreements are executed, the Fund will be receiving 316 quarterly reports and 79 annual reports each year. This volume will grow as the Fund makes future awards under the CDFI program. Currently, the program staff who will be reviewing these reports will also be responsible for certifying CDFIs and closing agreements with later awardees. As the volume and complexity of the Fund’s monitoring responsibilities grow, the Fund is likely to need additional monitoring capacity.
section_title: Systematic Evaluation Has Not Yet Begun
paragraphs: The CDFI Act and the associated Conference Report establish requirements for evaluating and reporting on the CDFI program. The act specifies that the Fund is to annually evaluate and report on the activities carried out by the Fund and the awardees. The Conference Report elaborates, stating that the Fund’s annual “performance report” will (1) analyze and compare the overall leverage of federal assistance with private resources and (2) determine the impact of spending these resources on the program’s investment areas, targeted populations, and qualified distressed communities. To generate data for these analyses, the statute requires the awardees to report annually on their progress in carrying out their assistance agreements and to compile such data as the Fund considers pertinent on the gender, race, ethnicity, national origin, or other characteristics of individuals served by the awardees to ensure that the targeted populations and low-income residents of investment areas are adequately served.
As directed by the Conference Report, the Fund has published an estimate of the private funding leveraged by the CDFI program. According to the Fund’s second annual report, the $37 million in assistance awarded to CDFIs in 1996 will leverage approximately three to four times that amount in new capital over the next several years. This estimate is based on discussions with CDFIs and CDFI trade association representatives, not on financial data collected from the awardees. Eventually, to better comply with the leveraging requirement, the Fund will need to develop and disseminate to awardees a method for calculating how much nonfederal assistance they are able to leverage with their awards. In addition, the Fund will need to develop a method for aggregating the leveraging ratios of the individual awardees into a leveraging ratio for the entire CDFI program.
To determine the effect of awardees’ expenditures on the CDFI program’s investment areas, targeted populations, and qualified distressed communities, the Fund has thus far compiled and published, in its 1996 and 1997 annual reports, general descriptions of a few CDFIs’ activities and anecdotes about individuals served by selected CDFIs. These reports, however, do not include analyses of the gender, race, ethnicity, national origin, or other characteristics of individuals served by the awardees. This demographic information has just started to become available to the Fund through the 1996 awardees’ annual reports.
It is still too early for the Fund to conduct a comprehensive evaluation of the CDFI program’s impact. The Fund made its first investment in a CDFI just 17 months ago, and data for a comprehensive evaluation are not yet available. Eventually, however, the Fund will need to conduct systematic research and analysis to determine the impact of awardees’ expenditures.
Although the CDFI Act does not indicate how the Fund is to perform impact analyses, in prior work on economic development programs, we have reported that they must be based on research that associates economic improvements in investment areas with a program’s expenditures and accounts for the influence of other factors. As noted, isolating the impact of community development initiatives from other influences is a particularly challenging task. Satisfying the Conference Report’s requirement for impact analyses, in our view, will be a long-term, ongoing commitment for the Fund that will require expertise in evaluation and procedures for systematically gathering and analyzing information as it becomes available.
Fund officials have acknowledged that their evaluation efforts must be enhanced, and they have planned or taken actions toward improvement. For instance, the Fund has begun hiring staff to conduct and/or supervise research and evaluations and has revised the assistance agreements for the 1997 awardees to require that they submit annual impact reports. However, although the Fund has recently hired a program manager for policy and research, it has not yet reached a final decision on what information it will require from the awardees for the Fund to use in evaluating the program’s impact. Two key sources of information about the program’s impact are available for each awardee—the business plan, which includes a community impact section describing the awardee’s projected accomplishments over a 5-year period, and the performance schedule, to the extent that it includes performance information for accomplishment measures. The Fund also has to determine how it will integrate awardees’ reported performance and the lessons learned from related research in the CDFI field into its evaluation plans.
section_title: Reports of 1996 Awardees’ Progress Are Preliminary and Difficult to Summarize
paragraphs: Given that most CDFIs only recently signed their assistance agreements, reports of accomplishments in the CDFI program are limited and preliminary. Furthermore, given that the different types of CDFIs support different types of activities and use different performance measures, any general summary of their performance will be difficult.
subsections: section_title: Early Quarterly Reports Describe Awardees’ Activities to Date
paragraphs: The vast majority of the 1996 awardees signed their assistance agreements between March 1997 and October 1997. Therefore, the Fund has only begun to receive the quarterly reports it requires on their performance.
Through February 1998, the Fund had received 41 quarterly reports from 19 CDFIs. Sixteen of these CDFIs submitted either two or three quarterly reports while the other three CDFIs, which had closed their agreements with the Fund later in 1997, submitted one report. As of February 1998, the Fund had received three annual reports. Neither the Fund nor an independent auditor had verified the accuracy of any of the data submitted in either the quarterly or the annual reports received by the Fund.
The 19 CDFIs that submitted quarterly reports through December 1997 include at least one of each of the five principal types of CDFIs introduced in chapter 1, as well as two other types of CDFIs—an intermediary for community development corporations that provides CDFIs with financial and technical assistance and a multifaceted community development corporation that manages more than one type of activity—a loan fund and a credit union. Figure 2.1 breaks down the 19 CDFIs by type.
To analyze the performance reported by the 1996 awardees through February 1998, we categorized the 165 performance measures included in the 26 assistance agreements that we reviewed. These measures fell into over 40 different categories, from lending to investment to training. The different types of CDFIs both resemble and differ from one another in the types of activities they support and in the performance measures they use. For example, all 10 of the nonprofit loan funds make loans, and many use the same performance measures—increases in the total numbers and dollar amounts of loans made and increases in net assets—but the different funds make different kinds of loans, from loans to small businesses; to loans for major home improvements; to loans to borrowers who are elderly, disabled, or have special needs. One of the loan funds also uses a unique performance measure—increases in the number of new loan products—to track its progress in supporting innovative activities. Examples of new loan products include loans for child care facilities and loans for community water and wastewater systems.
Given the variety of measures used, it is difficult to summarize all of the activity reported by the 19 awardees to date. To illustrate the cumulative reported activity, we totaled the data for the two most common measures—the total number of loans for both general and specific purposes and the total dollar value of these loans. According to our analysis, since the quarters in which the agreements were closed, the CDFIs cumulatively reported making over 1,300 loans totaling about $52 million. Of these, we identified 112 as business or commercial loans totaling about $7.4 million. Another 264 loans, totaling more than $15 million, were made either to individuals or to communities and included mortgage loans for purchasing or rehabilitating homes, personal loans, or loans for financing facilities. In addition, the CDFIs reported providing consumer counseling and technical training to 480 individuals or businesses.
We derived another summary observation of the effect of the CDFI program’s funding to date from our survey of the 24 1996 awardees with signed assistance agreements. When we asked them to identify the extent to which the funds they had received from the program had enabled them to increase their services or their client/customer base, 12 indicated either a very great or a great increase, 7 indicated either a moderate or some increase, 4 reported that it was too early to tell, and 1 did not respond to the question.
section_title: Conclusions
paragraphs: Revisions to the Fund’s assistance agreement should be easy to implement. For example, ensuring that measures address all key aspects of goals and adding baseline dates and values and information on target markets to facilitate program evaluation should not be difficult. However, encouraging the greater reporting of accomplishments will require overcoming awardees’ concerns about the possible imposition of sanctions for not meeting benchmarks for measures that are, to some extent, beyond the awardees’ control. These concerns may be alleviated by requiring the awardees to report accomplishments only in their annual impact reports. Awardees would be sanctioned only if they failed to submit the required reports.
In large part because of staffing limitations, the Fund has not yet completed postaward monitoring and evaluation systems. It recognizes the importance of such systems and is in the process of hiring staff and determining the systems’ requirements. The sooner the Fund completes this important task, the better it will be able either to identify and correct any instances of noncompliance or to identify and implement any opportunities for improvement.
section_title: Recommendations
paragraphs: To strengthen performance measurement in the CDFI program, GAO recommends that the Secretary of the Treasury instruct the Director of the Fund to take the following steps:
Encourage the greater reporting of accomplishments by awardees. To allay the concerns described by the 1996 awardees, the Director could require that accomplishments be reported in each awardee’s annual impact report. Accomplishment measures should include (1) those that are, without limitation, negotiated with each awardee on the basis of the awardee’s business plan and (2) those that are, to the maximum extent practicable, related to the awardee’s performance goals and measures.
Establish procedures for systematically reviewing each awardee’s goals to ensure that the measures address all key aspects of the related goals.
Reformat the assistance agreement to include baseline dates and values and information on target markets.
section_title: Agency Comments
paragraphs: The Fund’s Director and Deputy Director for Policy and Programs generally agreed with the information presented in this chapter. In addition, they suggested several revisions to improve the accuracy of the information. We have incorporated these revisions.
While these officials agreed with our conclusion that the Fund needs to encourage awardees to increase their reporting of accomplishments, they disagreed with our draft report’s proposed recommendation to (1) include accomplishment measures in the performance schedule of the assistance agreement and (2) waive the use of sanctions for measures outside the awardee’s control. As an alternative, the officials suggested that accomplishment measures be included in the annual impact report that the Fund will be requiring from each awardee. Such measures would not be subject to sanctions, and sanctions could be applied only when an awardee failed to submit the required report. Consequently, this approach would not require the Fund to apply sanctions for some measures and waive them for others. Because the Fund’s proposed alternative should achieve GAO’s and the Fund’s mutual objective of increasing awardees’ reporting of accomplishments, we revised our conclusions and recommendation accordingly.
Finally, the Fund concurred with our two recommendations for revising the assistance agreement to ensure that awardees’ performance measures address all key aspects of their related goals and include baseline dates and values and information on target markets.
section_title: Impact of the BEA Program Is Difficult to Assess
paragraphs: Banks have completed most of the activities for which they received 1996 bank enterprise awards. However, isolating the impact of the BEA program is difficult because other regulatory or economic incentives can encourage the same types of investment as the prospect of an award, and limitations in some banks’ data preclude the Fund, in a few cases, from knowing for certain that awardees have increased their net investments in targeted distressed communities as intended. Because the Fund did not require the banks that received 1996 awards to report material changes in the status of rewarded activities, it did not know whether rewarded investments remained long-term increases in the capital available to CDFIs and distressed communities. In addition, because the BEA program rewards banks for increasing activities that they undertake with their own funds, it does not (by statute) address what banks do with their awards. Nevertheless, most banks have reported that they have voluntarily reinvested their awards in lending and investments related to community development.
subsections: section_title: Banks Completed Most Activities for Which They Received 1996 Awards
paragraphs: In 1996, the Fund announced a total of $13.1 million in awards to 38 banks. These awards were based on reported increases of $65.8 million in investments in CDFIs and $60 million in loans and financial services in distressed communities. Located in 18 states and the District of Columbia, the 38 banks held assets ranging from $21 million to $320 billion. As figure 3.1 shows, about half of the increased activities that led to the awards supported CDFIs and about half promoted development in distressed communities.
Increase in CDFI-related suppport activities (in dollars)
As figure 3.2 shows, nearly three-quarters of the banks that supported CDFIs did so through equity investments while the remaining quarter did so through loans.
The Fund has attempted to measure the impact of the BEA program by estimating its leverage, that is, the amount of the private investments associated with the awards. The Fund estimates that the $13.1 million in 1996 awards leveraged over $125 million in private investments, a leveraging ratio of almost 10 to 1. This estimate includes amounts in the rewarded banks’ baselines for similar loans and investments made before the 6-month assessment period and included in the base amounts from which the Fund measured the increases in investment. If the leveraging effect of the BEA program were calculated only on the basis of the increases in investment that led to the awards, we estimate that the leveraging ratio for the 1996 awards would be about 7 to 1—a ratio generally consistent with the 15-percent award for increases in equity investments, which represent about three-quarters of the program’s rewarded activities.
As of January 1998, 58 percent of the 1996 awardees (22 of 38 banks) had completed all of the activities for which the Fund had announced an award. In addition, the Fund had disbursed nearly 80 percent of the 1996 award funds. The remaining 20 percent of the award funds were reserved for 16 banks that had not yet completed their planned activities or sought disbursement of their award funds. These 16 banks have just under 4 years to complete these activities and request disbursement of the associated award funds.
Four of our five case study banks did not complete as much lending and other financial service activity as they had expected. According to officials from three of these banks, they generally included activities in their applications that they believed would reach closing within the next 6-month assessment period, but some of these activities took longer to reach closing or never closed at all. Officials from one bank said that they did not find it unexpected or uncommon that, on occasion, a variety of factors could slow the normal process of negotiating loan terms, causing minor delays in moving a project to closing. For some of our case study banks, delays meant that a small number of the activities projected to close within the 6-month assessment period took longer than expected and closed after that period had ended. As a result, these banks increased their activities less than projected during the assessment period and will not receive the portions of their 1996 awards associated with the unaccomplished activities. Award funds that are not disbursed to banks remain with the Fund and are available for reallocation to future awardees or for use in the CDFI program as long as the budget authority for the funds has not expired.
section_title: Several Factors Obscure the Impact of BEA Awards on Banks’ Activities
paragraphs: According to officials at our case study banks and Fund officials, banks invest in CDFIs and provide loans and other financial services in distressed communities for a variety of reasons. In some cases, the prospect of receiving a BEA award may encourage such investment, but in other cases, regulatory or economic incentives exert a strong influence. As a result, it is difficult to isolate the impact of the BEA award from the effects of other incentives. Furthermore, more complete data on some banks’ investments are needed to ensure that the increases in investments in distressed areas rewarded by the BEA program are not being offset by decreases in other investments by the banks in the same areas. Finally, because the Fund did not require banks receiving 1996 awards to report any material changes in rewarded investments, it did not know whether rewarded investments remained long-term increases in the capital available to CDFIs or distressed communities.
subsections: section_title: Regulatory and Economic Incentives Also Motivate Increased Investment in CDFIs and Distressed Areas
paragraphs: Officials at our case study banks and Fund officials told us that the BEA program creates an incentive for or reduces the risk associated with a bank’s initiating and/or increasing investments in CDFIs and/or distressed communities. For example, according to an official at one of our case study banks, the bank would not have invested $250,000 in a local nonprofit CDFI’s predevelopment loan program if this activity had not been eligible for a 15-percent reward from the Fund. Typically, this official said, the bank does not support predevelopment loan programs, but the prospect of receiving an award mitigated some of the risk associated with this loan, enabling the bank to favorably consider and subsequently make this investment. As further evidence that the program is encouraging investment in CDFIs, a Fund official said that CDFIs have begun to market themselves to banks by noting that new or increased investments in them can result in awards from the Fund of up to 15 percent of the new or additional investment.
Our case study banks indicated, and Fund officials agreed, that the prospect of receiving a BEA award was not always the primary incentive for banks to undertake award-eligible activities. Rather, regulatory or economic incentives were important or had already prompted the banks to undertake such activities before they considered applying for a BEA award. One of the reasons our case study banks cited most frequently for undertaking award-eligible activities was to comply with the Community Reinvestment Act (CRA). CRA requires that federal bank regulatory agencies encourage the banks they regulate to help meet credit needs in all areas of the communities served (insofar as is consistent with safe and sound operations), assess the banks’ performance in meeting those needs, and take this performance into account when considering a bank’s request for regulatory approval of a regulated action, such as opening a new branch or acquiring or merging with another bank. Under CRA, banks do not receive direct financial or other rewards from the regulatory agencies for compliance. Also, the BEA program requires that banks not only invest in distressed areas but also increase their investments over and above what they have been doing.
The bank regulatory agencies assess each bank’s performance in meeting CRA’s requirements every 2 to 3 years. During these reviews, the examiners check to see whether CRA compliance activities are an ongoing part of the bank’s business. Because BEA-eligible activities, as a rule, count in terms of compliance with CRA, most banks are likely to be engaged in some activities that the BEA program rewards, and when the banks can show an increase in such activities over an assessment period, they can simultaneously demonstrate their compliance with CRA and their eligibility for a BEA award.
This overlap between activities that meet CRA’s requirements and are eligible for a BEA award held true at our case study banks. All of the activities that counted toward compliance also counted toward an award. Furthermore, two of the banks received BEA awards totaling over $324,000 for increases in investments they had made or agreed to make as part of ongoing working relationships with CDFIs that predated their applications for the awards. Another bank based its application for a $1.6 million award on those previously planned activities that bank officials judged most likely to reach closing during the bank’s 6-month assessment period. The Fund itself acknowledges that compliance with CRA may be a major incentive for some banks to undertake award-eligible activities. In fact, the Fund advertises the link to compliance with CRA in the promotional materials it distributes to banks when it publicizes the BEA program.
Economic incentives also motivated banks’ investments in award-eligible activities. According to officials at our case study banks, they do the kinds of business that the BEA program rewards in the markets targeted by the BEA program because they benefit from such business and/or have made a corporate commitment to community lending. The banks benefit, the officials said, because their award-eligible investments help them to maintain market share in areas targeted by the BEA program, to compete with other banks in these areas, or to build up markets that they expect will be profitable in the future. In addition, one bank indicated that its award-eligible activities lay the groundwork for building new markets with community groups, and two other banks cited improved community relations as a further incentive for their activities.
Although banks may derive future as well as current economic benefits from their award-eligible activities, these activities have to stand on their own without a subsidy from any of the banks’ other lines of business, according to officials at all five case study banks. These officials cited the solid track records of the CDFIs with which their banks were already working and/or the successful performance of their banks in distressed or low- and moderate-income communities as important reasons for continuing to do business there. The case study banks had measured their performance and demonstrated their success in distressed communities through measures such as loan repayment rates; reports on occupancy rates and the financial performance of housing projects financed by the banks; and, in one case, the frequent presence of a bank official on CDFI investees’ boards of directors.
Because awards in the BEA program can be relatively small (especially for larger banks) and are made retrospectively (that is, after banks have begun or completed activities), the awards may be too small or may come too late to have much influence on banks’ investment activities. To an extent that neither we nor the Fund can quantify, banks are receiving awards for activities that stem from ongoing relationships with CDFIs, nonprofit groups, or others in the community and/or for investments they would have made without the prospect of an award from the BEA program.
section_title: Some Banks Do Not Maintain Data Needed to Demonstrate Net Increases in Award-Eligible Activities in Distressed Communities
paragraphs: The Fund acknowledges that, in a few cases, some of the direct lending and other financial service activities it is rewarding—particularly agricultural, consumer, and small business loans—may come at the expense of similar eligible activities in the same distressed communities the awardees are serving. For this reason, the Fund was concerned that awardees could increase the loans and other services included in their applications while decreasing other eligible activities that are not included in their applications but are targeted to the same communities.
To be certain that the overall volume of activity in a distressed community has increased, a bank must know the location of the entity that received the loan, that the location is distressed, and that the overall volume of activity in that location has increased. To readily link a loan’s location with economic indicators of distress can require detailed information, such as data identifying the census tract in which the entity that received the loan is located. If a bank is not tracking the volume of its loan activity by census tract, the Fund cannot be sure that increases in some types of qualified activities have not been offset by decreases in other types of qualified activities in the same distressed community.
In reviewing the 1996 applications, the Fund discovered that some banks that were applying for awards on the basis of increases in these activities had not collected detailed information, such as loan activity by census tract, on the location of these activities. Consequently, they could not demonstrate that they had not decreased other award-eligible activities in distressed communities to support the increased activities included in their applications. For the 1996 awards, the Fund allowed banks without the requisite information to sign an addendum to their award agreement in which they stated that, to the best of their knowledge, they had not decreased other eligible activities that were not included in their application. While most such banks signed the addendum and submitted an application, an official at one of our case study banks said that the bank sought an award on the basis of its investments in CDFIs but not on the basis of any of its lending in distressed communities because its data on loans did not include census tract information and it was not willing to risk the chance that its overall level of activity in distressed communities might have decreased.
For the 1997 awards, the Fund hired a contractor to help banks determine if the areas they were seeking to target would be eligible under the BEA program’s guidelines. The Fund requires the banks to use this contractor’s services in the initial stages of applying to the program. The contractor determines which census tracts in a bank’s service area meet the Fund’s definition of a distressed community but does not identify which activities took place in those tracts. Many banks were not identifying the census tracts of their consumer, small business, or agricultural loans and could not have determined which ones took place in a distressed community identified by the Fund’s contractor unless they hand-coded or geocoded their lending data. According to a Fund official, using the Fund’s contractor means that each applicant has a better—but not perfect—idea of which loans in its service area took (or will take) place in distressed communities. Nonetheless, a bank that has not geocoded its loans would likely still find it very time consuming to assign a census tract to all of its consumer, small business, and agricultural loans because it would have to do so for all of these loans, not just those included in its application to the BEA program.
According to Fund officials, more and more banks are collecting census data on their loans and fewer banks had to sign the addendum for the 1997 awards than for the 1996 awards. However, because the Fund does not require banks to collect census data on their loans, Fund officials acknowledged that, to some extent, the Fund may still be rewarding banks that increased some activities at the expense of others that were not included in their applications but also took place in the same distressed communities. As a result, for banks that receive awards for direct lending and financial services but do not collect census data on these activities, the Fund cannot say with certainty that, in all cases, the BEA program is meeting its objective of rewarding increases in distressed communities because it cannot say with certainty that, in all cases, the rewarded increases did not come at the expense of other award-eligible activities.
section_title: Fund Was Not Systematically Informed of Material Changes in Banks’ Rewarded Investments
paragraphs: The 1996 BEA awardees were not obligated to report information, including any material changes in their rewarded investments, once they completed the tasks they had agreed to perform with their own funds to receive their awards. One of our case study banks received an award for increasing its investment in a CDFI that was later dissolved. Specifically, the bank received $37,500, or 15 percent of the $250,000 by which it increased its investment in a certified CDFI during its assessment period. However, several months after the bank received the award funds, the CDFI’s board of directors voted to dissolve the institution because of a declining market for its loans and dissatisfaction with its performance. Upon dissolving the CDFI, the board returned its remaining capital pro rata to the banks that had invested in it, including the bank that had received the award for its increased investment. Thus, even though the bank received an award from the BEA program, the rewarded increase in activity did not last. After our visit and the CDFI’s dissolution, the bank made a commitment to use all of the funds returned to it to capitalize a microloan fund at a different certified CDFI.
The Fund was not aware of this material change in our case study bank’s rewarded investment until we brought it to the Fund’s attention. Furthermore, because neither this nor any other 1996 awardee was required to report any material change, the Fund had no systematic means of learning of any significant change in a bank’s rewarded activity. The Fund has since established a requirement for awardees to report any material change in their rewarded investments so that it will be systematically informed of any important reduction in these investments.
section_title: Banks Are Using Award Funds for Additional Community Development Lending and Investment
paragraphs: The CDFI Act does not require banks to report how they use their BEA awards because the objectives of the BEA program and the rules governing it apply only to how the banks use their own funds. However, according to the Fund, most of the 1996 awardees reported using their awards to further the objectives of the BEA program. Each of our five case study banks also reported using its award money to expand its existing investments in community development. For example, one of the banks said that it used its award money to establish a community development leadership curriculum and training program, addressing topics such as innovative economic development and affordable housing strategies. This bank expects that its support will enable the group developing the curriculum to provide training to 100 senior managers from community development organizations. Another of the banks reported using its award funds to make a grant to the National Community Capital Association (formerly the National Association of Community Development Loan Funds) to enable the association to train CDFI staff and board members.However, neither we nor the Fund determined whether the banks used all or a portion of their award funds to benefit communities meeting the same eligibility criteria as those that benefited from the initial increases in the banks’ investments.
section_title: Agency Comments
paragraphs: In commenting on a draft of this report, the Fund’s Director and Deputy Director for Policy and Programs told us that, in response to the information we presented on a material change in one bank’s rewarded investment, the Fund has adopted a requirement for banks to report any material change in their rewarded investments. As a result, we are no longer making the recommendation to this effect that appeared in our draft report. We also made minor technical and clarifying changes to this chapter suggested by the Fund.
section_title: The Fund’s Strategic Plan Does Not Yet Fulfill the Requirements of the Results Act
paragraphs: The Fund’s current strategic plan contains the six basic elements required by the Results Act, but these elements generally lack the clarity, specificity, and linkage with one another that the act envisioned. Although the plan identifies key external factors that could affect the Fund’s mission, it does not relate these factors to the Fund’s strategic goals and objectives and does not indicate how the Fund will take the factors into account when assessing awardees’ progress. In addition, the plan does not explicitly describe the relationship of its activities to similar activities in other government agencies, and it does not indicate whether or how the Fund coordinated with other agencies in developing its strategic plan. Additionally, the capacity of the Fund to provide reliable information on the achievement of its strategic objectives at this point is somewhat unclear.
The Fund’s difficulties in developing a strategic plan are not unique. We have found that strategic planning efforts at all federal agencies are still works in progress. Many agencies are struggling, like the Fund, to set a strategic direction, coordinate crosscutting programs, and ensure the capacity to gather and use performance and cost data. As the Results Act directs, the Fund is taking steps to refine its strategic plan. These steps appear to address the difficulties we observed in the current plan.
subsections: section_title: Requirements of the Results Act
paragraphs: The Results Act requires that an agency’s strategic plan contain six key elements: (1) a comprehensive mission statement; (2) agencywide long-term (strategic) goals and objectives for all major functions and operations; (3) approaches (or strategies), skills, technologies, and the various resources needed to achieve the goals and objectives; (4) a description of the relationship between the long-term goals and objectives and the annual performance goals; (5) an identification of key factors, external to the agency and beyond its control, that could significantly affect the achievement of the strategic goals and objectives; and (6) a description of how program evaluations were used to establish or revise strategic goals and objectives and a schedule for future program evaluations.
OMB Circular A-11 provides agencies with additional guidance on developing their strategic plans and discusses additional information that they may include in those plans. The circular emphasizes the importance of agencies’ strategic plans, noting that they “provide the framework for implementing all other parts of  Act, and are a key part of the effort to improve performance of government programs and operations.”
Because the plan matches programs and activities to the agency’s mission and objectives, it can be used, according to the circular, to align the organization and budget structure of the agency with its mission, guide the agency in formulating its budget, and help the agency set priorities and allocate resources in accordance with these priorities. The Results Act anticipates that an agency may take several planning cycles to refine and perfect its strategic plan.
section_title: Opportunities for Improving the Fund’s Strategic Plan
subsections: section_title: Mission Statement
paragraphs: “the mission of the CDFI Fund, as drawn from the Riegle Community Development and Regulatory Improvement Act of 1994, is to promote economic revitalization and community development through investment in and assistance to community development financial institutions (CDFIs) and through encouraging insured depository institutions to increase lending, financial services and technical assistance within distressed communities and to invest in CDFIs.”
This statement generally satisfies the requirements of the Results Act and OMB Circular A-11 for a comprehensive mission statement summarizing an agency’s major functions. The statement explicitly refers to the Fund’s statutory objectives and indicates how these statutory objectives are to be achieved through the two core programs.
section_title: Strategic Goals and Objectives
paragraphs: The Results Act and OMB Circular A-11 require that each agency’s strategic plan set out goals and objectives for the major functions and operations of the agency and that the goals and objectives elaborate on how the agency is carrying out its mission. The Fund’s strategic plan articulates five goals, each with at least two objectives, as shown in table 4.1.
Although the plan does not characterize the goals as such, we observed that the first three goals are mission related and the last two are organizational. We also noted that the first three goals are results oriented and cover the major functions and operations of the Fund, as the Results Act and OMB Circular A-11 direct. However, the last two goals are process oriented—characterizing how the Fund intends to improve the performance and operations of its own programs instead of projecting an outcome resulting from the Fund’s actions. Because processes are not goals, these two would be more appropriately incorporated in the strategies needed to achieve strategic goals and objectives.
OMB Circular A-11 suggests that strategic goals and objectives be stated so as to allow a future assessment of their accomplishment. Because none of the 5 goals and 13 objectives in the strategic plan include baseline dates and values, deadlines, and target markets, the Fund’s goals and objectives do not meet this criterion. In contrast, one of the Federal Emergency Management Agency’s strategic goals is to “protect lives and prevent the loss of property from all hazards.” An objective for achieving this goal is “by the end of fiscal year 2002,  reduce by 10 percent the risk of loss of life and injury from hazards.” By including specific deadlines and targets, this objective meets the criterion.
section_title: Strategies to Achieve Goals and Objectives
paragraphs: The Results Act requires that each agency’s strategic plan describe how the agency’s goals and objectives are to be achieved. OMB’s guidance suggests that the plan briefly describe the operational processes; the skills and technologies; and the human, capital, information, and other resources needed to achieve the goals and objectives. Additionally, Circular A-11 recommends that strategies outline how agencies will communicate strategic goals throughout the organization and hold managers and staff accountable for achieving these goals. The Fund’s plan shows mixed results in meeting these requirements.
On the positive side, the plan clearly lists strategies for accomplishing each goal and objective. This approach is preferable to other approaches we have seen in which strategies were not integrally linked to objectives. Because the Fund’s plan more clearly links the key strategies, objectives, and goals, it is more valuable to users.
While the links in the Fund’s strategic plan are clear, the strategies outlined in the plan consist entirely of one-line statements. Because they generally lack detail, it is unclear whether their accomplishment would help achieve the plan’s goals and objectives. For example, it is unclear how “emphasizing high-quality standards in implementing the CDFI program” will strengthen and expand the national network of CDFIs.
Additionally, in discussing strategies to achieve its goals and objectives, the Fund’s strategic plan does not, as the Results Act requires, describe the resources—such as the staff, capital, and technologies—that are needed to achieve the objectives. Rather, the Fund’s plan contains a separate section that describes, in general terms, the resources needed to implement the entire plan. Specifically, this section states that “the Fund will use many resources to accomplish its goals, including anticipated appropriations, the knowledge and skills of its staff, information technology, financial systems, and operational processes.” However, it is not clear how these resources will be used to implement specific objectives. For example, one strategy—developing and implementing a secondary market initiative—is proposed for achieving one objective—increasing liquidity for CDFIs—but the resources needed to carry out this strategy are not specified.
The Fund’s strategic plan also does not include several elements specified in Circular A-11. For example, the plan does not include (1) schedules for initiating or completing significant actions, including underlying assumptions, or (2) an outline of the processes for communicating the goals and objectives to the Fund’s staff and for assigning accountability to managers and staff for achieving the strategic plan’s objectives.
section_title: Relationship Between Strategic and Annual Performance Goals
paragraphs: Under the Results Act, each strategic goal must be linked to annual performance goals. A performance goal is the target level of performance expressed as a tangible, measurable objective against which actual achievement is to be compared. An annual performance goal is to consist of two parts: (1) the measure that represents the specific characteristic of the program used to gauge performance and (2) the target level of performance to be achieved during a given fiscal year for the measure. While strategic plans are not required to identify specific performance measures, OMB Circular A-11 recommends that the plans briefly relate strategic goals and objectives to annual performance goals. The guidance suggests that the plans also include descriptions of the type, nature, and scope of the performance goals included in the performance plans, as well as the relevance and use of those performance goals to help determine the achievement of the strategic goals and objectives.
The Fund’s strategic plan lists 22 performance goals, which are clearly linked to specific strategic goals. However, the plan does not include a key performance goal—leveraging other resources—to meet two of its strategic goals, one of which is “to strengthen and expand the national network of CDFIs” and the other of which is to “encourage investments in CDFIs by insured depository institutions.” While this leveraging goal is embedded in the strategies that the Fund has outlined for achieving these two strategic goals, it is not included as a way to gauge progress under the annual performance goals.
Furthermore, the performance goals generally lack the specificity, as well as the baseline and end values, that would make them more tangible and measurable. For example, one performance goal is to “increase the number of applicants in the BEA program.” This goal would be more useful and measurable if it specified the baseline number of applicants as well as the number of additional applicants projected within a specific time frame.
Finally, some performance goals are stated more as strategies (e.g., “propose legislative improvements to the BEA program” or “survey program participants on policies and standards”) than as desired results, and the ways in which individual performance goals support strategic goals is not always clear. For instance, it is not readily apparent how the performance goal of proposing legislative improvements to the BEA program will support the strategic goal of encouraging banks’ investments in CDFIs.
section_title: Key External Factors
paragraphs: The Fund’s strategic plan only partially meets the requirement of the Results Act to describe key factors that are external to an agency and beyond its control that could significantly affect the achievement of its objectives. OMB Circular A-11 states that a strategic plan should describe each key external factor, indicate its link with a particular strategic objective, and describe how the achievement of the objective could be affected by the factor. The Fund’s plan briefly discusses external factors that could materially affect its performance, such as national and regional economic trends and changes in the demographics of the labor force that may require the development of a multifaceted and flexible human resource program. However, the plan does not link these external factors to specific strategic objectives. In addition, the plan does not cover all key external factors that could materially affect the Fund’s performance. For instance, the plan does not mention the continuation of outside funding, yet, as indicated in chapter 1, CDFIs must obtain outside funding to be eligible for participation in the CDFI program. Without the continuation of outside funding, the Fund’s ability to expand the network of CDFIs could be substantially diminished.
section_title: Use of Program Evaluations
paragraphs: The Results Act, supplemented by OMB’s guidance, requires that strategic plans describe (1) the program evaluations used to prepare the plans and (2) the schedule for future evaluations. The Fund’s strategic plan generally does not discuss the evaluations used in its development. Although the plan refers to past evaluations by the Department of the Treasury’s Office of Inspector General—which the Fund says were used to assist in the “development of the Fund’s programs and design of the Fund’s internal systems,” it is not clear how, or if, these evaluations were used to develop the goals, objectives, and strategies outlined in the Fund’s strategic plan.
The Results Act also requires a discussion of completed and future program evaluations, which can be a critical source of information to ensure the validity and reasonableness of goals and objectives and to explain results in the agency’s annual performance plan. The Results Act defines program evaluations as assessments, through objective measurement and objective analysis, of the manner and extent to which federal programs achieve their intended objectives. The Fund’s plan does discuss options that the Fund is considering for evaluating its own effectiveness and its impact on financial intermediaries dedicated to supporting community development. However, the plan does not include a schedule for future program evaluations. Furthermore, the list of options does not refer to the CDFI and BEA evaluations or surveys described in earlier sections of the plan.
section_title: Strategic Plan Could Better Address Crosscutting Activities
paragraphs: In its strategic plan, the Fund states that it will “coordinate its strategies with other Treasury bureaus and agencies with similar missions.” The Fund’s strategic plan does not specifically address the relationship of the Fund’s activities to similar activities in other agencies and does not indicate whether or how the Fund coordinated with other agencies in developing the strategic plan. Yet numerous government and private-sector agencies are involved in providing access to capital to achieve community and economic development. Interagency coordination is important for ensuring that crosscutting programs are mutually reinforcing and efficiently implemented. Therefore, the plan would be strengthened if it identified and incorporated some descriptions of other agencies’ programs with similar missions and discussed their influence on the Fund’s strategic objectives. For example, a recent study published by the President’s Community Empowerment Board identified several programs in other federal agencies with missions similar to that of the CDFI program, including the following:
The departments of Agriculture and of Housing and Urban Development administer the Empowerment Zone and Enterprise Community program, which was authorized to revitalize deteriorating urban and rural communities. This program targets federal grants to distressed communities for community redevelopment and social services and provides tax and regulatory relief to attract or retain businesses in the communities. In its objectives and in the types of communities it targets, this program is similar to the CDFI program. Furthermore, the program’s structure mirrors that of the CDFI program in that applicants must submit a strategic plan that, like the CDFI program’s assistance agreements, must outline baselines, methods, and benchmarks for measuring their success in the targeted communities. Finally, performance is tracked in both programs to measure the impact of awardees’ activities in the distressed communities.
The Small Business Administration (SBA) operates various programs to aid, counsel, assist and protect the interests of small businesses. For example, the Small Business Investment Company (SBIC) and the Specialized Small Business Investment Company (SSBIC) programs are designed to fill the gap between the availability of venture capital and the needs of small businesses that are starting up or growing. SBICs, which are licensed and regulated by SBA, are privately owned and managed investment firms that use their own capital, plus funds borrowed at favorable rates with an SBA guarantee, to make equity investments and/or loans to small businesses. SSBICs invest in small businesses owned by entrepreneurs who are socially or economically disadvantaged. Not only are the objectives of these programs consistent with those of the Fund, in that they provide access to capital for economic development, but they are also similar to those of venture capital CDFIs that provide financial and technical assistance to start-up businesses. Moreover, they are structured similarly to the Funds’ CDFI and BEA programs in that they provide access to capital by leveraging federal resources. SBA also administers a microloan program that increases the availability of very small loans to prospective small business borrowers. Under this program, SBA makes funds available to nonprofit intermediaries, which in turn make loans to eligible borrowers, much as the CDFI program makes funds available to CDFIs, which then make loans to microenterprises.
Among private organizations, the Ford Foundation is the largest supporter of community development. Specifically, it supports efforts to create economic opportunities and financial institutions that respond to the needs of the poor, as well as efforts to give the poor greater ownership and control of key community institutions. Several of the CDFIs that received 1996 awards from the Fund also received funding from the Ford Foundation.
section_title: The Fund Lacks Data to Measure Its Progress
paragraphs: To measure progress in achieving its strategic objectives, the Fund needs reliable data. The Fund has not yet developed its strategic plan sufficiently to identify the types and sources of data needed to evaluate its progress towards the objectives outlined in the plan.
Moreover, according to the KPMG Peat Marwick study identified in chapter 2, as of February 1998, the Fund has yet to set up a formal system, including procedures to continuously monitor, evaluate, and improve the effectiveness of the management controls associated with the CDFI program. These procedures would ensure that the periodic performance reports submitted by awardees are received, reviewed and acted upon by the Fund in the event of potential noncompliance. Until the Fund identifies the types of data needed to monitor and evaluate awardees and incorporates these data needs in a formal system, it will be hampered in its ability to report on its progress toward achieving its stated goals and objectives. The Fund intends to continue designing and developing a portfolio-monitoring database system during fiscal year 1998 as part of its efforts to design and implement its monitoring procedures.
section_title: Weaknesses in the Fund’s Strategic Plan Are Common Among Federal Agencies
paragraphs: The Fund’s strategic plan has shortcomings common to the plans of most other federal agencies. We reported on these shortcomings in our September 1997 review of 27 agencies’ draft strategic plans. We found that a significant amount of work remained to be done by executive branch agencies before their strategic plans can fulfill the requirements of the Results Act, serve as a basis for guiding agencies, and help congressional and other policymakers make decisions about activities and programs. Although all 27 of the draft plans included a mission statement, 21 plans lacked 1 or more of the other required elements. In summary, for the 27 draft strategic plans, we found that (1) most did not adequately link required elements in the plans; (2) several contained goals that were not as results oriented as they could have been; (3) many agencies did not fully develop strategies explaining how their long-term strategic goals would be achieved; (4) most agencies did not identify or provide for coordinating activities and programs that cut across multiple agencies; (5) the limited capacity of many agencies to gather performance information has hampered their efforts to identify appropriate goals and confidently assess their performance; and (6) no agency’s draft strategic plan provided adequately for program evaluations.
section_title: Recent Strategic Planning Initiatives Address Shortcomings in the Fund’s Current Plan
paragraphs: As is consistent with the Results Act’s requirement that agencies continually refine their plans, the Fund is updating its strategic plan and expects to have a revised plan by August 1998. According to a key Fund official, the updated plan will address not only the shortcomings we identified in a May 1998 authorization hearing on the Fund, but also those cited in the Department of the Treasury’s February 1998 review of the Fund’s plan. The following are among the key changes the Fund plans to make: In revising its strategic goals, it plans to eliminate the two organizational goals (i.e., to improve program performance and management operations) included in table 4.1 because they are not directly related to the Fund’s mission. It plans to change the format for presenting the goals and objectives by linking benchmarks and planned evaluations to each goal, as well as to the key external factors that could affect the Fund’s ability to meet these goals. The Fund believes that this change will improve its ability to assess how well its strategies and approaches for meeting its strategic goals are working. For example, one of the goals proposed in the Fund’s revised plan is to increase participation in the CDFI program. The Fund plans to hold workshops to increase participation in the program and to evaluate this strategy by collecting data to track the number of (1) participants in the workshops, (2) participants who apply for CDFI funding, and (3) applicants who receive CDFI awards. The Fund plans to use this information to evaluate the effectiveness of the workshops in increasing participation. It plans to revise its budget structure to better link program activities with funding sources. By presenting the budget this way, the Fund expects to improve its tracking of the resources used to implement the goals and objectives outlined in its strategic plan, as well as to develop annual plans that tie these goals and objectives to its budget. It plans to revise its performance goals to include a measure of its ability to leverage other resources. It plans to identify crosscutting organizations and programs and assess the extent to which its programs duplicate or complement these efforts.
section_title: Conclusions
paragraphs: The Fund is revising its strategic plan to address the major shortcomings we observed in the current plan. Through these revisions, the Fund will have better defined its goals and better identified its strategies for achieving them, including its strategy for allocating its resources, thereby laying the foundation for determining its success in reaching results-oriented as well as non-results-oriented goals and objectives. Because strategic planning is not static, the Fund will need to continuously revise and refine its strategic plan to reflect the dynamic nature of the CDFI industry. If this process is done well, the Fund’s strategic planning efforts should facilitate informed communication between the Fund and its stakeholders—that is, those organizations potentially affected by or interested in the Fund’s activities.
section_title: Agency Comments
paragraphs: The Fund officials agreed with the information in this chapter and, as we noted earlier, are taking steps to refine the Fund’s strategic plan. These steps appear to address the difficulties we observed.
section_title: Objectives, Scope, and Methodology
paragraphs: The legislation creating the Fund required that GAO report on the Fund’s structure, governance, and performance 30 months after the appointment of an Administrator of the Fund. However, as noted in chapter 1 and as agreed with your offices, this report does not review the structure and governance of the Fund because the Department of the Treasury’s Inspector General is conducting an audit addressing these issues. This report discusses (1) the progress of the Community Development Financial Institutions (CDFI) Fund in developing performance measures for awardees in the CDFI program and systems to monitor and evaluate their progress in meeting their performance goals, as well as the accomplishments they have reported to date; (2) the performance of banks under the Bank Enterprise Award (BEA) program, the impact of the program on banks’ activities and on distressed communities, and the uses to which banks have put their award funds; and (3) the Fund’s progress in meeting the Results Act’s requirements for strategic planning and the steps the Fund could take to improve its management. Our report focuses on the first round of awards, which the Fund made in 1996, and draws on our interviews with Fund officials; our case studies of awardees, including six CDFIs in the CDFI program and five banks in the BEA program, which we conducted to explore our objectives in more depth; the results of our survey of the CDFI field on its use of performance measures; and our analysis and review of the CDFI program’s assistance agreements.
To meet our objectives in reviewing the CDFI program, we reviewed the Fund’s process for setting goals and developing performance measures with the 31 1996 awardees and discussed the Fund’s performance measurement system—including its development, operation, and underlying assumptions—with various Fund officials responsible for working with the awardees. In addition, to supplement the information about the CDFI program that we gathered at the Fund’s headquarters in Washington, D.C., we randomly selected six awardees as case studies to gain these awardees’ perspectives on the process of developing performance measures, as well as to gather data that the Fund does not collect, such as information on the reporting requirements that other funding sources impose on awardees and the performance measures that these sources require.
We randomly selected these case studies from the universe of 24 awardees that the Fund told us had signed their assistance agreements by November 1, 1997. Because the CDFI program serves a wide variety of community development organizations, we stratified our random selection by the types of CDFI awardees described in chapter 1 to ensure that our case studies included at least one of each of the most common types of CDFIs. This stratification generally mirrored the categorization used by the CDFI Fund in its first annual report. If a CDFI category was represented by only one or two awardees, then that category was combined with the closest similar category. The six case study awardees were a community development bank holding company, a community development venture capital fund, a community development credit union, a microenterprise fund, a community development loan fund, and a multifaceted community development financial institution.
We obtained information from the awardees on the negotiations that took place between each of them and the Fund to develop the performance goals, measures, and benchmarks outlined in their assistance agreements. We discussed this information with knowledgeable Fund staff and reviewed documentation pertinent to the activities for which awards were made, including the awardees’ business plans, assistance agreements, and correspondence about the negotiation process.
To describe the performance measurement and monitoring systems used in the CDFI field, we conducted a national survey of CDFI organizations. To identify these organizations, we began by obtaining from the CDFI Fund in October 1997 its most recent list of certified CDFIs, dated June 1997, as well as lists of applicants for awards in the first (1996) and second (1997) funding rounds. In addition, we obtained membership lists from the following national community development professional organizations: the CDFI Coalition, the National Association of Community Development Credit Unions, the National Community Capital Association, and the Community Development Venture Capital Alliance. We also obtained lists from the Neighborhood Reinvestment Corporation and the Aspen Institute that identified neighborhood housing services and microenterprise development organizations with loan funds. Our objective was to survey not only certified CDFIs but also other CDFIs that might be eligible for certification. Using these lists, we identified a total of 925 organizations that described themselves as CDFIs. We do not believe that our list includes all such organizations nationwide. We recognize that there are probably other community development organizations that could be certified by the Fund as a CDFI, but are currently unknown to either the Fund or one of the national CDFI associations.
To encourage responses, we sent follow-up letters and a second survey to those organizations that did not return a survey after the first mailing. In total, 623 institutions responded to our questionnaire, for a 67-percent response rate. Respondents included 87 percent of the 1996 awardees and 77 percent of the 187 CDFIs certified by the Fund as of June 1997.
To categorize the goals and measures used by the 26 awardees that had signed their assistance agreements with the Fund by February 1998, we conducted a content analysis of the performance goals and measures included in their agreements. The questionnaire used for the survey of all CDFIs provided the framework for this analysis. The questionnaire contains an extensive list of responses that are grouped by the categories of accomplishments and activities for goals and measures. For example, one question asks about measures of community development accomplishments and lists 12 measures that an institution might use. The assistance agreements contained 87 goals and 165 measures. Each goal and each measure was classified as a specific activity or accomplishment by two evaluators who worked independently. Differences were resolved with a third evaluator, so that the evaluators reached complete agreement on their classification of goals and measures.
We also evaluated aspects of the quality of the goals and measures in the assistance agreements, using criteria for evaluation developed on the basis of the Results Act, supplemented by Circular A-11, the Office of Management and Budget’s (OMB) implementing guidance, and a GAO document entitled The Results Act: An Evaluator’s Guide to Assessing Agency Annual Performance Plans (GAO/GGD-10.1.20, Apr. 1998). These three documents provide more explicit guidance for developing performance goals and measures than the CDFI legislation. We refer to these three documents as “the Results Act and its guidance” throughout this appendix. These evaluation criteria included the specificity, objectivity, completeness, and appropriateness of the measures. An additional criterion was whether or not the goals and measures addressed the purpose of the CDFI program, that is, to promote economic revitalization or economic development.
Using these criteria, we constructed the following set of questions:
Does the goal promote economic revitalization and community development?
Does each measure have a clearly apparent or commonly accepted relationship to the intended result?
Are only certain observations in the measure?
Is what is to be observed or measured specified? Is any description of what is being measured given?
Do all the measures use terms that are generally known, or if not known, are they described? Is an evaluation date (target date) given? Is an evaluation value (target value) given? Is a baseline date given? Is a baseline value given? Is a target population or geographic area given?
Each question could be answered with a yes or a no. For nine questions, the unit of analysis was an individual goal and its associated measure(s), referred to as a “goal statement.” For the two remaining questions, the unit of analysis was an individual measure. Two teams of two GAO evaluators performed the assessments; each team assessed approximately half of the goal statements. The two members of each team independently assessed each goal statement and then compared their answers. Disagreements were discussed and resolved. After the members of each team reached agreement, we compared the assessments performed by the two teams, reviewing the ratings each had given to a subset of 8 goal units (11 measures) that both teams had assessed. Reliability (percentage of concurrence on ratings) between the teams was 96 percent.
We finished categorizing the goals and measures in the 26 assistance agreements by reviewing all of the goals and measures to determine whether the measures in each goal statement addressed all key aspects of each goal. We drew on our knowledge of the CDFI program, community development and housing finance, and OMB’s guidance on the Results Act to make this judgmental determination and to identify instances in which measures did or did not address all key aspects of the associated goals.
Finally, we reviewed the Fund’s statutory, regulatory, and other reporting and monitoring requirements, as well as an existing study of the Fund’s monitoring system, and we held discussions with Fund officials and officials at selected CDFI case studies to assess the Fund’s progress in developing systems to monitor and evaluate the progress of the CDFI awardees in meeting their performance goals. We also reviewed quarterly progress reports submitted by 19 of the 1996 awardees and held discussions with case studies officials to assess the awardees’ progress.
For the BEA program and its awardees, we performed work at Fund headquarters similar to that we performed for the CDFI program, reviewing the Fund’s guidance, policies, procedures, and other materials on the awards process and discussing these issues and others related to the program with the Fund officials administering the program. For data on the banks’ performance under the program, we relied on the Fund’s status report on the activities completed by awardees as of January 1998. This report includes the status of disbursements made by the Fund for specific activities completed in accordance with the awardees’ agreements with the Fund.
We conducted case studies of selected BEA awardees to explore our objectives in more depth, obtaining awardees’ perspectives on the process of applying for an award and gathering information from awardees that the Fund does not collect systematically, such as information on the incentives banks identify for participating in the program and the ways they monitor and measure the progress of their investments. We also verified with these awardees (1) the information we obtained from the Fund’s January 1998 status report and (2) the information the banks voluntarily reported to the Fund on their uses of award funds. We judgmentally selected a sample of five awardees for our case studies, ensuring that collectively they represented the full range of activities for which banks can receive awards. The five awardees and the activities on which their awards were based were as follows:
Fullerton Savings and Loan, Fullerton, California (increased lending for single-family and multifamily housing);
Bank of America Community Development Bank, Walnut Creek, California (increased lending for commercial real estate, multifamily housing, and small businesses);
First Union Bank, Washington, D.C. (increased lending for multifamily
Citibank, N.A., New York, New York (increased investments in CDFIs); and,
Wells Fargo Bank of Texas, N.A., Houston, Texas (increased investments in CDFIs).
To identify opportunities for improving the Fund’s management, we asked officials we had interviewed, both at Fund headquarters and at our CDFI and BEA case studies, to identify ways of improving the 1996 awards, monitoring, and performance measurement processes. We also reviewed studies of the Fund that outlined areas for improvement and determined the extent to which these areas had or had not been addressed in the 1997 process.
Our overall assessment of the Fund’s strategic plan was generally based on our knowledge of the Fund’s operations and programs, as well as on other information available at the time of our assessment. Specifically, the criteria we used to determine whether the Fund’s strategic plan complied with the requirements of the Results Act were the Results Act and OMB’s guidance on developing strategic plans (OMB Circular A-11, Part 2). To make judgments about the overall quality of the plan, we used our May 1997 guidance for congressional review of strategic plans (GAO/GGD-10.1.16) as a tool. To determine whether the plan contained information on interagency coordination, we relied on the President’s Community Empowerment Board’s Federal Programs Guide. Finally, we reviewed the Fund’s fiscal year 1997 annual report and consulted with knowledgeable staff in our Accounting and Financial Management Division as part of our efforts to assess whether the Fund had adequate systems in place to provide reliable information on performance.
We conducted our work at Treasury headquarters and at the offices of selected CDFI and BEA awardees throughout the country. We performed our review between July 1997 and June 1998 in accordance with generally accepted government auditing standards. We relied on data provided to us by the Fund, by awardees in both the CDFI and the BEA programs, and by CDFIs responding to our survey. We did not verify the data on award amounts, disbursements, or quarterly and annual reports for the CDFI awardees as a whole or for our case studies because our work focused on the process of developing performance goals and measures, which were themselves documented in the assistance agreements. Because our conclusions and recommendations are based on that process and the resulting goals and measures, and not on the financial or performance data, we consider them to be valid. We verified with the BEA awardees the data on award amounts, disbursements, and postaward uses of funds that we obtained from the Fund.
section_title: Major Contributors to This Report
subsections: section_title: Resources, Community, and Economic Development Division, Washington, D.C.
paragraphs: Leslie Black-Plumeau Carolyn Boyce Diane Brooks Stan Czerwinski Patricia Farrell Donahue Elizabeth R. Eisentadt Dennis Fricke Kimberly Hutchens Bill MacBlane John McGrail Lynn Musser Hattie Poole Marilyn Rubin James Vitarello John Vocino The first copy of each GAO report and testimony is free. Additional copies are $2 each. Orders should be sent to the following address, accompanied by a check or money order made out to the Superintendent of Documents, when necessary. VISA and MasterCard credit cards are accepted, also. Orders for 100 or more copies to be mailed to a single address are discounted 25 percent.
U.S. General Accounting Office P.O. Box 37050 Washington, DC  20013 Room 1100 700 4th St. NW (corner of 4th and G Sts. NW) U.S. General Accounting Office Washington, DC Orders may also be placed by calling (202) 512-6000  or by using fax number (202) 512-6061, or TDD (202) 512-2537.
Each day, GAO issues a list of newly available reports and testimony.  To receive facsimile copies of the daily list or any list from the past 30 days, please call (202) 512-6000 using a touchtone phone.  A recorded menu will provide information on how to obtain these lists.