Report to Congressional Committees

United States General Accounting Office

GAO

August 2001 RESIDENTIAL

ENERGY

ASSISTANCE

Effectiveness of

Demonstration

Program as Yet

Undetermined

GAO-01-723
Page i GAO-01-723 REACH Program Effectiveness

Letter 1

Appendix I REACH Project Activities 29

Appendix II Review of California’s REACH Project Evaluation

Report 30

Appendix III Review of Maryland’s REACH Project Evaluation

Report 34

Appendix IV Review of Massachusetts’ REACH Project

Evaluation Report 39

Appendix V Review of Michigan’s REACH Project Evaluation

Report 44

Appendix VI Review of Nebraska’s REACH Project Evaluation

Report 49

Appendix VII Review of Oregon’s REACH Project Evaluation

Report 53

Appendix VIII Comments From the Department of Health and

Human Services 59

Appendix IX GAO Contact and Staff Acknowledgments 72

Contents
Page ii GAO-01-723 REACH Program Effectiveness

Tables

Table 1: REACH Grant Funding by Fiscal Year and Type of Grant 8

Table 2: Grant Awards to States, by Fiscal Year 8

Table 3: Grant Awards to Tribes, Tribal Organizations, and Insular

Areas, by Fiscal Year 9

Table 4: Activities Planned by REACH Grantees 29

Abbreviations

DOE Department of Energy

HHS Department of Health and Human Services

LIHEAP Low-Income Home Energy Assistance Program

OCS Office of Community Services

PRISM Princeton Scorekeeping Method

REACH Residential Energy Assistance Challenge Option
Page 1 GAO-01-723 REACH Program Effectiveness

August 17, 2001

The Honorable Edward M. Kennedy, Chairman

The Honorable Judd Gregg, Ranking Minority Member

Committee on Health, Education, Labor and Pensions

United States Senate

The Honorable John A. Boehner, Chairman

The Honorable George Miller, Senior Democratic Member

Committee on Education and Workforce

House of Representatives

Paying for home heating and cooling can be difficult for low-income

families. With this year’s rising prices for natural gas, electricity, and other

fuels, meeting utility bills has become even harder for the nation’s neediest

citizens. The federal government has spent about $1.8 billion in each of the

past 2 years to help low-income households pay for home heating and

cooling through the Low-Income Home Energy Assistance Program

(LIHEAP). Within this larger program, the Congress established the

Residential Energy Assistance Challenge Option (REACH) program, which

provides grants that fund demonstration projects to test various

approaches to help low-income families reduce their energy usage and

become more self-sufficient in meeting their home energy needs. In a

sense, the REACH program serves as a “laboratory” for identifying better

ways to ensure that low-income families can afford home heating and

cooling.

Through the REACH program, the Office of Community Services (OCS)—

part of the Administration for Children and Families in the Department of

Health and Human Services (HHS)—provides grants to states and tribal

organizations to conduct demonstration projects. The states must conduct

these projects through community-based organizations; tribal

organizations may conduct projects themselves or through other

community-based organizations. OCS has funded 54 REACH grants since

the program began in fiscal year 1996, using an annual budget of about

$6 million, which is about one-half of 1 percent, on average, of the total

funding for LIHEAP. REACH projects have provided weatherization

materials, workshops on energy efficiency measures for homes, and

budget counseling, and have formed consumer cooperatives to purchase

home energy. State projects run for 3 years, and state grantees are

required to contract for third-party evaluations and to report after the

United States General Accounting Office

Washington, DC 20548
Page 2 GAO-01-723 REACH Program Effectiveness

conclusion of the projects on the effectiveness of the approaches that they

have tried.

In the Low-Income Home Energy Assistance Amendments of 1998, the

Congress required that we evaluate the REACH program and include in

our report information on states’ evaluations of their projects. (Tribal

projects are not required to do evaluations). As agreed with your staff, our

report addresses (1) grant recipients, amounts, and project activities;

(2) the goals of the REACH program; (3) the methodologies and results of

states’ project evaluations; and (4) plans to communicate lessons learned

from REACH projects and to foster the further use of successful

approaches.

By the end of fiscal year 2000, OCS had awarded $30 million in REACH

grants to 24 states and 12 tribal organizations to fund 54 separate projects

intended to help meet the home energy (heating and cooling) needs of

low-income households in a variety of ways. These grants ranged in

amount from $50,000 to $1.6 million. Most of the 54 REACH projects have

educated low-income clients about home energy efficiency through group

workshops or in individual homes. Many REACH projects have involved

energy-related repairs to homes and budget counseling, and 3 state

REACH projects are developing consumer cooperatives to purchase

electricity or bulk fuels, such as heating oil. However, some REACH

projects have included social services not directly related to meeting home

energy needs. For example, six projects included job skill or employment

development services, and one project provided funds to help clients pay

past-due rent or mortgage payments. One of the REACH program’s criteria

for grant selection—which is used to assess whether grant proposals

address the home energy needs of low-income families within a “holistic”

context of economic, social, and other barriers to self-sufficiency—may

have been misconstrued to allow projects to spend REACH funds on nonenergy-related activities.

The legislation authorizing REACH identifies three performance goals to

be included in individual REACH projects: reduce energy costs of

participating households, increase the regularity of home energy bill

payments, and increase energy suppliers’ contributions to reduce eligible

households’ energy burden. The legislation provides broad purposes for

the program, but HHS has not developed performance goals that define

the results that it expects to achieve. Without program performance goals

that are objective and quantifiable, HHS cannot effectively evaluate the

program’s overall performance and report to the Congress on what was

Results in Brief
Page 3 GAO-01-723 REACH Program Effectiveness

accomplished for the resources expended. Such goals would also provide

a clearer basis for selecting individual projects to fund.

Of the six project evaluations completed by states as of May 2001, only one

project’s design and implementation allowed statistically valid conclusions

to be made about the effect of project services on participant energy use.

This project focused on changing the behavior of low-income households

through teaching energy-efficient practices, and its evaluation report

indicated a statistically significant decrease in natural gas use by REACH

project participants compared with those in a control group. The five

other state project evaluations had analytical problems and other

shortcomings that limited their usefulness in assessing project results.

Some evaluations did not collect sufficient data on energy use by project

participants, while others did not adjust energy use data to account for

variations in the weather. Because of these and other shortcomings, the

evaluation reports could not be used to determine whether each REACH

project met its project-level performance goals. In addition, none of the six

reports evaluated whether the project met one of the project performance

goals stated in the authorizing legislation—increasing energy suppliers’

contributions to reduce the energy burdens of eligible households. None of

the project evaluation reports attempted to compare the costs of the

services provided through REACH with the costs of providing energy

assistance payments alone, yet REACH program guidance states that a

cost-effectiveness analysis should be an objective of each state project. As

additional state projects complete their 3-year terms, HHS will receive

additional evaluation reports. Program officials have been taking steps to

improve future projects, including developing guidance and hiring a

consulting firm to advise grantees on better project design and evaluation

planning.

HHS has not yet developed a comprehensive plan for how it can best

communicate summary information on best practices and lessons learned

from the REACH program. Such information will be important in

identifying—and fostering the further use of—any methods of meeting the

heating and cooling needs of low-income families that are more effective

than current methods. As part of its communications efforts, the REACH

program is designing a Web site about REACH program requirements and

projects. In addition, REACH program officials hold annual conferences at

which grantees report on their projects’ results.

We are making recommendations to HHS to address the lack of program

performance goals for REACH and to ensure that the results of

demonstration projects are communicated to organizations involved in
Page 4 GAO-01-723 REACH Program Effectiveness

assisting low-income households with their home energy needs. Given the

limited information available at this time, the Congress may want to

consider requiring HHS to report on REACH project results and program

performance after the projects funded in the first 3 years of the program

(fiscal years 1996, 1997, and 1998) have completed their evaluations, which

will probably occur late in 2002. At that time, the Congress may also wish

to consider whether the REACH program should continue indefinitely or

whether the program should have an end date after the completion of a

sufficient number of demonstration projects.

The Congress established the REACH program to

• minimize the health and safety risks that result from high energy burdens

on low-income Americans,

• prevent homelessness as a result of inability to pay energy bills,

• increase the efficiency of energy usage by low-income families, and

• target energy assistance to individuals who are most in need.

The REACH legislation requires that project plans provide a variety of

services and benefits, which may include energy efficiency education,

residential energy demand management services, counseling related to

energy budget management and payment plans, and negotiations with

home energy suppliers on behalf of eligible households. The legislation

further requires each state’s plan to describe performance goals for its

projects and the indicators the state will use to measure whether each

project has achieved its performance goals.

OCS’ stated purpose for the REACH projects is to demonstrate the longterm cost-effectiveness of supplementing energy assistance payments with

nonmonetary benefits that can increase the ability of eligible households

to meet energy costs and achieve energy self-sufficiency. REACH is part of

a much larger program, LIHEAP, which primarily provides financial

assistance to low-income households for home heating and cooling.

REACH projects may target their services to a portion of the population

eligible for LIHEAP, such as a geographic area or type of client

(households with elderly people, for example).

Since fiscal year 1996, REACH funding has ranged from $5.5 million to

$6.8 million annually, while the total funding for LIHEAP has ranged from

$1.2 billion to $2.1 billion. Over the course of the REACH program, its

funding has averaged about one-half of 1 percent of total LIHEAP funds.

The legislation establishing REACH provided for funding it from LIHEAP’s

Background
Page 5 GAO-01-723 REACH Program Effectiveness

incentive program for leveraging nonfederal resources. This incentive

program provides additional monies to states that, in the previous year,

obtained additional assistance for low-income households’ energy needs

from such sources as state funds, utility companies, and private charities.

The REACH legislation provides that, for each fiscal year, the Secretary of

HHS may allocate up to 25 percent of the funding for this incentive

program to the REACH program, and HHS has allocated approximately

this amount each year since the REACH program was first funded in fiscal

year 1996.

Through the REACH program, OCS awards grants to states1

 and to tribes,

tribal organizations, and insular areas (generally referred to in this report

as “tribal organizations”). OCS issues annual requests for grant proposals

and then awards grants on a competitive basis. On average, about

one-third of the applicants from fiscal year 1996 through fiscal year 2000

received a REACH grant. OCS’ Division of Community Demonstration

Programs manages the REACH program.

State and tribal organization grants differ in a few ways. OCS has

established a 3-year time frame for states to complete their projects, in

part to allow time for states to contract with the community-based

organizations that carry out the states’ REACH projects. Tribal projects

have a 17-month time frame if the tribal organization carries out the

project itself, or a 3-year time frame if the tribal organization chooses to

use a community-based organization. Unlike states, tribal organizations

are not required to conduct evaluations on the effectiveness of their

project approaches, but tribal organizations do submit final reports on

their projects to OCS. In addition, OCS has established differing maximum

grant amounts for state and tribal grants. According to the director of OCS’

Division of Community Demonstration Programs, these caps on grant

amounts reflect the differing amounts that states and tribal organizations

receive in their LIHEAP allotments and resulted from focus meetings with

tribal representatives held by OCS’ LIHEAP staff prior to the first REACH

request for grant proposals.

OCS’ annual requests for grant proposals, referred to as program

announcements, provide criteria for program eligibility, proposals,

 1

The District of Columbia and Puerto Rico may apply for REACH grants under OCS’

provisions for state grants.
Page 6 GAO-01-723 REACH Program Effectiveness

selection of projects to fund, and states’ evaluations of their projects.2 Only

states and tribal organizations that receive LIHEAP grants are eligible to

participate in REACH, and the households that may receive REACH

project services are those eligible for LIHEAP assistance. In its program

announcements for fiscal years 1996 through 2000, OCS required states’

and tribal organizations’ proposals for REACH grants to address the

following elements:3

• the organizational experience and capability of the community-based or

tribal organization(s) conducting the project;

• project staff skills and responsibilities;

• state-level management and organization of the project;

• project theory and design, including the target population and needs to be

addressed, activities, expected outcomes, goals, and work plan;

• budget appropriateness and justification;

• expected beneficial impact;

• “holistic” strategies addressing the economic and social barriers to selfsufficiency, and project innovations (required for state proposals only);

• community empowerment of areas characterized by severe poverty, high

unemployment, or other indicators of socioeconomic distress (required for

state proposals only); and

• evaluation of the project.

OCS uses the above proposal elements in its criteria for selecting which

projects to fund. OCS convenes panels to review and score the grant

proposals against the criteria; state and tribal organization proposals are

assessed separately. Factors in addition to the scoring may be used in

making selections, such as geographic distribution and OCS’ past

experiences with the applicants.

States’ evaluations of their projects are to address how effectively each

project was implemented, and whether and why the expected results and

goals were or were not achieved. OCS requires states to use third-party

evaluators to conduct the evaluations. Such evaluators are individuals or

 2

Consistent with a preference stated in the congressional committee report on the

authorizing legislation, OCS has not developed and published regulations for the REACH

program.

3

OCS made some changes in its fiscal year 2001 program announcement, including

regrouping some of the elements listed and requiring tribal organization applicants, as well

as state applicants, to address innovative approaches.
Page 7 GAO-01-723 REACH Program Effectiveness

firms that are organizationally distinct from the state agency or

community-based organizations involved in the REACH project.

Through the end of fiscal year 2000, OCS had awarded $30 million in

REACH grants to states and tribal organizations for 54 projects4

 (29 state

and 25 tribal projects) to address the home heating and cooling needs of

low-income households using a variety of approaches. The most

commonly used approaches have been energy efficiency education, home

energy audits, home weatherization, and budget counseling, while

innovative approaches have included forming consumer cooperatives for

energy purchasing and using solar and/or wind power. Several projects

have included activities not directly related to home energy—for example,

job skill or employment development services and financial assistance

toward overdue rent or mortgage payments. After its REACH grant period

is over, a state or tribal organization may wish to continue using the

approaches tested in the REACH project. Some activities could be

replicated in a state’s or tribal organization’s LIHEAP program, if the state

or tribal organization chose to do so.

OCS has awarded REACH grants for 29 state and 25 tribal projects. Table 1

below shows REACH grant funding and numbers of grants by year and

type of grant (state or tribal organization). In addition to the grant funding

shown in the table, HHS has provided a total of $1.1 million to states for

their costs in administering the grants, overseeing the community-based

organizations that carry out REACH projects, and contracting for the thirdparty evaluations. (Tribal organizations have not received such funding

because, to date, the tribal organizations have chosen to carry out their

REACH projects themselves and are not required to contract for thirdparty evaluations.) Including both the grant funding and administrative

funding for states, HHS provided $31 million in REACH program funds in

fiscal years 1996 through 2000.

 4

The REACH program considers the fiscal year 1996 award to Michigan as three separate

grants for three separate projects and therefore considers the total number of grants to be

56. Because Michigan addressed the projects in one evaluation report, we have counted

this as one grant award.

Demonstration

Projects Have Tried

Various Approaches

and Included Some

Activities Not Directly

Related to Home

Energy

Grant Recipients and

Amounts
Page 8 GAO-01-723 REACH Program Effectiveness

Table 1: REACH Grant Funding by Fiscal Year and Type of Grant

Dollars in thousands

State grants

Tribal organization

grants All grants

Fiscal year Funding Number Funding Number Funding Number

1996 $5,102 6 $379 4 $5,482 10

1997 5,658 6 362 3 6,020 9

1998 5,201 7 436 4 5,638 11

1999 5,558 5 805 7 6,363 12

2000 5,482 5 969 7 6,451 12

Total $27,002 29 $2,951 25 $29,953 54

Note: The funding figures above do not include the administrative funding for states.

Source: GAO compilation of HHS data.

Tables 2 and 3 below list the grant awards made to states and tribal

organizations, respectively, in fiscal years 1996 through 2000. Four

states—Michigan, Nebraska, Oregon, and Pennsylvania—have received

more than one grant, for separate projects. Similarly, five tribal

organizations and one insular area have received more than one grant.

State grants ranged in amount from $166,667 to $1.6 million and averaged

$931,108. Tribal organization grants ranged in amount from $50,000 to

$199,276 and averaged $118,051.

Table 2: Grant Awards to States, by Fiscal Year

State Grant amount

1996

California $1,492,500

Maryland 169,178

Massachusetts 479,611

Michigan 761,111

Nebraska 600,000

Oregon 1,600,000

1997

Arkansas 210,000

Indiana 1,501,840

Kentucky 266,002

New York 1,600,000

Pennsylvania 1,200,000

Vermont 880,497
Page 9 GAO-01-723 REACH Program Effectiveness

State Grant amount

1998

Alaska 1,328,723

Arizona 1,000,000

Illinois 166,667

Iowa 372,391

Maine 1,562,050

Michigan 337,437

Utah 433,853

1999

Connecticut 1,499,708

Michigan 341,635

Nebraska 1,136,042

North Carolina 1,000,000

Oregon 1,580,600

2000

Alabama 1,000,000

Montana 1,500,000

Nevada 1,100,000

Pennsylvania 882,277

Washington 1,000,000

Total $27,002,122

Source: GAO compilation of HHS data.

Table 3: Grant Awards to Tribes, Tribal Organizations, and Insular Areas, by Fiscal Year

Tribe, Tribal Organization, or Insular Area Location Grant amount

1996

Central Council of the Tlingit and Haida Alaska $129,234

Grand Traverse Band of Ottawa and Chippewa Indians Michigan 50,103

Sault Ste. Marie Tribe of Chippewa Indians Michigan 150,000

South Puget Intertribal Planning Agency Washington 50,000

1997

Cherokee Nation Oklahoma 199,276

Grand Traverse Band of Ottawa and Chippewa Indians Michigan 50,103

South Puget Intertribal Planning Agency Washington 112,134

1998

Central Council of the Tlingit and Haida Alaska 175,000

Grand Traverse Band of Ottawa and Chippewa Indians Michigan 70,738

Lumbee Regional Development Association North Carolina 140,711

United Tribes of Kansas and Southeast Nebraska Kansas and Nebraska 50,000
Page 10 GAO-01-723 REACH Program Effectiveness

Tribe, Tribal Organization, or Insular Area Location Grant amount

1999

American Samoa Government Insular area in the Pacific 123,866

Central Council of the Tlingit and Haida Alaska 175,000

Fort Belknap Indian Community Montana 70,819

Grand Traverse Band of Ottawa and Chippewa Indians Michigan 65,109

Lumbee Regional Development Association North Carolina 149,995

Northern Cheyenne Tribe Montana 150,000

United Tribes of Kansas and Southeast Nebraska Kansas and Nebraska 70,000

2000

American Samoa Government Insular area in the Pacific 124,304

Blackfeet Tribe Montana 150,000

Central Council of the Tlingit and Haida Alaska 175,000

Grand Traverse Band of Ottawa and Chippewa Indians Michigan 149,895

Little River Band of Ottawa Indians Michigan 150,000

Lumbee Regional Development Association North Carolina 150,000

United Tribes of Kansas and Southeast Nebraska Kansas and Nebraska 70,000

Total $2,951,287

Source: GAO compilation of HHS data.

The activities conducted under the demonstration grants have varied.

Almost all of the REACH projects included multiple activities and

addressed more than one of the following areas:

• reducing energy use for home heating and cooling by increasing efficiency;

• helping clients pay for energy bills through budget planning, consumer

education, consumer cooperatives, and other methods;

• reducing the use of other utilities (water and electric lighting); and

• providing social services not directly related to home heating and cooling

needs.

Appendix I provides further details about the types of activities included in

REACH project plans and the number of projects conducting each type of

activity.

The most common project activity has been energy efficiency education,

which was included in 45 of 53 projects.5

 This education has been provided

through group workshops for the projects’ clients or through in-home

counseling of individual households. Some grantees have also developed

 5

One of the 54 projects, North Carolina’s, is not included in our description of grant

activities because it was being redesigned at the time of our review.

Project Activities
Page 11 GAO-01-723 REACH Program Effectiveness

energy educational materials, such as pamphlets or videos, some of which

were designed for children. Other activities aimed at reducing energy use

have included home energy audits and weatherization. Home energy

audits were used by 25 REACH projects, often in conjunction with

weatherization of the home or education in how to reduce energy use. In a

home energy audit, a detailed inspection of the home is carried out to

identify repair and weatherization needs and practices of the household

that result in inefficient energy use. For example, doors and windows are

inspected for drafts, attics for insulation needs, and furnaces for

maintenance needs. Families may be advised of energy-saving practices

such as turning down their thermostats at night or using fans to reduce the

need for air conditioning.

Twenty REACH projects have provided home weatherization work by

construction contractors. While the Department of Energy’s (DOE)

Weatherization Assistance Program helps many low-income families, some

homes need repairs that exceed the per-house dollar limit of the DOE

program. The REACH projects that included weatherization typically went

beyond the services provided by the DOE program or served clients who

had not been addressed by the DOE program. Some REACH projects

worked in conjunction with DOE’s program. In addition, 20 REACH

projects provided clients with do-it-yourself weatherization kits and

training in how to install the weatherization measures. The kits included

items such as caulk, weather-stripping, and plastic coating for windows.

Budget counseling was provided in 30 of the REACH grant projects. Such

counseling helps clients to plan ahead for paying for their essential needs,

including energy bills; to identify areas where they could cut costs; and to

become better informed about credit practices. Budget planning can help

low-income households avoid utility cutoffs due to unpaid bills, which are

followed by the need to pay reconnection fees and past-due bills. Utility

cutoffs are not uncommon for low-income households. According to OCS

data, during the 1992-93 heating season, 1 million LIHEAP-eligible

households (3.3 percent of such households) reported that they were

unable to use their main source of heat for 2 hours or more because they

were unable to pay for their main heating fuel.6

 Sixteen REACH projects

also negotiated with energy vendors on behalf of their clients to obtain

 6

These are the most recent data available; see Low Income Home Energy Assistance

Program: Report to Congress for Fiscal Year 1996, U.S. Department of Health and Human

Services, Office of Community Services, Division of Energy Assistance, Sept. 2000.
Page 12 GAO-01-723 REACH Program Effectiveness

payment plans or forgiveness of past-due bills, and four projects provided

funds to help pay past-due bills and/or reconnection fees. Seven projects

provided consumer education on utility deregulation and the consumer

choices provided by deregulation.

Some REACH projects addressed the use of utilities for purposes other

than heating and cooling. Thirteen projects provided energy-efficient light

bulbs. Energy-efficient lighting was the only focus of the two grants

received by American Samoa because the Samoan government determined

that this was the one measure most likely to reduce the high cost of

electricity for their low-income households. Six projects addressed water

conservation in the home by providing devices such as low-flow

showerheads or repairing plumbing. Such plumbing measures also reduce

energy usage for hot water heaters.

Some REACH projects included social services not directly related to

home energy needs. Seventeen of the REACH projects provided social

services through case management. For example, Nebraska’s fiscal year

1996 grant project included caseworker visits to help families identify and

address their concerns in areas such as improving job prospects, eating

nutritional food, and obtaining health services. Only Indiana’s project

provided funds to help clients pay past-due rent or mortgage payments.

Six projects included job skill or employment development services. For

example, Indiana’s fiscal year 1997 grant included the use of REACH funds

for job skill training, transportation to work, and day care; $258,000 of

Indiana’s planned project budget addressed such services to help clients

move from welfare to work. In many states, services that support the

transition of welfare recipients to the workforce—such as job training,

transportation, and child care—are provided by the much larger

Temporary Assistance to Needy Families program ($22.6 billion in federal

and state funding in fiscal year 1999).

The REACH program’s criteria regarding “holistic approaches” may have

been misconstrued to allow projects to fund social services that are not

directly related to home energy needs. Specifically, the REACH program

announcement for fiscal year 2000 stated:

“OCS is interested in having Applicants approach the energy needs of low-income families

within a holistic context of the economic, social, physical, and environmental barriers to

self-sufficiency. Thus applicants should include in their REACH Plan an explanation of how

the proposed projects(s) will be integrated with and support other anti-poverty or

development strategies within the target community or communities.... Thus REACH

initiatives are expected to be closely coordinated with other public and private sector
Page 13 GAO-01-723 REACH Program Effectiveness

programs involved with community revitalization, housing rehabilitation and

weatherization, and family development.”

Similar language has been included in each program announcement

requesting REACH grant proposals. Furthermore, the criteria used to

select grant proposals to fund have included holistic program strategies

and project innovations. REACH grant proposals are scored on a number

of criteria—such as the organization’s experience and the project strategy

and design—and each criterion has a maximum number of possible points.

Out of a total of 100 possible points, state proposals for fiscal years 1996

through 2000 could receive up to 10 points in the selection process for the

criterion “holistic program strategies and project innovations.” When

asked about this criterion and the non-energy-related activities, REACH

program officials said that the criterion might need to be clarified and that

they also had some concerns about the emphasis on nonenergy activities

in a few projects. However, they also noted that the legislation sets broad

purposes for REACH, including preventing homelessness and minimizing

health and safety risks resulting from high energy burdens on low-income

households. Because energy burden is defined as home energy

expenditures divided by household income, the officials stated that some

grantees had chosen to address increasing clients’ incomes.

Some REACH projects have involved innovative activities. Examples of

innovative activities include the installation of solar and/or wind power for

low-income households in two tribal projects: the Grand Traverse Band of

Ottawa and Chippewa Indians (fiscal year 2000 grant) and the Cherokee

Nation (fiscal year 1997 grant). Another innovative activity used in three

state projects was the formation of consumer cooperatives to reduce the

cost of energy to low-income households. Vermont’s fiscal year 1997

project formed a cooperative and purchased a home heating oil company.

The other two state projects—New York’s and Connecticut’s—are forming

cooperatives to purchase energy from deregulated utilities, allowing lowincome households to aggregate their purchasing power and negotiate

lower prices. Pennsylvania’s fiscal year 2000 project is addressing cooling

needs in urban areas, following a number of heat-related deaths among

elderly residents, particularly in urban row houses. The project is both

introducing an innovative use of heat-reflective coatings on roofs of inner

city row houses and providing fans and safety devices that permit

windows to be locked in both open and closed positions.

REACH grantees have also developed innovative ways of working with

other organizations and delivering services. The project of the Central
Page 14 GAO-01-723 REACH Program Effectiveness

Council of the Tlingit and Haida tribes in Alaska trained AmeriCorps

personnel7

 to provide energy efficiency education. Because these

personnel remain in the Alaskan villages, the project proposal stated that

the impact of the services would continue beyond the completion of the

grant period. Nevada’s project for its fiscal year 2000 grant is using the

concept of individual development accounts to provide clients with

matching funds for their savings. The REACH clients’ savings accounts are

to be used for replacing the furnace or appliances, changing to a lower

cost fuel, or paying higher winter utility bills. Kentucky’s project used

volunteers from local organizations to install weatherization kit materials

in the homes of disabled and elderly clients who could not do the work

themselves.

After REACH projects are completed, some project activities may be

replicated in a state’s or tribal organization’s LIHEAP program, if the state

or tribal organization chooses to do so. (The state or tribal organization

may also choose to use its own funding to continue the activities.) The

legislation authorizing LIHEAP allows states to spend up to 5 percent of

their allotted LIHEAP funds for services that encourage and enable

households to reduce their home energy needs and therefore their need

for energy assistance. According to LIHEAP program officials, such

services may include energy efficiency education, budget counseling, and

assistance in obtaining discounts or payment plans from energy vendors.

According to states’ fiscal year 2001 plans, 25 of 50 states and the District

of Columbia planned to use a portion of their LIHEAP funds for energy use

reduction services, and their planned services included energy education,

energy needs assessment, liaison with energy vendors, budget counseling,

and case worker services for clients. The legislation also allows states to

spend up to 15 percent of their allotted LIHEAP funds for low-cost

weatherization or other energy-related home repair for low-income

households. States may also apply to HHS for waivers to use up to 25

percent of their allotment for this purpose. Forty-four of 50 states and the

District of Columbia planned to provide weatherization in fiscal year 2001.

Several types of activities used in REACH projects may not be covered by

these provisions of LIHEAP, such as forming or expanding energy

consumer cooperatives, installing solar and wind power units, providing

 7

AmeriCorps, known as the domestic Peace Corps, engages more than 40,000 Americans a

year in community service.

Replication of REACH

Project Activities
Page 15 GAO-01-723 REACH Program Effectiveness

efficient light bulbs, and providing matching funds for clients’ savings to

be used for energy purposes. It is uncertain whether states may use

LIHEAP funds if they wish to continue or expand such activities after the

conclusion of their REACH project. LIHEAP officials noted that because

LIHEAP is a block grant program, states decide how to design their

programs and interpret the statutory provisions regarding energy use

reduction services and weatherization. Furthermore, LIHEAP’s funding

caps in these areas, as well as the need among low-income households for

direct assistance with energy bills, would determine the extent to which

the activities tested in a REACH project could be provided more broadly

to a state’s or tribal organization’s LIHEAP clients.

The legislation authorizing REACH identifies performance goals to be used

by individual REACH projects. For the program as a whole, however, HHS

has not developed performance goals and measurable indicators that

define the results that it expects to achieve. Furthermore, HHS’

performance plans do not address how the REACH program relates to the

larger LIHEAP program of which it is a part.

The legislation authorizing the REACH program requires that REACH

project plans describe performance goals for each project, which are to

include

• a reduction in the energy costs of participating households over 1 or more

fiscal years,

• an increase in the regularity of home energy bill payments by eligible

households, and

• an increase in energy vendor (such as utility companies) contributions

towards reducing energy burdens of eligible households.8

The legislation further requires that project plans include a description of

the indicators that each state will use to measure whether the

performance goals have been achieved. For example, in addressing the

performance goal of more regular energy bill payments, Oregon’s proposal

for its fiscal year 1999 grant specified the following indicator: 50 percent of

households with past-due energy bills will reduce their arrearages by

 8

The REACH grant proposals did not always address all three of the performance goals,

and REACH program officials indicated that their understanding of the legislation was that

each state grantee was not necessarily required to address all of them.

REACH Program

Lacks Performance

Goals and Measurable

Indicators
Page 16 GAO-01-723 REACH Program Effectiveness

40 percent after 12 months of participation. In addition, OCS requires

states to address how results will be measured in their plans for evaluating

their projects.

Despite the program’s use of project-level performance goals and

indicators, the REACH program as a whole lacks performance goals and

measurable indicators. According to a REACH official, the Administration

for Children and Families decided not to address the REACH program in

its performance plan, which was developed to meet the requirements of

the Government Performance and Results Act of 1993. While agencies’

performance plans do not have to include the complete array of goals and

measures used in managing individual programs, the development and use

of performance goals and indicators can be beneficial to any federal

program. Specifically, performance goals and measurable indicators can

• help a program to focus its efforts on achieving results and on those

activities most closely linked to program goals,

• provide a clearer basis for selecting projects to fund,

• provide a basis for determining how well the program is performing and

what has been achieved in return for the resources invested in it, and

• facilitate reporting to the Congress and the public on the program’s

performance.

For the REACH program in particular, performance goals and measurable

indicators could address two needs: They could both provide a basis for

grant selection and enhance effectiveness in carrying out federal roles.

First, as described above, OCS has selected several grant proposals that

used a substantial portion of their grant funding for activities that were not

directly related to home energy needs, such as job skill or employment

development services. The use of performance goals could help OCS to

better target grant selection to projects that are closely linked to program

goals. Second, the federal roles in the REACH program are not fully

addressed by the project-level goals. For example, the federal role in the

REACH program currently includes selecting grantees, providing program

guidance, helping to strengthen project design and states’ evaluations of

their projects, and providing information to grantees and others involved

in providing energy assistance. The effectiveness of federal efforts in the

program could be enhanced by results-oriented performance measures

addressing these roles.

Furthermore, the performance plan for the Administration for Children

and Families (which includes OCS) does not address the relationship

between REACH and LIHEAP, or between REACH and other related
Page 17 GAO-01-723 REACH Program Effectiveness

federal programs, such as DOE’s Weatherization Assistance Program.

Performance plans can be useful tools for identifying the need for

coordination among programs, and for ensuring that the goals of related

programs are congruent and that crosscutting efforts are mutually

reinforcing. In the case of REACH, HHS could use this performance plan

to articulate REACH’s purpose in demonstrating and providing incentives

for LIHEAP grantees to try new approaches. Without HHS’ addressing the

relationship between REACH and LIHEAP, in particular, it is unclear

whether HHS expects the REACH program to have a broader impact on

the activities of LIHEAP grantees, beyond a particular REACH grant

project and its limited time frame.

The six states that received funding in fiscal year 1996 have prepared

evaluation reports: California, Maryland, Massachusetts, Michigan,

Nebraska, and Oregon. However, only Nebraska’s evaluation report, with

some qualification, fairly reflects the REACH project’s effects on energy

use. The remaining five evaluation reports have substantial design and

implementation shortcomings that compromise the validity of the reports’

findings. In addition, all six evaluation reports have other shortcomings

that preclude an overall assessment of the projects’ effectiveness.

(Appendixes II through VII summarize each state’s evaluation report,

including its goals and measures, project assumptions, evaluation design,

services provided, participant and control group selection, evaluation

findings, reported limitations, and GAO’s observations on analytical

problems and other shortcomings.) OCS is aware of the shortcomings in

these initial evaluations and has been taking several steps to improve

future project evaluations.

The design and implementation of Nebraska’s fiscal year 1996 project

allowed statistically valid conclusions to be made in the state’s evaluation

report about the effects of project services on participant energy use. The

project, which was carried out by a community-based organization,

focused on changing the behavior of low-income households to achieve

economic self-sufficiency through decreased energy use. This approach

was based on the assumption that most of the target population were

renters who frequently moved and that the housing available to them was

generally substandard. It was assumed that the only way to reduce energy

use was to change household behavior through teaching energy-efficient

practices.

Initial Project

Evaluations Have

Limitations, but the

Program Is Taking

Action to Improve

Future Evaluations

Nebraska’s Evaluation

Report Fairly Reflected

Results
Page 18 GAO-01-723 REACH Program Effectiveness

The evaluation report noted that REACH participants significantly

decreased their use of natural gas from preproject to postproject

compared with a control group of nonparticipants, whose consumption

showed no statistical change. The outcome for electricity use was that

while REACH participants showed, on average, no change in electricity

use, nonparticipants increased electricity use significantly over the course

of the project.

The evaluation report attributed the project’s successful implementation

to the project design and the collection of sufficient data to the

community-based organization’s experience, expertise, and knowledge of

the target population, as well as to the early involvement of the evaluator

in the project’s design. Cash incentives provided to the test and control

groups also appeared to help limit participant attrition (leaving the project

before its completion). In addition, administrative controls designed to

ensure access to utility bills seem to have played a role in obtaining

sufficient data for analysis.

The other five states’ REACH projects experienced design and

implementation constraints that decreased the confidence that can be

placed on the findings contained in their evaluation reports. For example,

Massachusetts and Michigan did not use control groups as a means of

assessing whether project participants fared differently from similar

households that did not receive project benefits. California and Maryland

experienced difficulties in collecting complete income or utility data and

experienced client attrition at rates that call into question the likelihood

that project effects could be assessed.

To some extent, these state REACH projects faced common challenges:

(1) forming an adequate control/comparison group against which to

compare project outcomes; (2) maintaining client participation in project

activities; (3) collecting complete, accurate, and reliable data (such as

client income and utility bills); and (4) adjusting (“normalizing”) energy

consumption data for changes in weather.

• A properly formed control group, composed of people who did not receive

REACH project services but who are similar in other respects, allows a

comparison of what might have happened in the absence of the project.

Project designs that lack a control group can demonstrate that changes

occur in such factors as energy use, but the changes cannot be attributed

to the project since other factors might be responsible for the changes.

Any preexisting differences between a test group and a control group

Other Projects Faced

Evaluation Challenges
Page 19 GAO-01-723 REACH Program Effectiveness

could also be the cause of any observed project effects and, therefore,

should be addressed in the project’s design.

• Attrition of participants can reduce the amount of data available for

analysis and introduce biased results. Attrition bias can occur if those

remaining in the project systematically differ from those who dropped out

in ways that are likely to affect the outcome of the project.

• Lack of complete, reliable, and accurate data will result in imprecise or

biased results. Data that are not collected for an entire heating or cooling

season, for example, can lead to faulty assessments of typical energy use.

• Failure to normalize energy use data makes it difficult to determine the

extent to which changes in energy use result from project activities or

from changes in weather that affect the need to heat or cool a residence.

For example, if the postproject data on energy use reflect a warmer winter

than the preproject data, a valid comparison of energy use for these two

periods should adjust the data to help determine if perceived energy

savings were due to warmer winter temperatures or to project activities.

In addition, other shortcomings precluded an overall assessment of the

projects’ effectiveness. None of the evaluation reports addressed one of

the three project-level performance goals stated in REACH legislation and

program announcements: increasing energy suppliers’ contributions to

reduce the energy burdens of eligible households. In addition, although the

REACH program announcements have stated that an objective of every

state project plan should be to measure whether its activities are more

cost-effective in the long term than energy assistance payments alone,

none of the project evaluation reports provided such an analysis. Finally,

most of the evaluation reports did not report lessons learned or best

practices that could be valuable to other projects. For example, if the

report had discussed the strategies used to successfully collect energy use

data, this information could have been useful to other projects. This lack

of data was often cited in evaluation reports as a principal reason for not

being able to measure project effectiveness.

The REACH program is aware of the challenges noted above and has been

taking several steps to improve future project evaluations, including

providing guidance and technical assistance. First, OCS has developed

guidance for project planning and evaluation in its demonstration

programs, which include the REACH program. According to REACH

officials, this guidance was first given to REACH grantees in 1997 or 1998.

While this was too late to help the fiscal year 1996 grantees plan their

evaluations and related data needs, the guidance addresses many of the

evaluation limitations noted above and, if followed by state grantees,

REACH Program Is Taking

Steps to Improve

Evaluations
Page 20 GAO-01-723 REACH Program Effectiveness

could help to improve future evaluations. For example, the guidance

discusses

• selecting an evaluator—indicators of a good evaluator and questions to

ask in an interview;

• using a logic model (described below) to design and evaluate a project;

• addressing challenges to gathering data on project participants, such as

their reluctance to provide income data;

• using comparison groups that are as similar as possible and overrecruiting for comparison groups to allow for attrition; and

• developing evaluation reports that are clear and complete.

The guidance was revised in 2000 to increase its emphasis on the logic

model, according to REACH program officials. A logic model identifies

underlying assumptions of the project, project activities (such as energy

efficiency education), immediate and intermediate outcomes expected

from the activities (such as improved understanding of behaviors that can

affect energy use in the home), and final project goals (such as a reduction

in energy use). The REACH program encourages grant applicants to use a

logic model in their applications and project planning.

Second, the REACH program provides technical assistance to grantees in

their project and evaluation planning. States that receive grants are

required to submit evaluation plans, which are reviewed by REACH

program officials and the consulting firm that is providing technical

assistance to the program. The REACH officials and consultant then

discuss with the project team any improvements that are needed, and the

grantee submits a revised evaluation plan. Once an acceptable evaluation

plan is completed, the REACH program sends a letter to indicate approval

of the plan. According to officials of the REACH program’s consulting

firm, they emphasize improving the logic model to strengthen the projects’

designs and their ability to measure whether expected outcomes and goals

are achieved by project activities. REACH officials also review draft

evaluation reports and may ask states to revise and improve them.

Third, guidance and assistance with evaluations have also been provided

through program conferences. The REACH program held conferences for

evaluators in July 2000 and July 2001. At the July 2000 evaluators’

conference, the participants discussed logic models, common issues in

gathering and analyzing data, evaluation methods, and lessons learned.

Developing quality evaluations has also been discussed at the annual

REACH program conferences, and evaluators as well as representatives of

states, tribal organizations, and community-based organizations may
Page 21 GAO-01-723 REACH Program Effectiveness

participate in the conference. For example, the REACH conference held in

January 2001 included a presentation and discussion on developing logic

models, indicators, and evaluation plans. In addition, as the state grantees

reported on their completed fiscal year 1997 projects, the conference

participants discussed many specific evaluation issues.

While the REACH program and its consultants can assist grantees in

planning their evaluations, conducting the evaluation is a responsibility

shared among the states, the community-based organizations with whom

the states contract, and third-party evaluators. State officials monitor and

oversee the projects and the evaluation reports. Staff of the communitybased organizations are responsible for collecting the data needed to

complete the evaluation. The staff of the community-based organizations

may need training to understand the importance of data gathering and to

carry out this task effectively. Third-party evaluators help to design the

evaluation, analyze data, and prepare the evaluation report. Some of the

problems we noted in the six completed evaluations related to reporting,

such as not clearly explaining methods of analysis or data limitations. For

future REACH evaluations to improve, the states, community-based

organizations, and evaluators—as well as the REACH program and its

technical assistance consultants—must carry out their functions well.

OCS has not yet planned how it can best communicate information to state

officials and others about the results of REACH projects, such as what

approaches prove to be the most successful in meeting the home energy

needs of low-income households. As the REACH program proceeds and

state grantees complete additional project evaluations, more information

will become available about which approaches demonstrated in REACH

projects were successful and which were less successful. In addition to

evaluating the results of their projects, grantees report on the processes

and procedures they used to carry them out. Completed REACH projects

will provide information and tools such as

• pamphlets, videos, and other materials for energy efficiency education;

• forms for collecting data from clients; and

• experiences and lessons learned in such areas as how to leverage

contributions from energy vendors and how to coordinate among various

organizations involved in energy assistance.

A comprehensive communications plan would help ensure that this

information is put to good use by identifying what information to

communicate, to what audiences, and by what methods. A

HHS Has Not Yet

Planned How to

Communicate REACH

Project Successes and

Lessons Learned
Page 22 GAO-01-723 REACH Program Effectiveness

communications plan would also estimate the amount of funding needed

for communications. REACH program officials realize that publications

such as summaries of successful approaches could provide information in

a form more readily accessible than the individual evaluation reports.

Other OCS programs have developed summary publications of best

practices and have found them to be frequently requested, according to

the director of OCS’ Division of Community Demonstration Programs. For

instance, one publication provided lessons learned from 8 years of OCS

demonstration programs (not including the REACH program, which had

not yet begun at the time). As the REACH program matures and more

information becomes available, OCS will have a better basis for identifying

and summarizing best practices. For instance, information will become

available from six project evaluations that are due during the fall of 2001

(from the state grants awarded in fiscal year 1997), and by the end of 2002,

a total of 19 state projects will probably have completed their evaluations.

To date, the REACH program’s communications efforts have included

developing a Web site and providing information through conferences.

According to program officials, the REACH Web site, located within OCS’

Web site for its demonstration programs, is expected to become available

in the summer of 2001. Currently, a pilot version of the REACH Web site is

available at the Web site of the REACH program’s consulting firm.

According to REACH program officials, the OCS Web site will provide the

same types of information as the pilot Web site. The pilot REACH Web site

includes

• the current OCS program announcement requesting proposals for REACH

grants,

• summaries of past and ongoing REACH projects,

• summaries of the REACH program conferences, and

• listings of contact points for grants and evaluations.

Conferences are also used to communicate project results and lessons

learned. The REACH program hosts an annual conference for certain

people involved with REACH projects. According to the REACH program

announcements, state grantees are expected to fund travel by the state

project directors, community-based organization project directors, and

chief evaluators to the annual conference in each of the 3 years of the

project. Tribal organizations, which have a shorter project term, are to

fund travel to one conference. At the conferences, grantees that are

finishing their projects make presentations about their project approaches

and results. REACH program officials also provide information geared to

new grantees about program and reporting requirements. The conference
Page 23 GAO-01-723 REACH Program Effectiveness

has also included topical presentations; for example, the conference in

January 2001 included presentations on the characteristics of good project

evaluation methods, solar power, and designs for low-energy-usage homes.

According to REACH officials, they and their consulting firm have also

arranged for presentations on REACH projects at the annual LIHEAP

conference and other energy-related conferences.

Grantees also have responsibilities for communicating about their REACH

projects. OCS requires that states’ REACH project plans address

disseminating results of the individual projects among LIHEAP grantees,

utility companies, and others interested in increasing the self-sufficiency

of the poor. States are allowed to budget up to $5,000 of each grant for

dissemination purposes. Tribal organizations are allowed to budget up to

$1,000 for dissemination purposes. Such state and tribal efforts are

important, but they address only individual projects, not the broader

compilation of learning from a number of projects over a period of years.

Because the REACH program lacks performance goals and measurable

indicators, HHS has not defined the relationship between REACH and its

parent LIHEAP program and cannot assess the program’s overall

effectiveness. Considering the recent rise in home heating and cooling

costs, the REACH program’s role in testing approaches to help low-income

families to meet their home energy needs is an important one and should

be clearly articulated. We believe that the development of performance

goals and measurable indicators could provide the Congress with better

information about what has been accomplished for the resources

expended. Furthermore, performance goals could provide HHS’ Office of

Community Services with a clearer basis for selecting grant proposals to

fund. In addition, by addressing the relationship between LIHEAP and

REACH in its performance plan, HHS’ Administration for Children and

Families could clarify the role of the REACH program and whether it

expects REACH to have an effect on the activities of LIHEAP grantees,

beyond a particular REACH grant and its limited time frame.

Some of the REACH projects funded to date have included activities other

than addressing clients’ home heating, home cooling, and energy payment

needs, such as job skill or employment development services. The REACH

program’s requests for proposals with “holistic approaches” may have

been misconstrued by grant applicants and by those reviewing proposals

to allow projects to use REACH funds for non-energy-related activities.

However, with only about $6 million in funding annually, we question

whether non-energy-related activities—including some activities typically

Conclusions
Page 24 GAO-01-723 REACH Program Effectiveness

addressed through other, much larger social service programs—are an

effective use of limited REACH program funds.

While the evaluations conducted on the first year of state grants have

many shortcomings that limit their usefulness in assessing project

effectiveness, the REACH program recognizes the problems and has been

taking steps to help improve future project evaluations. It is too soon to

gauge the effectiveness of these efforts, but we noted some improvements

in states’ plans for the next set of evaluations. We encourage the REACH

program to continue its efforts to improve the design and methodology of

evaluations because valid evaluations are vital to realizing the potential of

the REACH program in testing new approaches.

With only six project evaluations completed, there is currently not enough

information to reach a conclusion about the effectiveness of the REACH

program. By the end of 2002, a total of 19 state projects will probably have

completed their evaluations, and OCS will have awarded a total of about

80 grants, roughly half of them to states, which will eventually report on

project results. While the legislation authorizing the REACH program

requires this GAO review, it does not require HHS to report to the

Congress as more information on project results becomes available.

Developing program performance goals and measurable indicators and

obtaining better data on the REACH program’s effectiveness would enable

HHS to provide the Congress the information that it needs to assess the

program. Furthermore, the legislation did not specify an ending date for

the REACH program, as is sometimes the case for demonstration

programs. It may be appropriate to reassess the REACH program during

2003 and consider whether and how long it should continue.

Finally, the lack of a comprehensive plan for communicating the results of

REACH projects and fostering the further use of effective approaches

could limit the impact of the REACH program. Without well-planned and

adequately funded communications, the results of REACH projects may

fail to have an impact on LIHEAP and other programs that provide energy

assistance to low-income families. Summaries reporting on best practices

could be more effective as communication tools than the state evaluation

reports themselves for reaching the state, federal, and community officials

involved in addressing the home energy needs of low-income households.

To the extent that OCS shares such information among the state, tribal,

and community-based organizations that provide energy assistance, the

organizations can make better-informed decisions about replicating

successful approaches and avoiding problematic ones. Furthermore, wellinformed organizations can avoid “reinventing the wheel” by, for instance,
Page 25 GAO-01-723 REACH Program Effectiveness

not investing in developing energy education materials that may already be

available. In light of additional project evaluations and other information

that will become available over the next several years, OCS needs to plan

for its communications efforts.

To better target the use of the limited resources of the REACH program

and provide for reporting on program performance, we recommend that

the Secretary of HHS direct the Administration for Children and Families

and its Office of Community Services to

• develop program performance goals for REACH that are objective,

measurable, and quantifiable;

• address the relationship between the REACH and LIHEAP programs in its

performance plan; and

• ensure that REACH funds are used for activities directly related to the

home energy (heating and cooling) needs of low-income households.

To ensure that the results of REACH projects are effectively

communicated to the government agencies and private organizations

involved in addressing low-income households’ energy needs, we

recommend that the Secretary direct the Office of Community Services to

develop a communication plan for the REACH program describing

intended audiences, types of information to be communicated,

communication methods appropriate to the intended audiences, and the

funding needed.

With HHS developing information for performance reporting and obtaining

additional project evaluations, the Congress may want to consider

requiring HHS to report on REACH program effectiveness and project

results in several years, after the projects funded in the first 3 years of the

program have completed their evaluations by the end of 2002. Once HHS

has reported, the Congress may also wish to consider whether the REACH

program should continue indefinitely or whether the program should have

an end date after a sufficient number of demonstration projects.

The Department of Health and Human Services provided written

comments on a draft of this report. These comments are reprinted in

appendix VIII, along with our responses. HHS generally agreed with our

recommendations.

Recommendations for

Executive Action

Matters for

Congressional

Consideration

Agency Comments

and Our Response
Page 26 GAO-01-723 REACH Program Effectiveness

However, while HHS agreed with our third recommendation that it ensure

that REACH funds are used for activities directly related to the home

energy needs of low-income households, it made two comments related to

this recommendation. First, HHS disagreed with our assessment that

language in its program announcements on holistic program strategies

may have been misconstrued to encourage non-energy-related activities.

We continue to believe that HHS should review the language of its

program announcement, as it plans to do, to ensure that the REACH

program does not fund activities that are not directly related to the home

energy needs of low-income households. The language in the program

announcement is used as criteria for reviewing and selecting grant

proposals, as well as for providing guidance to applicants for grants. Our

recommendation concerns only REACH program funds; grantees would

not be precluded from using other sources of funds for such activities or

from coordinating with other social service efforts. Second, HHS stated

that the term “residential energy” is understood to include all household

energy use, not just home heating and cooling. However, we note that the

authorizing legislation does not define the term “residential energy.”

Therefore, we believe that HHS should apply the definition of home energy

in the authorizing legislation—namely, a source of heating or cooling in

residential dwellings. We have changed the wording of our

recommendation to make this more clear.

HHS suggested that our matters for congressional consideration include

the possibility of expanding the REACH program, as well as the options of

continuing it or setting an end date for the program. We have not

incorporated this comment because the effectiveness of the REACH

program has not yet been determined and because we believe that HHS

should be required to report to the Congress in 2003 on the REACH

program’s effectiveness.

HHS also made a number of technical comments, which we have

incorporated as appropriate.

We reviewed all of the REACH grants that have been awarded since the

program was first funded in fiscal year 1996. However, in reviewing

project evaluations, we focused only on states’ projects, because tribal

organizations are not required to do project evaluations. We also reviewed

documents about the LIHEAP program and interviewed LIHEAP officials

in order to better understand the context and purpose of the REACH

program. Specific actions that we took to accomplish each of our

objectives are listed as follows:

Scope and

Methodology
Page 27 GAO-01-723 REACH Program Effectiveness

• To obtain information about grant amounts, recipients, and project

activities, we reviewed REACH program summary documents and grant

proposals; interviewed state officials responsible for most of the state

grants awarded in the first 2 years of the program; and attended the

REACH program’s annual conference, where we heard presentations by

grantees.

• To obtain information about REACH program goals, objectives, and

performance measures, we reviewed the authorizing legislation, HHS

performance plans, requirements and guidance related to the Government

Performance and Results Act of 1993, and REACH program

announcements that request grant proposals. We also interviewed

program officials.

• To analyze the results of state evaluations, we reviewed the project design

and implementation of the six completed evaluations. We assessed the

adequacy of key aspects of project design and implementation since these

factors determine the confidence that can be placed on an evaluation’s

findings. By determining whether the evaluation reports contained critical

methodological flaws, we ascertained whether the reported findings were

so qualified as to preclude their use in assessing the REACH project’s

effectiveness. In addition, we summarized these evaluations, contacted the

state officials responsible for these six projects to discuss the evaluations,

and discussed efforts to improve evaluations with REACH program

officials and the program’s consultants.

• To identify the REACH program’s communications efforts, we interviewed

program officials and reviewed the material made available on the REACH

program’s pilot Web site and through REACH program conferences. We

did not assess the communications efforts of REACH grantees.

We conducted our review from December 2000 through July 2001 in

accordance with generally accepted government auditing standards.

We are sending copies of this report to the Secretary of HHS, the Director

of the Office of Management and Budget, appropriate congressional

committees, and other interested parties. We will also make copies

available to others upon request.
Page 28 GAO-01-723 REACH Program Effectiveness

If you or your staff have any questions about this report, please call me on

(202) 512-3841. Other key contributors to this report are listed in

appendix IX.

Jim Wells

Director, Natural Resources

 and Environment
Appendix I: REACH Project Activities

Page 29 GAO-01-723 REACH Program Effectiveness

Table 4 summarizes the activities planned for the 53 Residential Energy

Assistance Challenge Option (REACH) projects that have been funded

since the beginning of the program: 28 state and 25 tribal organization

projects. (The fiscal year 1999 North Carolina grant is not included in the

table because it was being redesigned at the time of our review because

the state had not deregulated utilities as had been expected when the

project was originally being designed.) Most project plans included

multiple activities. Several activities used by only one or a few projects are

not included below.

Table 4: Activities Planned by REACH Grantees

Activity

Number of state projects

providing the activity

Number of tribal organization

projects providing the activity

Total projects

providing the activity

Providing energy education through

workshops or in-home counseling

26 19 45

Conducting home energy audits 16 9 25

Providing home weatherization repairs 12 8 20

Providing weatherization kits 8 12 20

Providing other energy-related home repairs,

such as fixing windows or roofs

8 4 12

Replacing inefficient furnaces or appliances 15 5 20

Providing energy-efficiency devices, such as

programmable thermostats

7 3 10

Providing energy safety devices, such as

carbon monoxide detectors

4 48

Installing solar or wind power systems 0 2 2

Providing budget counseling 17 13 30

Negotiating with energy vendors to obtain

payment plans or forgiveness of past-due

bills

11 5 16

Providing financial incentives for reducing

energy use or paying utility bills regularly

4 9 13

Educating consumers about utility

deregulation

7 07

Paying past-due utility bills or connection

fees

4 04

Forming consumer cooperatives for energy

purchasing

3 03

Providing water conservation devices or

plumbing repairs

3 36

Providing energy-efficient light bulbs 9 4 13

Providing case management social services 16 1 17

Providing job skill or employment

development services

4 26

Making payments toward past-due rent or

mortgages

1 01

Source: GAO analysis of the grantees’ project proposals.

Appendix I: REACH Project Activities
Appendix II: Review of California’s REACH

Project Evaluation Report

Page 30 GAO-01-723 REACH Program Effectiveness

The primary goal of the California REACH project was to help low-income

households reduce their energy use by providing energy conservation and

other services. Specific project goals follow:

• Dwellings would become more energy-efficient.

• Health and safety risks would be mitigated.

• Families would become knowledgeable about energy use and

conservation.

• Families would make and keep energy conservation goals.

• Families would reach a stable level of energy consumption.

• Families would reduce their demand for Low-Income Home Energy

Assistance Program (LIHEAP) payment assistance.

• Eligible recipients would enroll in utility-supported rate discount

programs.

• Families that needed additional help in other areas would receive

assistance.

• Families would become stable and develop problem solving skills and

coping abilities.

The evaluation report did not clearly identify major project assumptions,

other than recognizing an expected participant attrition rate of 20 to 25

percent. To compensate for expected attrition, the project design reflected

plans for increasing the participant base.

The project was designed to determine which combinations of services

(described below) were the most effective in reducing energy use of lowincome households. The project was designed to use test groups and

control group comparisons. It was to use six test groups (one for each of

six combinations of services), with corresponding subgroups located at

three separate locations. Services were to be provided to REACH

participants by four different community-based organizations. A seventh

group—the control group—was not to receive services other than energy

assistance payments.

Utility bill data and other information were to be collected for all seven

groups before and after project implementation. The results of the REACH

groups were to be compared and evaluated.

Appendix II: Review of California’s REACH

Project Evaluation Report

Goals and Measures

Project Design and

Implementation

Project Assumptions

Evaluation Design
Appendix II: Review of California’s REACH

Project Evaluation Report

Page 31 GAO-01-723 REACH Program Effectiveness

The evaluation report identified five services to be provided by

community-based organizations through case management: (1) basic and

(2) enhanced energy conservation education, (3) basic and (4) enhanced

home weatherizing, and (5) family intervention (including referral to other

agencies when needed). The control group was to receive only energy

assistance payments.

In addition to the above services that would be evaluated, the communitybased organization was to provide a variety of other services, such as

outreach, eligibility determinations, residential assessments, and family

assessments.

Eligible project participants were households that received energy

assistance payments, had lived at the same residence for the past year, and

were not expected to move in the current year. In addition, households

were required to have metered energy—electricity, gas, or propane—for

their individual residence and could not have received previous home

weatherizing services. Project participation was further restricted to

households that spoke limited English, had a high energy burden,1

 received

public assistance for children, and/or were unable to pay current utility

bills. The evaluation report stated that participant eligibility was verified

using the standard process for determining eligibility for energy assistance

payments.

The evaluation report indicated that households were to have been

randomly selected and assigned to the various test and control groups on

the basis of the services they were to receive.

According to the evaluation report, there was no statistically significant

reduction in the use or cost of energy among the test groups. The report

found that the average monthly household energy use and cost actually

 1

Energy burden is defined as expenditures of a household for home energy (heating and

cooling) divided by the income of the household.

Services to Be Provided

REACH Participant and

Control Group Selection

Evaluation Findings

and Reported

Limitations

Findings
Appendix II: Review of California’s REACH

Project Evaluation Report

Page 32 GAO-01-723 REACH Program Effectiveness

increased slightly during the project. Overall, average monthly electrical

and natural gas use was also found to have increased. The results of the

individual test groups, however, were reported to be mixed. For example,

6 groups reduced their electrical use, 7 groups reduced their natural gas

use, and 10 groups reduced their total energy cost.

The evaluation report also concluded that data were insufficient to

determine whether the project had helped increase the regularity of utility

bill payments. In addition, the evaluation report noted that according to

the results of an energy awareness survey, participant knowledge of

energy conservation measures had increased slightly. The report did not

specifically assess the project’s impact on the other eight project goals.

Data collection problems were identified by the evaluation report as a

major problem that could affect conclusions about the project’s outcomes.

None of the community-based organizations provided all of the data

needed for the evaluation. One community-based organization’s work was

not included in the evaluation because it did not collect any follow-up

data. The information provided by the other three community-based

organizations was incomplete because they either lost or did not

completely collect the data. The evaluation report also noted that the

project did not collect data on the regularity of utility bill payments, and

the report provided no reason for this.

The evaluation report attributed these data collection problems to a lack

of controls over the data collection activities of the service providers—

noting that procedures for data collection were limited, and the service

providers had no incentives to collect follow-up data. The report also said

that community-based organization staff turnover and the lack of training

in data collection requirements contributed to the problem and

recommended that in the future final award payments be withheld until all

the data are provided.

In addition, the evaluation report noted that energy use data provided by

the community-based organization was not adjusted for changes in

weather and concluded that this could have significantly limited the

evaluation’s findings and conclusions.

The effectiveness of the California REACH project cannot be determined

on the basis of the information presented in the project evaluation report

because of limitations in the evaluation’s implementation and analysis. The

Reported Limitations

Our Observations
Appendix II: Review of California’s REACH

Project Evaluation Report

Page 33 GAO-01-723 REACH Program Effectiveness

report also provided insufficient information to determine the validity of

the findings and did not provide other information needed to assess the

project’s overall effectiveness.

Several key issues compromised this REACH evaluation effort. For

example, the evaluation did not compare the test groups with the control

group as called for in the project design. Without a control group there is

no assurance that changes found in the test groups did not result from

factors other than the combination of services provided. Moreover, while

the report noted that the project’s impact on utility bill payments could not

be assessed because of insufficient data, it did not identify or explain the

effect of attrition and the loss of data on the statistical validity of its

findings. The substantial household attrition that occurred and the

resulting loss of data could have further compromised the validity of the

project results because households that completed the project may differ

from those that dropped out in ways likely to affect the outcome.

In addition, as noted in the evaluation report, energy use data were not

adjusted for weather conditions, thereby further compromising the

statistical validity of the findings on energy use. For example, if

temperatures were significantly colder in winter or warmer in summer, a

corresponding increase in energy use would also be expected. If this

occurred during the period after energy conservation services were

provided, these services could have been responsible for stabilizing

household energy use or reducing the amount of energy that might

otherwise have been consumed. Such a finding could not have been

reported because the effects of changes in weather had not been taken

into account when measuring results.

In addition to not including all of information necessary to assess the

validity of its findings, the evaluation report did not provide other

important information needed to assess the overall performance of the

project. For example, the report did not address the project’s performance

regarding one of the project performance goals identified in the

authorizing legislation and the Department of Health and Human Services

(HHS) program announcement—increasing the contributions energy

suppliers make to reducing the energy burden of eligible households.

Finally, the evaluation report did not provide a complete discussion of the

project design, including information on project assumptions, hypotheses

to be tested in terms of measurable objectives, or lessons learned.

Analytical Problems

Other Shortcomings
Appendix III: Review of Maryland’s REACH

Project Evaluation Report

Page 34 GAO-01-723 REACH Program Effectiveness

The goals of the Maryland REACH project were to help participants

(1) reduce their electric bills, (2) improve their budgeting skills, and

(3) increase their education or work opportunities with the ultimate goal

of preparing them for self-sufficiency once the project was completed.

The evaluation report did not clearly identify major project assumptions.

The project was designed to use two groups of low-income households—a

test group and a control group—to assess of services provided. Data on

participants in the test group were to be collected (1) from participant

surveys—before and after the project—and (2) from utility statements for

the year before the project, when available, and for the year during project

implementation. Participant data before project implementation were to

be compared with participant data collected during project

implementation so that the effect of the services could be evaluated.

Utility bill statements were also to be obtained for the control group

receiving financial assistance for energy bills, so that a comparison could

be made between the test group and the control group. The evaluation

report also noted that energy use data were to be compared by “seasons”

to control for the effect of weather.

Services were to be provided by a community-based organization using a

case management approach. Each household was to receive four

caseworker contacts. During the first contact the caseworker would

perform an initial assessment of household energy use, income, education,

and occupation. During the second contact, the family assessment was to

be updated. The caseworker would also obtain an application for energy

assistance from the household and provide energy conservation

counseling. During additional contacts, the family assessment would

continue to be updated and energy conservation counseling provided.

Referrals to public assistance programs for help with rent or mortgage

payments, utility bills, home weatherizing, or job training would be made

when necessary. Other services would include a newsletter and

workshops on energy conservation. (The newsletter was eventually

cancelled because it was not being read, and the workshops were

cancelled because of low attendance.)

Appendix III: Review of Maryland’s REACH

Project Evaluation Report

Goals and Measures

Project Design and

Implementation

Project Assumptions

Evaluation Design

Services to Be Provided
Appendix III: Review of Maryland’s REACH

Project Evaluation Report

Page 35 GAO-01-723 REACH Program Effectiveness

In addition, program participants were often referred to other agencies

providing various kinds of assistance, and more than one referral could be

made for each participant. There were 152 referrals made for a variety of

social assistance services, including employment and education. Other

referrals were made for home weatherization assistance (86), such as

instruction on energy conservation; energy assistance payments (73); and

mortgage or rent assistance (20).

The local energy assistance office initially identified eligible households,

and later participants were recruited from energy assistance applicants

and referrals from other agencies. A questionnaire was used to establish

eligibility. Over a 25-month period, a total of 124 participants entered the

program and 92 participants were dropped from the program, primarily

because they could not be contacted.

The control group was selected from households receiving energy

assistance payments. Its members did not receive the REACH project

services noted above. The evaluation report noted that the size of the

control group was adjusted monthly to compensate for changes in the

number of project participants.

The evaluation report compared participant pretest and posttest data on

energy use and found that while households reduced their use of

electricity, the overall reduction was not statistically significant. However,

the report noted that a statistically significant reduction was seen in

winter. The average monthly energy use was reported to have decreased

by 61 kilowatt hours, from 1,275 kilowatt hours to 1,214 kilowatt hours.

The average cost of household monthly utility bills was reported to have

decreased by $16, from $137 to $121.

The evaluation report also found that the use of electricity by the control

group had increased by 55 kilowatt hours, and that the difference between

the two groups was statistically significant. As a result, the evaluation

report concluded that participating households were able to stabilize their

use of electricity while the control group households were not. The report

did not assign a dollar cost savings to the difference between the groups.

REACH Participant and

Control Group Selection

Evaluation Findings

and Reported

Limitations

Findings
Appendix III: Review of Maryland’s REACH

Project Evaluation Report

Page 36 GAO-01-723 REACH Program Effectiveness

Utility bills were also used to assess the impact of program services on

arrearages. According to the evaluation report, the number of late

payments and termination notices decreased for households participating

in the project, but neither decrease was statistically significant. Although

the number of termination notices significantly increased for the control

group, the number of actual terminations decreased, while the number of

actual terminations increased for participating households. According to

the evaluation report, this difference was statistically significant and

suggested that the project goal of improving budgeting skills was not

accomplished.

The evaluation report also analyzed monthly utility bill payments and

balances carried over from month to month to assess the ability of

households to manage their personal finances. The evaluation report

found a slight increase in electricity bill payments that was not statistically

significant but found a reduction in monthly balances that was statistically

significant. Neither difference was found to be significant when compared

with the control group.

On the basis of these findings, the evaluation report concluded that the

project achieved the following:

• Households received $60,000 in assistance for energy use and rent or

mortgage assistance.

• A total of 363 referrals were made to agencies providing assistance.

• Indications implied that households reduced or stabilized their energy use.

• Households received fewer termination notices.

• Households reported less difficulty in paying utility bills and in conserving

energy.

The report, however, also concluded that the project did not meet some

expectations:

• Over half of the households entering the project were dropped because

they could not be contacted later in the project.

• Newsletters and workshops were discontinued because of a lack of

interest.

• Participants did not decrease the amount of arrearages in their utility bills.

• A decrease was not achieved in the termination of utility services.
Appendix III: Review of Maryland’s REACH

Project Evaluation Report

Page 37 GAO-01-723 REACH Program Effectiveness

According to the evaluation report, the attrition rate of households

resulted in difficulty obtaining adequate information for the project

evaluation. In addition, survey responses were obtained in two ways: 29

households completed surveys, and 10 additional responses were obtained

through telephone interviews.

The effectiveness of the Maryland REACH project cannot be determined

on the basis of information presented in the project evaluation report. The

assessment has limitations that affect the validity of conclusions that can

be made regarding reducing energy use and other goals. For example, the

report does not present information on the comparability of the test and

control groups. In addition, there was a lack of information on the number

of months of utility data obtained for REACH and control group members.

The report did not address the possible effects of participant attrition on

the comparability of these groups. Moreover, energy use data were not

adjusted for the effects of weather. The report also had other

shortcomings that precluded an overall assessment of the project’s

effectiveness.

The report did not provide sufficient information on the test and control

groups to allow an assessment of their comparability. Discussion of

factors that could affect comparability of the test and control group was

incomplete in several respects. For example, the report did not provide

sufficient information on the number of months of utility bills available for

both groups. Estimates based on data for a few months can be more easily

influenced by factors unrelated to the project services than those based on

data for an entire year. Although the report stated there was a statistically

significant difference in energy use between the groups for the year, it did

not discuss the possible reasons a statistically significant difference was

not observed in winter when the control group also showed a decrease in

energy use.

The validity of the conclusions regarding energy use could also be

compromised by attrition in the test and control groups because

households completing the project may not be representative of the target

population as a whole. The report, however, did not identify the effect of

attrition and the resulting loss of data on the validity of its findings.

In addition, energy use data were not adjusted for possible changes in the

weather from preproject implementation to postproject completion. The

decrease in winter energy use noted in the report could have occurred

Reported Limitations

Our Observations

Analytical Problems
Appendix III: Review of Maryland’s REACH

Project Evaluation Report

Page 38 GAO-01-723 REACH Program Effectiveness

because of a relatively milder winter during the project. The seasonal

comparison of the test and control groups actually showed no significant

statistical difference between the groups during the project.

The evaluation report did not provide other important information needed

to assess the overall effectiveness of the project. For example, the report

did not address the project’s performance in terms of the performance

goal stated in the authorizing legislation and the HHS program

announcement—increasing energy supplier contributions to reducing the

energy burden of eligible households. The evaluation report also did not

provide a complete discussion of the project design, including information

on the project’s assumptions, hypotheses to be tested in terms of

measurable objectives, or lessons learned.

Other Shortcomings
Appendix IV: Review of Massachusetts’

REACH Project Evaluation Report

Page 39 GAO-01-723 REACH Program Effectiveness

The overall goal of the Massachusetts REACH project was to reduce

homelessness by fostering self-sufficiency. In that context, the grant

application identified three main project goals:

• reducing household energy costs,

• increasing the regularity of household energy bill payments, and

• increasing contributions by energy suppliers toward reducing households’

energy burden.

The evaluation report also identified five objectives for the REACH project

participants:

• Increase awareness of energy consumption.

• Increase awareness of the ability to shop for the least expensive energy

supplier.

• Learn budgeting.

• Make regular payments to energy suppliers.

• Negotiate arrearage forgiveness with energy service providers.

The evaluation report did not—except in the calculation of energy

burden—clearly identify major project assumptions. Assumptions related

to the calculation of household energy burden were discussed in some

detail (see below).

The project was designed as a single group of REACH participants

receiving a variety of services. The study stated that financial constraints

prevented using a control group. Participant data were to be collected

both before and after the project using a standard data collection

instrument, questionnaires, or both. Utility bills were to be obtained from

energy suppliers. The information collected would then be used to assess

changes in household self-sufficiency, knowledge of energy use, the effect

of specific interventions, and changes in energy burden by comparing preREACH data with post-REACH data.

The project focused on assisting households through case management

services. Caseworkers were to provide financial counseling and energy

Appendix IV: Review of Massachusetts’

REACH Project Evaluation Report

Goals and Measures

Project Design and

Implementation

Project Assumptions

Evaluation Design

Services to Be Provided
Appendix IV: Review of Massachusetts’

REACH Project Evaluation Report

Page 40 GAO-01-723 REACH Program Effectiveness

education and make referrals to other public assistance agencies for

services dealing with home energy conservation, employment, training,

day care, and improving language skills. Energy assistance payments were

to be the only immediate financial assistance provided—although 148 of

164 households (over 90 percent) reported receiving other public

assistance. Caseworkers also were to facilitate household access to an

arrearage forgiveness program offered by a major utility company. The

evaluation report noted that, during implementation, caseworkers

provided intake to over 460 households that were either homeless or at

risk of being homeless, and maintained contact with over 350 of these

households.

Project participants were selected from low-income households eligible to

receive energy assistance payments, and caseworkers recruited clients

from assistance programs for the homeless. The report did not describe

the selection procedure. As noted above, the project did not use a control

group.

According to the evaluation report, project results were difficult to

quantify, and the true measure of success was the range of services

provided in response to participants’ needs. However, information

gathered by case managers showed no change in self-sufficiency by

project participants. The evaluation report addressed project performance

in terms of the three main project goals.

• Reducing household energy costs

The evaluation report calculated the energy burden to households using a

formula developed to measure energy costs, or debt, as a portion of

income. On the basis of the results of the calculation, the evaluation report

found that the energy burden—the proportion of household income

represented by energy-related debt—decreased during the project from

$345 (105 percent) to $252 (64 percent). However, the evaluation report

noted that complete data—both pre- and postproject data for the same

households—were available for only five project participants. The report

also stressed that the calculation was based on the unlikely assumption

REACH Participant and

Control Group Selection

Evaluation Findings

and Reported

Limitations

Findings
Appendix IV: Review of Massachusetts’

REACH Project Evaluation Report

Page 41 GAO-01-723 REACH Program Effectiveness

that income would remain the same over time and on unreliable energy

bill data.

In addition, the evaluation report stated that, although the goal of reducing

energy costs appeared to have been achieved, the lack of posttest followup data prevented a determination of whether the objectives of increased

awareness of energy use and energy supplier options were accomplished.

• Increasing the regularity of household energy bill payments

The evaluation report assumed that participant utility bills would be

available to provide a record of payments. However, only electric bills

were available, limiting payment analysis to the 22 households using

electricity to heat their homes. Of these households, follow-up data existed

for only two, according to the evaluation report, making assessment of the

outcome impossible.

• Increasing contributions by energy providers toward reducing the

households’ energy burden

The evaluation report included information provided by the project staff

on the arrearage forgiveness program offered by a major public utility

company to households receiving energy assistance payments and

indicated that caseworkers processed 45 applications from project

participants for this program.

The evaluation report also identified lessons learned from experience with

the project and from interviews with case managers. Lessons learned

included the need for

• additional guidance from HHS on the conduct of evaluations,

• involving evaluators in project design,

• obtaining access to utility bills from utility companies, and

• thorough training for project staff involved in data collection.

The evaluation report noted that in many cases data were either unreliable

or missing, making an assessment of project outcomes impossible. The

report attributed this data problem to several factors, including the

transient nature of participating households, lack of training for

caseworkers, and failure to involve the evaluator early in the program

design.

Reported Limitations
Appendix IV: Review of Massachusetts’

REACH Project Evaluation Report

Page 42 GAO-01-723 REACH Program Effectiveness

The report also cited shortcomings in the assumptions used to calculate

and analyze household energy burden that adversely affected the

assessment of project performance. In addition, the report stated that data

records were sometimes constructed by case managers according to their

memory of events occurring weeks or even months earlier. The evaluation

report also noted that a scale used to measure self-sufficiency was never

tested for validity or reliability.

The effectiveness of the Massachusetts REACH project cannot be

determined using the information presented in the project evaluation

report. Performance could not be assessed and reported because of the

lack of data. Data were missing primarily because of attrition—

participants leaving the project before completion. In addition, the project

used a pretest/posttest project design that attempted to measure the

impact of delivering a variety of services but did not include a control

group. A comparison group is necessary to help ensure that the effects

observed resulted from services provided and not other variables and to

evaluate the effectiveness of the services compared with energy assistance

payments alone. The report had additional shortcomings that would

preclude an overall assessment of the project’s effectiveness.

On the plus side, the evaluation report discussed project performance in

terms of the performance goals set out in the authorizing legislation and

the REACH program announcement. The report also discussed the

difficulty encountered in measuring project performance because of data

collection problems and described the limits on statistical analysis due to

insufficient data. The report also identified issues relating to the

calculation of energy burden that could adversely affect the analysis. In

addition, the report provided a description of lessons learned that could

benefit the design of future projects.

Statistical estimates of project performance could not be made because of

the lack of data. Problems with households leaving the project before

completion, accompanied by the inability to obtain utility bills from all

energy providers, resulted in insufficient information to assess the effect

of services provided on either household energy use or the regularity of

utility payments. Attrition (leaving the project before completion) raises

the potential for biased results if those remaining in the project

systematically differ from those who drop out in ways that are likely to

affect the outcome. Concerns regarding data quality also arise from the

Our Observations

Analytical Problems
Appendix IV: Review of Massachusetts’

REACH Project Evaluation Report

Page 43 GAO-01-723 REACH Program Effectiveness

construction of records by case managers on the basis of memory months

after events occurred.

However, even if sufficient data had been available for statistical analysis,

failure to include a control group in the project design would have limited

conclusions regarding the effect of project services on household energy

use and changes in the regularity of utility bill payments. When a control

group is used, the credibility of the identified cause of observed outcomes

is generally greater because external factors have been taken into account.

Additional analytical problems identified in the evaluation report include

assumptions implicit in the energy burden calculation and the use of a

scale to measure changes in self-sufficiency that was not tested for either

reliability or validity. These shortcomings further limit findings or

conclusions that can be made about the project.

While the evaluation report clearly identified certain problems

encountered in assessing performance and the resulting limitations of its

findings, it did not provide or address the lack of other information needed

to assess overall project performance. For example, the report did not

address the project’s performance in terms of the performance goal stated

in the authorizing legislation and the HHS program announcement—

increasing energy suppliers’ contributions to reducing the energy burden

of eligible households. The report noted that the information needed to

assess this goal would be provided by the project staff in a separate report.

Other Shortcomings
Appendix V: Review of Michigan’s REACH

Project Evaluation Report

Page 44 GAO-01-723 REACH Program Effectiveness

The Michigan REACH project was comprised of two subprojects with

different goals. One subproject’s primary goal was the reduction of

household energy use. The other subproject’s primary goal was to educate

households on energy deregulation, provide consumer advocacy, and

explore the feasibility of bulk energy purchases.

The evaluation report identified eight performance measures that were to

be used to evaluate the impact of services on households:

• reducing utility bills,

• increasing the regularity of utility bill payments,

• increasing earned income,

• reducing reliance on energy assistance programs,

• increasing knowledge of energy conservation methods,

• increasing understanding of utility bills,

• reducing energy-related safety problems, and

• maintaining or increasing the availability of affordable housing.

The evaluation report did not clearly identify major project assumptions.

The project was divided into two subprojects with different goals for

different providers of education services. One subproject was to focus on

improving energy conservation and developing life skills, such as

budgeting. The other subproject was to focus on increasing knowledge

about energy deregulation. Both subprojects, however, would use a preand posttest evaluation design—collecting data before and after project

implementation—to evaluate the effectiveness of the services provided. A

control group that did not receive services was to be created from eligible

households to permit comparison with households that received services.

In the energy conservation and life skills education subproject, household

energy use data would be collected by seven service providers—located in

different areas—for the year before and the year after services were

provided. The evaluation report indicated that these data were to come

from utility bills obtained from energy suppliers. In addition, participants

would also be mailed a questionnaire after project completion to obtain

their views on the project.

Appendix V: Review of Michigan’s REACH

Project Evaluation Report

Goals and Measures

Project Design and

Implementation

Project Assumptions

Evaluation Design
Appendix V: Review of Michigan’s REACH

Project Evaluation Report

Page 45 GAO-01-723 REACH Program Effectiveness

In the energy deregulation education subproject, households would

complete a questionnaire before and after project implementation to

determine changes in their knowledge about energy deregulation.

The energy conservation and life skills education subproject was designed

to help low-income households develop plans identifying changes directed

at achieving an immediate reduction in energy use. Education would be

provided through workshops. Other workshops would also be provided

that addressed broader issues, such as preparing household budgets and

finding employment. This subproject initially targeted 1,100 households at

various locations throughout the state.

The other subproject was designed to prepare low-income households for

the effect of utility deregulation through education and to serve as a

consumer advocate. This subproject was to consist of placing

informational articles on energy deregulation in newsletters and in

pamphlets that were delivered to participating households.

The evaluation report stated that the subprojects had a target population

of households at or below 150 percent of the federal poverty level. The

report also stated that households in the deregulation education

subproject were participants in the Head Start Program. The report did not

further explain the selection criteria or process other than to note that the

control group would be selected from eligible households.

According to the evaluation report, there was no significant difference

before and after the project in natural gas use by participants at the three

locations for which data were available. However, on the basis of the

results of two locations reporting pre- and postproject data, the report

found the project effect on electricity use mixed: a significant reduction

was reported at one location but not at the other.

On the basis of participant responses to the postproject survey, the

evaluation assessed the results, or outcomes, of four performance

measures in qualitative terms:

Services to Be Provided

REACH Participant and

Control Group Selection

Evaluation Findings

and Reported

Limitations

Findings
Appendix V: Review of Michigan’s REACH

Project Evaluation Report

Page 46 GAO-01-723 REACH Program Effectiveness

• Increased regularity of utility bill payments—most participants described

this workshop education component as helpful.

• Reduced reliance on energy assistance programs—about one-third of the

participants indicated that their use of emergency energy assistance had

decreased.

• Increased knowledge of energy conservation methods—the majority of the

participants felt that the energy workshop was helpful.

• Increased understanding of utility bills—most participants found the

educational information helpful.

Also, on the basis of participant responses to the surveys, before and after

the project, the evaluation report concluded that there was no change in

household understanding of utility deregulation.

The evaluation report noted that the data results cited on natural gas use

should be used with caution because of the small number of households

for which data were available. Three locations provided data on natural

gas use, while two locations provided data on electricity use. Data on gas

use were adjusted for weather, but data on electricity use were not

adjusted. The report also noted that the control group was not valid

because some households may have also received weatherization services.

In addition, the evaluation report stated that changes were made to the

project in the second year, but no postproject data were available to

include in the final evaluation report.

Regarding the survey of participants, the evaluation report noted that the

assessment of perceptions of energy conservation and life skills

workshops was based on 299 surveys returned out of 740 surveys mailed—

a response rate of 40 percent.

According to the evaluation report, it was not possible to measure the

effect of the project on earned income, reductions in housing safety

problems, and availability of affordable housing.

The effectiveness of the Michigan REACH project cannot be determined

because of limitations in design and implementation. For example, the

lack of data on participant energy use, along with other analytical

problems, such as the lack of a control group and failure to adjust energy

use data for weather conditions, compromised the validity of the findings

on energy use. As a result, a valid assessment of the effect of project

activities on energy use or regularity of participant utility bill payments

Reported Limitations

Our Observations
Appendix V: Review of Michigan’s REACH

Project Evaluation Report

Page 47 GAO-01-723 REACH Program Effectiveness

cannot be made. In addition, there were limited months available on which

to base pre- and postintervention estimates, ranging from 2 to 4 months.

Moreover, the indication of a decrease in electric usage for one location

was based on data that were not adjusted for effects of changes in

weather. In addition, the report had other shortcomings that precluded an

assessment of the project’s overall effectiveness.

The first Michigan subproject focused on changing participant behavior

through education. The inability to obtain quality data on participants’

energy use from the service providers and the local energy supplier—

especially data on postintervention energy use—meant that conclusions

were based on relatively few records. For example, postintervention data

were not available on gas use for four of the seven locations and not

available for electricity use for five locations. Moreover, the

postintervention data did not cover an entire year—data were available for

only 2 to 4 months.

Overall attrition was also an issue. The evaluation report states that 1,028

participants were served by seven locations from October 1997 to

September 1999 but that pre- or postintervention fuel consumption data

were usable for 570 participants. Matched pre- and postintervention fuel

data were available for 93 participants using natural gas and 116

participants using electricity. These data come from three of the seven

locations. These limited data restrict inferences that could be drawn from

the originally intended locations.

Additionally, attrition occurred within the locations for which pre- and

postintervention data were available. One of the three locations served 200

participants and obtained 34 useable natural gas records and 43 electric

records. The second location served 156 participants and obtained 40

useable natural gas records and 73 electric records. The third location

served 150 participants and obtained 19 useable natural gas records and

no usable electricity data. This level of attrition raises the issue of bias

because participants that remain in the project may differ from those

dropping out in ways that are likely to affect the outcome. The high

attrition rate also reduces the precision of any statistical estimate of

change in fuel usage.

Even if sufficient data had been available for statistical analysis, not

having a valid control group for comparison greatly limited the ability to

make a valid assessment of project effects. Similarly, failure to adjust

Analytical Problems
Appendix V: Review of Michigan’s REACH

Project Evaluation Report

Page 48 GAO-01-723 REACH Program Effectiveness

energy use data for weather conditions prevented a valid assessment of

participant energy use.

In addition to not including all of the information needed to assess the

statistical analysis and validity of its findings, the evaluation report did not

provide other important information needed to assess the project’s overall

effectiveness. For example, the report did not address the project’s

performance in terms of the project performance goal provided in the

authorizing legislation and the HHS program announcement: increasing

energy suppliers’ contributions to reducing the energy burden of eligible

households. The report also did not provide a complete discussion of the

project design, including critical project assumptions, hypotheses to be

tested in terms of measurable objectives, or lessons learned from

implementing the design.

Other Shortcomings
Appendix VI: Review of Nebraska’s REACH

Project Evaluation Report

Page 49 GAO-01-723 REACH Program Effectiveness

The primary goal of the Nebraska REACH project was to increase the

economic self-sufficiency of low-income families by increasing household

energy efficiency. The project sought to decrease participants’ household

utility bills by reducing energy use. A second project goal was to increase

participants’ knowledge of energy conservation practices and personal

finance. Additional project goals included increasing the environmental

comfort and health and safety of the participating households.

The Nebraska REACH project was to focus on changing the behavior of

the target population—low-income households—as the best way of

achieving project goals. This approach was based on the assumptions that

low-income households were most often renters who frequently moved

and that the housing available to them was generally substandard. Given

these premises, it was assumed that the only way to reduce energy use in

this population would be to change household behavior through teaching

energy-efficient practices. It was also assumed that it would have been too

expensive to weatherize substandard housing, and upgrading the structure

would have benefited the household only temporarily, until its next move.

According to the evaluation report, the Nebraska REACH project provided

a unique opportunity to evaluate a social services project using a “true

experimental design.” The project was designed to use two groups of lowincome households—a test group and a control group—to evaluate the

primary hypothesis that changing behavior could reduce energy use. Data

collected on the two groups, both before and after project implementation,

would be analyzed using a standard computer program—the Princeton

Scorekeeping Method (PRISM) software—to determine the statistical

significance of the results. The PRISM software uses data from monthly

utility bills to produce a weather-adjusted index of energy use for both the

test and control groups. The program also compares the differences in

energy use among households in the test group and between households in

the test group and control group. Members of both the test group and the

control group would be required to sign forms authorizing the release of

their utility bills to the project.

In addition, knowledge of energy conservation practices and personal

financial management obtained from the workshops would be measured

Appendix VI: Review of Nebraska’s REACH

Project Evaluation Report

Goals and Measures

Project Design and

Implementation

Project Assumptions

Evaluation Design
Appendix VI: Review of Nebraska’s REACH

Project Evaluation Report

Page 50 GAO-01-723 REACH Program Effectiveness

using pre- and postknowledge tests and a third test given 6 months after

project completion.

Households were to receive four related services: (1) a home energy audit,

(2) instruction in energy conservation practices and personal financial

management, (3) case management, and (4) provision of basic home

weatherizing materials. Home energy audits would include an interview

and inspection by a certified energy specialist to assess the household’s

energy efficiency and form the basis of the action plan. Workshops

providing instruction on energy conservation and personal finance would

be available to participants each month, and training manuals would be

distributed providing detailed instruction on these issues. Case

management would include developing a household action plan, preparing

family assessments, and conducting monthly progress reviews during

home visits. Materials such as weather stripping, along with instruction on

installation, would also be provided. In addition to these services, project

participants would receive between $200 and $350 in vouchers to pay

utility bills. Control group members would receive $50 in vouchers for

utility bills.

Potential applicants were recruited through advertisements and social

service agency referrals. Case managers determined project eligibility

using four criteria: participants had to (1) have income at or below the

poverty level, (2) have resided at their current address for at least 1 year,

(3) not be planning to move, and (4) need assistance in paying utility bills.

The control group was randomly selected from every third eligible

applicant. There were no significant differences between the two groups in

either education or other demographic characteristics. During

implementation, the project provided services and cash assistance to 439

participants and cash assistance and goods to 202 control group members.

Separate analyses were done for natural gas and electricity use using the

PRISM computer program. PRISM provided normalized annual

consumption data (pre- and postproject) and normalized annual savings

separately for both gas and electricity.

Services to Be Provided

REACH Participant and

Control Group Selection

Evaluation Findings

and Reported

Limitations

Findings
Appendix VI: Review of Nebraska’s REACH

Project Evaluation Report

Page 51 GAO-01-723 REACH Program Effectiveness

The PRISM analysis indicated a statistically significant decreased use of

natural gas by REACH project participants compared with those in the

control group. Program participants had normalized annual savings of

$48.12 per household more than the control group—or a savings of

122 hundred cubic feet.

The result of the PRISM analysis for electricity use revealed a different use

pattern. Although the analysis showed no change in electricity use by

project participants, the analysis revealed an annual increase in electricity

use of 2,716 kilowatt hours by the control group. The evaluation report

stated that this effect was consistent with the hypothesis that electricity

use would be lower for project participants—an annual savings of

$149.38 per household.

On the basis of these findings, the evaluation report concluded that project

participants had achieved greater economic self-sufficiency through a

reduction in utility costs—an average of $197.50 each year—compared

with the nonparticipant control group. However, the report also noted that

an actual reduction in utility use for project participants was confirmed

only for natural gas and not for electricity.

In addition, given the results of the knowledge tests administered to

participants, the report concluded that the instruction provided during the

project resulted in a significant increase in knowledge about energy

conservation practices and personal financial management.

The evaluation report attributed the project’s ability to implement the

project design as planned and to obtain sufficient data to perform the

statistical analysis to the community-based organization’s experience,

expertise, and knowledge of the target population and to the early

involvement of the evaluator in the project’s design. Cash incentives

provided to the test and control groups also appeared to have reduced

participant attrition. In addition, administrative controls designed to

ensure access to utility bills seemed to have played a role in obtaining

sufficient data for analysis.

The evaluation report noted several factors that greatly reduced the

number of households whose energy use data could be used in the PRISM

analysis. For example, frequent changes in residence eliminated many

households from the analysis. In addition, frequent changes in household

size and composition, along with utility shutoffs, made much of the data

Reported Limitations
Appendix VI: Review of Nebraska’s REACH

Project Evaluation Report

Page 52 GAO-01-723 REACH Program Effectiveness

too unreliable for use in the PRISM analyses; only a small percentage of

the database was considered appropriate for analysis.

According to the evaluation report, analysis was limited to determining the

overall effect of the project, not the effect of specific services or

combination of services (i.e., identifying which services were most

effective in achieving a reduction in energy use.)

The effectiveness of the Nebraska REACH project can be assessed with

some qualification, given the information presented in the evaluation

report. The project’s design and implementation allowed statistically valid

conclusions to be made about the effect of project services on participant

energy use. The report did not address the two other performance goals of

reducing utility bill arrearages and increasing contributions by energy

suppliers and did not provide other information needed to fully assess the

project’s overall effectiveness.

The evaluation report did not provide an assessment of potential bias due

to attrition, which may result if those remaining in the project

systematically differ from those who dropped out, and its possible effect

on the analyses and conclusions.

Although the evaluation report provided a statistical analysis regarding the

project’s main hypothesis, the report did not address two of the project

performance goals identified in the authorizing legislation and the HHS

program announcement. Specifically, the report did not address the

performance goal of increasing the regularity of utility bill payments or the

performance goal of increasing contributions by energy suppliers. Finally,

the evaluation report also did not state the hypotheses in terms of

measurable objectives. The report did not provide a discussion of lessons

learned or best practices that could be valuable to other projects. For

example, if the report had discussed the strategies used to successfully

collect energy use data, this information could have been useful to other

projects. This lack of information is often cited as a principle reason for

not being able to measure project performance.

Our Observations

Analytical Problems

Other Shortcomings
Appendix VII: Review of Oregon’s REACH

Project Evaluation Report

Page 53 GAO-01-723 REACH Program Effectiveness

The primary goal of the Oregon REACH project was to help low-income

households manage their energy costs more effectively. The evaluation

report identifies four related project goals:

• reducing household energy use;

• reducing household energy cost;

• increasing the regularity of utility payments, reducing arrearages in utility

bills, or both; and

• eliminating health and safety risks related to energy use.

The long-term objectives of the project were described as sustained

reduced energy use, overall improvement in economic self-sufficiency, and

eventual elimination of reliance on energy assistance payments.

Three primary measures were used in the evaluation report to assess the

impact of the services provided by the project:

• Reducing household energy use—75 percent of participating households

will reduce energy use by 15 percent.

• Reducing household energy burden—no operational objective stated.

• Reducing arrearages in utility bills—75 percent of participating

households will reduce arrearages in utility bills, and 50 percent of

households will not incur new arrearages for 6 months.

Other measures used in the evaluation report addressed program

activities, such as completing action plans and enrolling participants in

social service programs.

The Oregon REACH project was based on the premise that providing

services to low-income households—such as information on energy

conservation practices and personal financial management—would result

in changes in their behavior that would reduce their energy costs and

utility bill arrearages. The evaluation report identified three general

assumptions that guided development of the project:

• Coordinating services provided by organizations within the community

would be more effective than uncoordinated assistance.

Appendix VII: Review of Oregon’s REACH

Project Evaluation Report

Goals and Measures

Project Design and

Implementation

Project Assumptions
Appendix VII: Review of Oregon’s REACH

Project Evaluation Report

Page 54 GAO-01-723 REACH Program Effectiveness

• Household knowledge of energy conservation, as well as assistance in

weatherizing homes, is necessary to reduce reliance on energy assistance

payments.

• Households that participated in the project would be willing and able to

make changes in their behavior and personal financial management that

could affect their energy costs and the regularity of their utility payments.

According to the evaluation report, the Oregon REACH project was

conceived as a “quasi-experimental” design. The project was designed as

an experiment to use two test groups and a control group to determine the

effect of the services provided. The report noted that the inclusion of a

control group was one of the distinctive features of the design.

A different combination of services would be provided to the two test

groups. One test group would receive a complete set of services—

including home weatherizing and heating system repairs—as needed. The

second test group would receive all of the project services, as needed,

with the exception of home weatherizing and heating system repairs.

Households within each group would receive different combinations of

services on the basis of need and availability of the service in the

community. The control group would not receive any of the project

services.

Data on energy use would be collected for all three groups, when possible,

for the year preceding the project, the year of project implementation, and

the year after project completion. Members of each group would be

required to sign forms authorizing the release of their utility bills to the

project. These data would be used to evaluate energy use, utility

arrearages, and energy burden by performing a statistical analysis of the

differences among groups.

Both participants and caseworkers would be surveyed after project

implementation to obtain their views on the project’s implementation and

usefulness. An incentive payment of $20 would be given to households to

complete the survey.

Services were to be provided to participating households by 13

community-based organizations across the state. Services would be

provided through case management that focused on working with

households to develop an action plan for reducing their energy use

according to an assessment of their needs. Case workers would also

Evaluation Design

Services to Be Provided
Appendix VII: Review of Oregon’s REACH

Project Evaluation Report

Page 55 GAO-01-723 REACH Program Effectiveness

provide instruction to households in energy conservation practices and

personal financial management, facilitate negotiation with energy

suppliers to develop payment plans for reducing arrearages, and make

referrals to other organizations providing social services.

Other services to be provided would include financial assistance to help

pay for utility bills and reduce arrearages, home energy audits, and home

weatherizing assistance. Project funds were also to be used to replace

water heaters, furnaces, and thermostats and to provide carbon monoxide

detectors and heating repairs, as needed. An emergency payment of

$200 was to be allowed for especially needy households.

Households were selected from those already receiving energy assistance.

Households also had to have utility bill arrearages equal to one-half the

energy assistance payment, an energy burden greater than 15 percent of

income, and an energy-related health or safety risk. In addition,

participants were selected on the basis of their motivation and the

priorities of local communities. According to the report, the control group

was selected from energy assistance recipients at each participating

community-based organization. The sampling procedure was not

discussed.

According to the evaluation report, the Oregon REACH project was largely

successful in achieving two of its primary goals: reducing energy use and

reducing arrearages in household energy bills while increasing the

regularity of payments to energy providers. The report noted that the

project strongly supported the assumption that coordinated services

effectively reduce energy use, energy costs, and energy burden for lowincome households. In addition, the report stated that its analysis

confirmed that the services helped households reduce arrearages and

increase the regularity of utility bill payments. The evaluation report

assessed the effect of services provided to households completing the

project. The evaluation report stated that participant attrition was about 8

percent in the last half of the first year and 16 percent in the second year

of the project.

REACH Participant and

Control Group Selection

Evaluation Findings

and Reported

Limitations

Findings
Appendix VII: Review of Oregon’s REACH

Project Evaluation Report

Page 56 GAO-01-723 REACH Program Effectiveness

The evaluation report found that both test groups reduced their electricity

use by 11 percent for the year after project implementation. Of the 173

households receiving services, 58 (or 33.5 percent) used less energy, and

40 (or 23 percent) achieved a reduction of at least 15 percent. This was 52

percent short of the project goal of 75 percent of households achieving the

15-percent reduction. In addition, households were found to have reduced

their energy burden by 2.5 percent.

The evaluation report also found that both test groups reduced arrearages

in their utility bills as a result of participation in the project and that this

difference was statistically significant. According to the report, the project

goal that 50 percent of households not incur new arrearages for 6 months

was met, and the number of households with arrearages decreased from

59 percent (102 participants) to 36 percent (63 participants). Moreover, the

evaluation report found that households receiving the additional services

of weatherizing and repairs achieved a slightly larger reduction in their

energy bill arrearages than the test group not receiving those services—

$77 compared with $55—but noted that the difference was not statistically

significant.

In the exit surveys, most households indicated that the project had helped

make their homes healthier, safer, and more comfortable and energyefficient. Similarly, in the staff survey, case workers indicated satisfaction

with the project.

As a result of all these findings, the evaluation report concluded that the

Oregon REACH project greatly assisted low-income households in

achieving a greater degree of energy self-sufficiency.

The evaluation report did not specifically identify limitations other than to

note that the difference in the energy use among the groups before project

implementation was statistically significant, suggesting that the

households may not have been assigned to the test and control groups in a

random fashion. The control group had the lowest level of energy use both

before and after project implementation.

The evaluation report also noted that not all households received the full

benefits because they did not complete the project and that it was difficult

to contact some participants because they did not keep appointments or

did not have telephone service. Two hundred and twenty four households

were listed as members of the test groups, and fewer households were

Reported Limitations
Appendix VII: Review of Oregon’s REACH

Project Evaluation Report

Page 57 GAO-01-723 REACH Program Effectiveness

used in the various analyses. Data were not available for participants

entering the project in the second year.

The effectiveness of the Oregon REACH project cannot be determined on

the basis of the information presented in the project evaluation report.

Although the study did have a control group, the omission of a comparison

of the test groups with the control group is a shortcoming in the analyses:

• Only the energy consumption analysis compared the control group directly

with the REACH groups.

• The assessments of both energy burden and change in arrearages used two

REACH test groups but did not compare changes between the two groups.

Instead, they assessed changes from pre-REACH to post-REACH project

points within each of the groups separately.

• The study did not assess the impact of the project by comparing the

change in energy use of the two REACH groups to the change in energy

use of the control group.

The report also noted that participants might not have been assigned to

groups in a random manner, leaving the possibility that the changes

observed could be the result of factors other than the services provided.

Finally, the report had other shortcomings that precluded an overall

assessment of the project’s effectiveness.

The evaluation report did not identify important limitations of the analysis

or their effect on the conclusions. As noted, failure to compare the test

groups and the control groups for assessment of changes in energy use

and arrearages limits the usefulness of the findings. The report also did not

address the effect of the criteria and procedure for selecting both the test

and control groups, as well as the attrition rates from these groups. These

factors also adversely affect the report’s analysis and conclusions. For

example, assignment to the test and control groups appears not to have

been on a random basis. Test group selection resulted in a group of

households with the highest energy use, whereas control group selection

resulted in a group of households with the lowest energy use. Such

assignment issues weaken the conclusions that can be made about the

effect of REACH services on energy use. In addition, the report did not

state whether energy use data were adjusted for weather.

Our Observations

Analytical Problems
Appendix VII: Review of Oregon’s REACH

Project Evaluation Report

Page 58 GAO-01-723 REACH Program Effectiveness

In addition to not including all of the information necessary to assess the

statistical analysis and validity of the project’s results, the evaluation

report did not provide other important information needed to assess the

project’s overall effectiveness. For example, the report did not address one

of the project performance goals stated in the authorizing legislation and

the HHS program announcement—increasing energy suppliers’

contributions to reducing the energy burden of eligible households. In

addition, the report did not provide information on the dollar amount of

the expected reduction in energy use or the amount of financial assistance

given to households. Finally, the evaluation report did not provide a

discussion of lessons learned. A discussion of best practices—the

techniques, procedures, and controls—used to ensure sufficient data for

analysis might have been useful to future projects.

Other Shortcomings
Appendix VIII: Comments From the

Department of Health and Human Services

Page 59 GAO-01-723 REACH Program Effectiveness

Appendix VIII: Comments From the

Department of Health and Human Services

Note: GAO comments

supplementing those in

the report text appear at

the end of this appendix.
Appendix VIII: Comments From the

Department of Health and Human Services

Page 60 GAO-01-723 REACH Program Effectiveness

See comment 1. Page

numbers in the draft

report may differ from

those in this report.
Appendix VIII: Comments From the

Department of Health and Human Services

Page 61 GAO-01-723 REACH Program Effectiveness

See comment 2.
Appendix VIII: Comments From the

Department of Health and Human Services

Page 62 GAO-01-723 REACH Program Effectiveness

See comment 3.
Appendix VIII: Comments From the

Department of Health and Human Services

Page 63 GAO-01-723 REACH Program Effectiveness
Appendix VIII: Comments From the

Department of Health and Human Services

Page 64 GAO-01-723 REACH Program Effectiveness

See comment 2.

See comment 2.
Appendix VIII: Comments From the

Department of Health and Human Services

Page 65 GAO-01-723 REACH Program Effectiveness
Appendix VIII: Comments From the

Department of Health and Human Services

Page 66 GAO-01-723 REACH Program Effectiveness

See comment 6.

See comment 5.

Now on pp. 16-17.

See comment 4.
Appendix VIII: Comments From the

Department of Health and Human Services

Page 67 GAO-01-723 REACH Program Effectiveness

Now on p. 17.

See comment 7.
Appendix VIII: Comments From the

Department of Health and Human Services

Page 68 GAO-01-723 REACH Program Effectiveness

See comment 8.
Appendix VIII: Comments From the

Department of Health and Human Services

Page 69 GAO-01-723 REACH Program Effectiveness
Appendix VIII: Comments From the

Department of Health and Human Services

Page 70 GAO-01-723 REACH Program Effectiveness

The following are GAO’s comments on the Department of Health and

Human Services’ letter dated July 23, 2001.

1. The technical comments have been incorporated as appropriate.

Several more substantive technical comments are addressed

individually below.

2. The authorizing legislation defines home energy as “a source of heating

or cooling in residential dwellings.” While HHS argues that residential

energy commonly refers to all household energy use, for the purposes

of this review, we used the narrower definition stated in the legislation.

We have also changed the wording of our third recommendation,

concerning the use of REACH program funds, to clarify that REACH

program funds should be used for activities directly related to the

home heating and cooling needs of low-income households. We do not

dispute that a broad range of services is authorized in the legislation,

but these services should relate to home heating and cooling.

3. We are not including the possibility of expanding the program in our

matters for congressional consideration because the effectiveness of

the REACH program has not yet been determined and because we

believe that HHS should be required to report to the Congress in 2003

on the REACH program’s effectiveness.

4. While coordination among programs is addressed in a general way in

the REACH program announcements, we believe that HHS should

more specifically articulate its view of the REACH program’s purpose

and relationship to LIHEAP in the Administration for Children and

Families’ performance plan. We also note that a performance plan is

more readily accessible to the Congress, the public, and other agencies

than the program announcements intended for potential grant

applicants.

5. Our draft report did not state that the use of randomized control

groups is the only acceptable design, so we have not made any

wording changes. Although there are legitimate questions concerning

using randomized control group designs in social service programs,

such designs have sometimes been effectively used. For instance,

when resources are not sufficient to provide services to all who are

eligible, random assignment to control groups can be feasible and

ethical and provide convincing results. When randomized control

groups are not feasible, evaluations can be designed to use

GAO Comments
Appendix VIII: Comments From the

Department of Health and Human Services

Page 71 GAO-01-723 REACH Program Effectiveness

nonequivalent control groups, and, with suitable caveats, such designs

may yield defensible results. Regardless of the design used, evaluations

of project results must be able to distinguish between results that are

most likely due to the project services and results that may be due to

other, external factors.

6. We have added language indicating that the Office of Community

Services is aware of the evaluation challenges that we cite.

7. Pages 14 and 15 of our report describe the legislative authority and

restrictions on the use of LIHEAP funds, as well as the need to use

much of LIHEAP’s funding for direct assistance with energy bills. We

also report that, as allowed by law, 25 states plan to use a portion of

their fiscal year 2001 LIHEAP funds for activities such as energy

education and budget counseling and 44 states plan to provide

weatherization services. As HHS’ comments note, opportunity exists to

continue some of the activities tested under REACH grants through

states’ LIHEAP funds. We have made minor wording changes to more

clearly recognize that not all REACH activities may fall within the

legislative and funding constraints of the LIHEAP program.

8. Coordinating with other programs is not the same as spending REACH

grant funds for non-energy-related activities. We believe that HHS’

planned efforts to clarify language in its program announcement

should help ensure that REACH funds are used only for activities

directly related to the home energy needs of low-income households

and should provide a clearer basis for reviewing and selecting

proposals.
Appendix IX: GAO Contact and Staff

Acknowledgments

Page 72 GAO-01-723 REACH Program Effectiveness

Jim Wells (202) 512-3841

In addition, Julie Gerkens, Kathleen Gilhooly, Curtis Groves,

Rachel Hesselink, Judy Pagano, and Don Pless made key contributions to

this report.

Appendix IX: GAO Contact and Staff

Acknowledgments

GAO Contact

Acknowledgments

(360030)
The first copy of each GAO report is free. Additional copies of reports are

$2 each. A check or money order should be made out to the

Superintendent of Documents. VISA and MasterCard credit cards are also

accepted.

Orders for 100 or more copies to be mailed to a single address are

discounted 25 percent.

Orders by mail:

U.S. General Accounting Office

P.O. Box 37050

Washington, DC 20013

Orders by visiting:

Room 1100

700 4th St., NW (corner of 4th and G Sts. NW)

Washington, DC 20013

Orders by phone:

(202) 512-6000

fax: (202) 512-6061

TDD (202) 512-2537

Each day, GAO issues a list of newly available reports and testimony. To

receive facsimile copies of the daily list or any list from the past 30 days,

please call (202) 512-6000 using a touchtone phone. A recorded menu will

provide information on how to obtain these lists.

Orders by Internet

For information on how to access GAO reports on the Internet, send an

e-mail message with “info” in the body to:

Info@www.gao.gov

or visit GAO’s World Wide Web home page at:

http://www.gao.gov

Contact one:

• Web site: http://www.gao.gov/fraudnet/fraudnet.htm

• E-mail: fraudnet@gao.gov

• 1-800-424-5454 (automated answering system)

Ordering Information

To Report Fraud,

Waste, and Abuse in

Federal Programs