Residential Energy Assistance: Effectiveness of Demonstration Program as Yet Undetermined
Released: 2001-08-17T00:00:00

Letter

Background

The Congress established the REACH program to minimize the health and safety risks that result from high energy burdens prevent homelessness as a result of inability to pay energy bills, increase the efficiency of energy usage by low-income families, and target energy assistance to individuals who are most in need.

The REACH legislation requires that project plans provide a variety of services and benefits, which may include energy efficiency education, residential energy demand management services, counseling related to energy budget management and payment plans, and negotiations with home energy suppliers on behalf of eligible households. The legislation further requires each state’s plan to describe performance goals for its projects and the indicators the state will use to measure whether each project has achieved its performance goals.

OCS’ stated purpose for the REACH projects is to demonstrate the long- term cost-effectiveness of supplementing energy assistance payments with nonmonetary benefits that can increase the ability of eligible households to meet energy costs and achieve energy self-sufficiency. REACH is part of a much larger program, LIHEAP, which primarily provides financial assistance to low-income households for home heating and cooling. REACH projects may target their services to a portion of the population eligible for LIHEAP, such as a geographic area or type of client (households with elderly people, for example).

Since fiscal year 1996, REACH funding has ranged from $5.5 million to $6.8 million annually, while the total funding for LIHEAP has ranged from $1.2 billion to $2.1 billion. Over the course of the REACH program, its funding has averaged about one-half of 1 percent of total LIHEAP funds. The legislation establishing REACH provided for funding it from LIHEAP’s incentive program for leveraging nonfederal resources. This incentive program provides additional monies to states that, in the previous year, obtained additional assistance for low-income households’ energy needs from such sources as state funds, utility companies, and private charities. The REACH legislation provides that, for each fiscal year, the Secretary of HHS may allocate up to 25 percent of the funding for this incentive program to the REACH program, and HHS has allocated approximately this amount each year since the REACH program was first funded in fiscal year 1996.

Through the REACH program, OCS awards grants to states and to tribes, tribal organizations, and insular areas (generally referred to in this report as “tribal organizations”). OCS issues annual requests for grant proposals and then awards grants on a competitive basis. On average, about one-third of the applicants from fiscal year 1996 through fiscal year 2000 received a REACH grant. OCS’ Division of Community Demonstration Programs manages the REACH program.

State and tribal organization grants differ in a few ways. OCS has established a 3-year time frame for states to complete their projects, in part to allow time for states to contract with the community-based organizations that carry out the states’ REACH projects. Tribal projects have a 17-month time frame if the tribal organization carries out the project itself, or a 3-year time frame if the tribal organization chooses to use a community-based organization. Unlike states, tribal organizations are not required to conduct evaluations on the effectiveness of their project approaches, but tribal organizations do submit final reports on their projects to OCS. In addition, OCS has established differing maximum grant amounts for state and tribal grants. According to the director of OCS’ Division of Community Demonstration Programs, these caps on grant amounts reflect the differing amounts that states and tribal organizations receive in their LIHEAP allotments and resulted from focus meetings with tribal representatives held by OCS’ LIHEAP staff prior to the first REACH request for grant proposals.

OCS’ annual requests for grant proposals, referred to as program announcements, provide criteria for program eligibility, proposals, selection of projects to fund, and states’ evaluations of their projects. Only states and tribal organizations that receive LIHEAP grants are eligible to participate in REACH, and the households that may receive REACH project services are those eligible for LIHEAP assistance. In its program announcements for fiscal years 1996 through 2000, OCS required states’ and tribal organizations’ proposals for REACH grants to address the following elements:the organizational experience and capability of the community-based or tribal organization(s) conducting the project; project staff skills and responsibilities; state-level management and organization of the project; project theory and design, including the target population and needs to be addressed, activities, expected outcomes, goals, and work plan; budget appropriateness and justification; expected beneficial impact; “holistic” strategies addressing the economic and social barriers to self- sufficiency, and project innovations (required for state proposals only); community empowerment of areas characterized by severe poverty, high unemployment, or other indicators of socioeconomic distress (required for state proposals only); and evaluation of the project.

OCS uses the above proposal elements in its criteria for selecting which projects to fund. OCS convenes panels to review and score the grant proposals against the criteria; state and tribal organization proposals are assessed separately. Factors in addition to the scoring may be used in making selections, such as geographic distribution and OCS’ past experiences with the applicants.

States’ evaluations of their projects are to address how effectively each project was implemented, and whether and why the expected results and goals were or were not achieved. OCS requires states to use third-party evaluators to conduct the evaluations. Such evaluators are individuals or firms that are organizationally distinct from the state agency or community-based organizations involved in the REACH project.

Demonstration Projects Have Tried Various Approaches and Included Some Activities Not Directly Related to Home Energy

Through the end of fiscal year 2000, OCS had awarded $30 million in REACH grants to states and tribal organizations for 54 projects (29 state and 25 tribal projects) to address the home heating and cooling needs of low-income households using a variety of approaches. The most commonly used approaches have been energy efficiency education, home energy audits, home weatherization, and budget counseling, while innovative approaches have included forming consumer cooperatives for energy purchasing and using solar and/or wind power. Several projects have included activities not directly related to home energy—for example, job skill or employment development services and financial assistance toward overdue rent or mortgage payments. After its REACH grant period is over, a state or tribal organization may wish to continue using the approaches tested in the REACH project. Some activities could be replicated in a state’s or tribal organization’s LIHEAP program, if the state or tribal organization chose to do so.

Grant Recipients and Amounts

OCS has awarded REACH grants for 29 state and 25 tribal projects. Table 1 below shows REACH grant funding and numbers of grants by year and type of grant (state or tribal organization). In addition to the grant funding shown in the table, HHS has provided a total of $1.1 million to states for their costs in administering the grants, overseeing the community-based organizations that carry out REACH projects, and contracting for the third- party evaluations. (Tribal organizations have not received such funding because, to date, the tribal organizations have chosen to carry out their REACH projects themselves and are not required to contract for third- party evaluations.) Including both the grant funding and administrative funding for states, HHS provided $31 million in REACH program funds in fiscal years 1996 through 2000.

Tables 2 and 3 below list the grant awards made to states and tribal organizations, respectively, in fiscal years 1996 through 2000. Four states—Michigan, Nebraska, Oregon, and Pennsylvania—have received more than one grant, for separate projects. Similarly, five tribal organizations and one insular area have received more than one grant. State grants ranged in amount from $166,667 to $1.6 million and averaged $931,108. Tribal organization grants ranged in amount from $50,000 to $199,276 and averaged $118,051.

Project Activities

The activities conducted under the demonstration grants have varied. Almost all of the REACH projects included multiple activities and addressed more than one of the following areas: reducing energy use for home heating and cooling by increasing efficiency; helping clients pay for energy bills through budget planning, consumer education, consumer cooperatives, and other methods; reducing the use of other utilities (water and electric lighting); and providing social services not directly related to home heating and cooling needs.

Appendix I provides further details about the types of activities included in REACH project plans and the number of projects conducting each type of activity.

The most common project activity has been energy efficiency education, which was included in 45 of 53 projects. This education has been provided through group workshops for the projects’ clients or through in-home counseling of individual households. Some grantees have also developed energy educational materials, such as pamphlets or videos, some of which were designed for children. Other activities aimed at reducing energy use have included home energy audits and weatherization. Home energy audits were used by 25 REACH projects, often in conjunction with weatherization of the home or education in how to reduce energy use. In a home energy audit, a detailed inspection of the home is carried out to identify repair and weatherization needs and practices of the household that result in inefficient energy use. For example, doors and windows are inspected for drafts, attics for insulation needs, and furnaces for maintenance needs. Families may be advised of energy-saving practices such as turning down their thermostats at night or using fans to reduce the need for air conditioning.

Twenty REACH projects have provided home weatherization work by construction contractors. While the Department of Energy’s (DOE) Weatherization Assistance Program helps many low-income families, some homes need repairs that exceed the per-house dollar limit of the DOE program. The REACH projects that included weatherization typically went beyond the services provided by the DOE program or served clients who had not been addressed by the DOE program. Some REACH projects worked in conjunction with DOE’s program. In addition, 20 REACH projects provided clients with do-it-yourself weatherization kits and training in how to install the weatherization measures. The kits included items such as caulk, weather-stripping, and plastic coating for windows.

Budget counseling was provided in 30 of the REACH grant projects. Such counseling helps clients to plan ahead for paying for their essential needs, including energy bills; to identify areas where they could cut costs; and to become better informed about credit practices. Budget planning can help low-income households avoid utility cutoffs due to unpaid bills, which are followed by the need to pay reconnection fees and past-due bills. Utility cutoffs are not uncommon for low-income households. According to OCS data, during the 1992-93 heating season, 1 million LIHEAP-eligible households (3.3 percent of such households) reported that they were unable to use their main source of heat for 2 hours or more because they were unable to pay for their main heating fuel. Sixteen REACH projects also negotiated with energy vendors on behalf of their clients to obtain payment plans or forgiveness of past-due bills, and four projects provided funds to help pay past-due bills and/or reconnection fees. Seven projects provided consumer education on utility deregulation and the consumer choices provided by deregulation.

Some REACH projects addressed the use of utilities for purposes other than heating and cooling. Thirteen projects provided energy-efficient light bulbs. Energy-efficient lighting was the only focus of the two grants received by American Samoa because the Samoan government determined that this was the one measure most likely to reduce the high cost of electricity for their low-income households. Six projects addressed water conservation in the home by providing devices such as low-flow showerheads or repairing plumbing. Such plumbing measures also reduce energy usage for hot water heaters.

Some REACH projects included social services not directly related to home energy needs. Seventeen of the REACH projects provided social services through case management. For example, Nebraska’s fiscal year 1996 grant project included caseworker visits to help families identify and address their concerns in areas such as improving job prospects, eating nutritional food, and obtaining health services. Only Indiana’s project provided funds to help clients pay past-due rent or mortgage payments. Six projects included job skill or employment development services. For example, Indiana’s fiscal year 1997 grant included the use of REACH funds for job skill training, transportation to work, and day care; $258,000 of Indiana’s planned project budget addressed such services to help clients move from welfare to work. In many states, services that support the transition of welfare recipients to the workforce—such as job training, transportation, and child care—are provided by the much larger Temporary Assistance to Needy Families program ($22.6 billion in federal and state funding in fiscal year 1999). “OCS is interested in having Applicants approach the energy needs of low-income families within a holistic context of the economic, social, physical, and environmental barriers to self-sufficiency. Thus applicants should include in their REACH Plan an explanation of how the proposed projects(s) will be integrated with and support other anti-poverty or development strategies within the target community or communities.… Thus REACH initiatives are expected to be closely coordinated with other public and private sector programs involved with community revitalization, housing rehabilitation and weatherization, and family development.”

Similar language has been included in each program announcement requesting REACH grant proposals. Furthermore, the criteria used to select grant proposals to fund have included holistic program strategies and project innovations. REACH grant proposals are scored on a number of criteria—such as the organization’s experience and the project strategy and design—and each criterion has a maximum number of possible points. Out of a total of 100 possible points, state proposals for fiscal years 1996 through 2000 could receive up to 10 points in the selection process for the criterion “holistic program strategies and project innovations.” When asked about this criterion and the non-energy-related activities, REACH program officials said that the criterion might need to be clarified and that they also had some concerns about the emphasis on nonenergy activities in a few projects. However, they also noted that the legislation sets broad purposes for REACH, including preventing homelessness and minimizing health and safety risks resulting from high energy burdens on low-income households. Because energy burden is defined as home energy expenditures divided by household income, the officials stated that some grantees had chosen to address increasing clients’ incomes.

Some REACH projects have involved innovative activities. Examples of innovative activities include the installation of solar and/or wind power for low-income households in two tribal projects: the Grand Traverse Band of Ottawa and Chippewa Indians (fiscal year 2000 grant) and the Cherokee Nation (fiscal year 1997 grant). Another innovative activity used in three state projects was the formation of consumer cooperatives to reduce the cost of energy to low-income households. Vermont’s fiscal year 1997 project formed a cooperative and purchased a home heating oil company. The other two state projects—New York’s and Connecticut’s—are forming cooperatives to purchase energy from deregulated utilities, allowing low- income households to aggregate their purchasing power and negotiate lower prices. Pennsylvania’s fiscal year 2000 project is addressing cooling needs in urban areas, following a number of heat-related deaths among elderly residents, particularly in urban row houses. The project is both introducing an innovative use of heat-reflective coatings on roofs of inner city row houses and providing fans and safety devices that permit windows to be locked in both open and closed positions.

REACH grantees have also developed innovative ways of working with other organizations and delivering services. The project of the Central Council of the Tlingit and Haida tribes in Alaska trained AmeriCorps personnel to provide energy efficiency education. Because these personnel remain in the Alaskan villages, the project proposal stated that the impact of the services would continue beyond the completion of the grant period. Nevada’s project for its fiscal year 2000 grant is using the concept of individual development accounts to provide clients with matching funds for their savings. The REACH clients’ savings accounts are to be used for replacing the furnace or appliances, changing to a lower cost fuel, or paying higher winter utility bills. Kentucky’s project used volunteers from local organizations to install weatherization kit materials in the homes of disabled and elderly clients who could not do the work themselves.

Replication of REACH Project Activities

After REACH projects are completed, some project activities may be replicated in a state’s or tribal organization’s LIHEAP program, if the state or tribal organization chooses to do so. (The state or tribal organization may also choose to use its own funding to continue the activities.) The legislation authorizing LIHEAP allows states to spend up to 5 percent of their allotted LIHEAP funds for services that encourage and enable households to reduce their home energy needs and therefore their need for energy assistance. According to LIHEAP program officials, such services may include energy efficiency education, budget counseling, and assistance in obtaining discounts or payment plans from energy vendors. According to states’ fiscal year 2001 plans, 25 of 50 states and the District of Columbia planned to use a portion of their LIHEAP funds for energy use reduction services, and their planned services included energy education, energy needs assessment, liaison with energy vendors, budget counseling, and case worker services for clients. The legislation also allows states to spend up to 15 percent of their allotted LIHEAP funds for low-cost weatherization or other energy-related home repair for low-income households. States may also apply to HHS for waivers to use up to 25 percent of their allotment for this purpose. Forty-four of 50 states and the District of Columbia planned to provide weatherization in fiscal year 2001.

Several types of activities used in REACH projects may not be covered by these provisions of LIHEAP, such as forming or expanding energy consumer cooperatives, installing solar and wind power units, providing efficient light bulbs, and providing matching funds for clients’ savings to be used for energy purposes. It is uncertain whether states may use LIHEAP funds if they wish to continue or expand such activities after the conclusion of their REACH project. LIHEAP officials noted that because LIHEAP is a block grant program, states decide how to design their programs and interpret the statutory provisions regarding energy use reduction services and weatherization. Furthermore, LIHEAP’s funding caps in these areas, as well as the need among low-income households for direct assistance with energy bills, would determine the extent to which the activities tested in a REACH project could be provided more broadly to a state’s or tribal organization’s LIHEAP clients.

REACH Program Lacks Performance Goals and Measurable Indicators

The legislation authorizing REACH identifies performance goals to be used by individual REACH projects. For the program as a whole, however, HHS has not developed performance goals and measurable indicators that define the results that it expects to achieve. Furthermore, HHS’ performance plans do not address how the REACH program relates to the larger LIHEAP program of which it is a part.

The legislation authorizing the REACH program requires that REACH project plans describe performance goals for each project, which are to include a reduction in the energy costs of participating households over 1 or more an increase in the regularity of home energy bill payments by eligible an increase in energy vendor (such as utility companies) contributions towards reducing energy burdens of eligible households.

The legislation further requires that project plans include a description of the indicators that each state will use to measure whether the performance goals have been achieved. For example, in addressing the performance goal of more regular energy bill payments, Oregon’s proposal for its fiscal year 1999 grant specified the following indicator: 50 percent of households with past-due energy bills will reduce their arrearages by 40 percent after 12 months of participation. In addition, OCS requires states to address how results will be measured in their plans for evaluating their projects.

Despite the program’s use of project-level performance goals and indicators, the REACH program as a whole lacks performance goals and measurable indicators. According to a REACH official, the Administration for Children and Families decided not to address the REACH program in its performance plan, which was developed to meet the requirements of the Government Performance and Results Act of 1993. While agencies’ performance plans do not have to include the complete array of goals and measures used in managing individual programs, the development and use of performance goals and indicators can be beneficial to any federal program. Specifically, performance goals and measurable indicators can help a program to focus its efforts on achieving results and on those activities most closely linked to program goals, provide a clearer basis for selecting projects to fund, provide a basis for determining how well the program is performing and what has been achieved in return for the resources invested in it, and facilitate reporting to the Congress and the public on the program’s performance.

For the REACH program in particular, performance goals and measurable indicators could address two needs: They could both provide a basis for grant selection and enhance effectiveness in carrying out federal roles. First, as described above, OCS has selected several grant proposals that used a substantial portion of their grant funding for activities that were not directly related to home energy needs, such as job skill or employment development services. The use of performance goals could help OCS to better target grant selection to projects that are closely linked to program goals. Second, the federal roles in the REACH program are not fully addressed by the project-level goals. For example, the federal role in the REACH program currently includes selecting grantees, providing program guidance, helping to strengthen project design and states’ evaluations of their projects, and providing information to grantees and others involved in providing energy assistance. The effectiveness of federal efforts in the program could be enhanced by results-oriented performance measures addressing these roles.

Furthermore, the performance plan for the Administration for Children and Families (which includes OCS) does not address the relationship between REACH and LIHEAP, or between REACH and other related federal programs, such as DOE’s Weatherization Assistance Program. Performance plans can be useful tools for identifying the need for coordination among programs, and for ensuring that the goals of related programs are congruent and that crosscutting efforts are mutually reinforcing. In the case of REACH, HHS could use this performance plan to articulate REACH’s purpose in demonstrating and providing incentives for LIHEAP grantees to try new approaches. Without HHS’ addressing the relationship between REACH and LIHEAP, in particular, it is unclear whether HHS expects the REACH program to have a broader impact on the activities of LIHEAP grantees, beyond a particular REACH grant project and its limited time frame.

Initial Project Evaluations Have Limitations, but the Program Is Taking Action to Improve Future Evaluations

The six states that received funding in fiscal year 1996 have prepared evaluation reports: California, Maryland, Massachusetts, Michigan, Nebraska, and Oregon. However, only Nebraska’s evaluation report, with some qualification, fairly reflects the REACH project’s effects on energy use. The remaining five evaluation reports have substantial design and implementation shortcomings that compromise the validity of the reports’ findings. In addition, all six evaluation reports have other shortcomings that preclude an overall assessment of the projects’ effectiveness. (Appendixes II through VII summarize each state’s evaluation report, including its goals and measures, project assumptions, evaluation design, services provided, participant and control group selection, evaluation findings, reported limitations, and GAO’s observations on analytical problems and other shortcomings.) OCS is aware of the shortcomings in these initial evaluations and has been taking several steps to improve future project evaluations.

Nebraska’s Evaluation Report Fairly Reflected Results

The design and implementation of Nebraska’s fiscal year 1996 project allowed statistically valid conclusions to be made in the state’s evaluation report about the effects of project services on participant energy use. The project, which was carried out by a community-based organization, focused on changing the behavior of low-income households to achieve economic self-sufficiency through decreased energy use. This approach was based on the assumption that most of the target population were renters who frequently moved and that the housing available to them was generally substandard. It was assumed that the only way to reduce energy use was to change household behavior through teaching energy-efficient practices.

The evaluation report noted that REACH participants significantly decreased their use of natural gas from preproject to postproject compared with a control group of nonparticipants, whose consumption showed no statistical change. The outcome for electricity use was that while REACH participants showed, on average, no change in electricity use, nonparticipants increased electricity use significantly over the course of the project.

The evaluation report attributed the project’s successful implementation to the project design and the collection of sufficient data to the community-based organization’s experience, expertise, and knowledge of the target population, as well as to the early involvement of the evaluator in the project’s design. Cash incentives provided to the test and control groups also appeared to help limit participant attrition (leaving the project before its completion). In addition, administrative controls designed to ensure access to utility bills seem to have played a role in obtaining sufficient data for analysis.

Other Projects Faced Evaluation Challenges

The other five states’ REACH projects experienced design and implementation constraints that decreased the confidence that can be placed on the findings contained in their evaluation reports. For example, Massachusetts and Michigan did not use control groups as a means of assessing whether project participants fared differently from similar households that did not receive project benefits. California and Maryland experienced difficulties in collecting complete income or utility data and experienced client attrition at rates that call into question the likelihood that project effects could be assessed.

To some extent, these state REACH projects faced common challenges: (1) forming an adequate control/comparison group against which to compare project outcomes; (2) maintaining client participation in project activities; (3) collecting complete, accurate, and reliable data (such as client income and utility bills); and (4) adjusting (“normalizing”) energy consumption data for changes in weather.

A properly formed control group, composed of people who did not receive REACH project services but who are similar in other respects, allows a comparison of what might have happened in the absence of the project. Project designs that lack a control group can demonstrate that changes occur in such factors as energy use, but the changes cannot be attributed to the project since other factors might be responsible for the changes. Any preexisting differences between a test group and a control group could also be the cause of any observed project effects and, therefore, should be addressed in the project’s design.

Attrition of participants can reduce the amount of data available for analysis and introduce biased results. Attrition bias can occur if those remaining in the project systematically differ from those who dropped out in ways that are likely to affect the outcome of the project.

Lack of complete, reliable, and accurate data will result in imprecise or biased results. Data that are not collected for an entire heating or cooling season, for example, can lead to faulty assessments of typical energy use.

Failure to normalize energy use data makes it difficult to determine the extent to which changes in energy use result from project activities or from changes in weather that affect the need to heat or cool a residence. For example, if the postproject data on energy use reflect a warmer winter than the preproject data, a valid comparison of energy use for these two periods should adjust the data to help determine if perceived energy savings were due to warmer winter temperatures or to project activities.

In addition, other shortcomings precluded an overall assessment of the projects’ effectiveness. None of the evaluation reports addressed one of the three project-level performance goals stated in REACH legislation and program announcements: increasing energy suppliers’ contributions to reduce the energy burdens of eligible households. In addition, although the REACH program announcements have stated that an objective of every state project plan should be to measure whether its activities are more cost-effective in the long term than energy assistance payments alone, none of the project evaluation reports provided such an analysis. Finally, most of the evaluation reports did not report lessons learned or best practices that could be valuable to other projects. For example, if the report had discussed the strategies used to successfully collect energy use data, this information could have been useful to other projects. This lack of data was often cited in evaluation reports as a principal reason for not being able to measure project effectiveness.

REACH Program Is Taking Steps to Improve Evaluations

The REACH program is aware of the challenges noted above and has been taking several steps to improve future project evaluations, including providing guidance and technical assistance. First, OCS has developed guidance for project planning and evaluation in its demonstration programs, which include the REACH program. According to REACH officials, this guidance was first given to REACH grantees in 1997 or 1998. While this was too late to help the fiscal year 1996 grantees plan their evaluations and related data needs, the guidance addresses many of the evaluation limitations noted above and, if followed by state grantees, could help to improve future evaluations. For example, the guidance discusses selecting an evaluator—indicators of a good evaluator and questions to ask in an interview; using a logic model (described below) to design and evaluate a project; addressing challenges to gathering data on project participants, such as their reluctance to provide income data; using comparison groups that are as similar as possible and over- recruiting for comparison groups to allow for attrition; and developing evaluation reports that are clear and complete.

The guidance was revised in 2000 to increase its emphasis on the logic model, according to REACH program officials. A logic model identifies underlying assumptions of the project, project activities (such as energy efficiency education), immediate and intermediate outcomes expected from the activities (such as improved understanding of behaviors that can affect energy use in the home), and final project goals (such as a reduction in energy use). The REACH program encourages grant applicants to use a logic model in their applications and project planning.

Second, the REACH program provides technical assistance to grantees in their project and evaluation planning. States that receive grants are required to submit evaluation plans, which are reviewed by REACH program officials and the consulting firm that is providing technical assistance to the program. The REACH officials and consultant then discuss with the project team any improvements that are needed, and the grantee submits a revised evaluation plan. Once an acceptable evaluation plan is completed, the REACH program sends a letter to indicate approval of the plan. According to officials of the REACH program’s consulting firm, they emphasize improving the logic model to strengthen the projects’ designs and their ability to measure whether expected outcomes and goals are achieved by project activities. REACH officials also review draft evaluation reports and may ask states to revise and improve them.

Third, guidance and assistance with evaluations have also been provided through program conferences. The REACH program held conferences for evaluators in July 2000 and July 2001. At the July 2000 evaluators’ conference, the participants discussed logic models, common issues in gathering and analyzing data, evaluation methods, and lessons learned. Developing quality evaluations has also been discussed at the annual REACH program conferences, and evaluators as well as representatives of states, tribal organizations, and community-based organizations may participate in the conference. For example, the REACH conference held in January 2001 included a presentation and discussion on developing logic models, indicators, and evaluation plans. In addition, as the state grantees reported on their completed fiscal year 1997 projects, the conference participants discussed many specific evaluation issues.

While the REACH program and its consultants can assist grantees in planning their evaluations, conducting the evaluation is a responsibility shared among the states, the community-based organizations with whom the states contract, and third-party evaluators. State officials monitor and oversee the projects and the evaluation reports. Staff of the community- based organizations are responsible for collecting the data needed to complete the evaluation. The staff of the community-based organizations may need training to understand the importance of data gathering and to carry out this task effectively. Third-party evaluators help to design the evaluation, analyze data, and prepare the evaluation report. Some of the problems we noted in the six completed evaluations related to reporting, such as not clearly explaining methods of analysis or data limitations. For future REACH evaluations to improve, the states, community-based organizations, and evaluators—as well as the REACH program and its technical assistance consultants—must carry out their functions well.

HHS Has Not Yet Planned How to Communicate REACH Project Successes and Lessons Learned

OCS has not yet planned how it can best communicate information to state officials and others about the results of REACH projects, such as what approaches prove to be the most successful in meeting the home energy needs of low-income households. As the REACH program proceeds and state grantees complete additional project evaluations, more information will become available about which approaches demonstrated in REACH projects were successful and which were less successful. In addition to evaluating the results of their projects, grantees report on the processes and procedures they used to carry them out. Completed REACH projects will provide information and tools such as pamphlets, videos, and other materials for energy efficiency education; forms for collecting data from clients; and experiences and lessons learned in such areas as how to leverage contributions from energy vendors and how to coordinate among various organizations involved in energy assistance.

A comprehensive communications plan would help ensure that this information is put to good use by identifying what information to communicate, to what audiences, and by what methods. A communications plan would also estimate the amount of funding needed for communications. REACH program officials realize that publications such as summaries of successful approaches could provide information in a form more readily accessible than the individual evaluation reports. Other OCS programs have developed summary publications of best practices and have found them to be frequently requested, according to the director of OCS’ Division of Community Demonstration Programs. For instance, one publication provided lessons learned from 8 years of OCS demonstration programs (not including the REACH program, which had not yet begun at the time). As the REACH program matures and more information becomes available, OCS will have a better basis for identifying and summarizing best practices. For instance, information will become available from six project evaluations that are due during the fall of 2001 (from the state grants awarded in fiscal year 1997), and by the end of 2002, a total of 19 state projects will probably have completed their evaluations.

To date, the REACH program’s communications efforts have included developing a Web site and providing information through conferences. According to program officials, the REACH Web site, located within OCS’ Web site for its demonstration programs, is expected to become available in the summer of 2001. Currently, a pilot version of the REACH Web site is available at the Web site of the REACH program’s consulting firm. According to REACH program officials, the OCS Web site will provide the same types of information as the pilot Web site. The pilot REACH Web site includes the current OCS program announcement requesting proposals for REACH grants, summaries of past and ongoing REACH projects, summaries of the REACH program conferences, and listings of contact points for grants and evaluations.

Conferences are also used to communicate project results and lessons learned. The REACH program hosts an annual conference for certain people involved with REACH projects. According to the REACH program announcements, state grantees are expected to fund travel by the state project directors, community-based organization project directors, and chief evaluators to the annual conference in each of the 3 years of the project. Tribal organizations, which have a shorter project term, are to fund travel to one conference. At the conferences, grantees that are finishing their projects make presentations about their project approaches and results. REACH program officials also provide information geared to new grantees about program and reporting requirements. The conference has also included topical presentations; for example, the conference in January 2001 included presentations on the characteristics of good project evaluation methods, solar power, and designs for low-energy-usage homes. According to REACH officials, they and their consulting firm have also arranged for presentations on REACH projects at the annual LIHEAP conference and other energy-related conferences.

Grantees also have responsibilities for communicating about their REACH projects. OCS requires that states’ REACH project plans address disseminating results of the individual projects among LIHEAP grantees, utility companies, and others interested in increasing the self-sufficiency of the poor. States are allowed to budget up to $5,000 of each grant for dissemination purposes. Tribal organizations are allowed to budget up to $1,000 for dissemination purposes. Such state and tribal efforts are important, but they address only individual projects, not the broader compilation of learning from a number of projects over a period of years.

Conclusions

Because the REACH program lacks performance goals and measurable indicators, HHS has not defined the relationship between REACH and its parent LIHEAP program and cannot assess the program’s overall effectiveness. Considering the recent rise in home heating and cooling costs, the REACH program’s role in testing approaches to help low-income families to meet their home energy needs is an important one and should be clearly articulated. We believe that the development of performance goals and measurable indicators could provide the Congress with better information about what has been accomplished for the resources expended. Furthermore, performance goals could provide HHS’ Office of Community Services with a clearer basis for selecting grant proposals to fund. In addition, by addressing the relationship between LIHEAP and REACH in its performance plan, HHS’ Administration for Children and Families could clarify the role of the REACH program and whether it expects REACH to have an effect on the activities of LIHEAP grantees, beyond a particular REACH grant and its limited time frame.

Some of the REACH projects funded to date have included activities other than addressing clients’ home heating, home cooling, and energy payment needs, such as job skill or employment development services. The REACH program’s requests for proposals with “holistic approaches” may have been misconstrued by grant applicants and by those reviewing proposals to allow projects to use REACH funds for non-energy-related activities. However, with only about $6 million in funding annually, we question whether non-energy-related activities—including some activities typically addressed through other, much larger social service programs—are an effective use of limited REACH program funds.

While the evaluations conducted on the first year of state grants have many shortcomings that limit their usefulness in assessing project effectiveness, the REACH program recognizes the problems and has been taking steps to help improve future project evaluations. It is too soon to gauge the effectiveness of these efforts, but we noted some improvements in states’ plans for the next set of evaluations. We encourage the REACH program to continue its efforts to improve the design and methodology of evaluations because valid evaluations are vital to realizing the potential of the REACH program in testing new approaches.

With only six project evaluations completed, there is currently not enough information to reach a conclusion about the effectiveness of the REACH program. By the end of 2002, a total of 19 state projects will probably have completed their evaluations, and OCS will have awarded a total of about 80 grants, roughly half of them to states, which will eventually report on project results. While the legislation authorizing the REACH program requires this GAO review, it does not require HHS to report to the Congress as more information on project results becomes available. Developing program performance goals and measurable indicators and obtaining better data on the REACH program’s effectiveness would enable HHS to provide the Congress the information that it needs to assess the program. Furthermore, the legislation did not specify an ending date for the REACH program, as is sometimes the case for demonstration programs. It may be appropriate to reassess the REACH program during 2003 and consider whether and how long it should continue.

Finally, the lack of a comprehensive plan for communicating the results of REACH projects and fostering the further use of effective approaches could limit the impact of the REACH program. Without well-planned and adequately funded communications, the results of REACH projects may fail to have an impact on LIHEAP and other programs that provide energy assistance to low-income families. Summaries reporting on best practices could be more effective as communication tools than the state evaluation reports themselves for reaching the state, federal, and community officials involved in addressing the home energy needs of low-income households. To the extent that OCS shares such information among the state, tribal, and community-based organizations that provide energy assistance, the organizations can make better-informed decisions about replicating successful approaches and avoiding problematic ones. Furthermore, well- informed organizations can avoid “reinventing the wheel” by, for instance, not investing in developing energy education materials that may already be available. In light of additional project evaluations and other information that will become available over the next several years, OCS needs to plan for its communications efforts.

Recommendations for Executive Action

To better target the use of the limited resources of the REACH program and provide for reporting on program performance, we recommend that the Secretary of HHS direct the Administration for Children and Families and its Office of Community Services to develop program performance goals for REACH that are objective, measurable, and quantifiable; address the relationship between the REACH and LIHEAP programs in its ensure that REACH funds are used for activities directly related to the home energy (heating and cooling) needs of low-income households.

To ensure that the results of REACH projects are effectively communicated to the government agencies and private organizations involved in addressing low-income households’ energy needs, we recommend that the Secretary direct the Office of Community Services to develop a communication plan for the REACH program describing intended audiences, types of information to be communicated, communication methods appropriate to the intended audiences, and the funding needed.

Matters for Congressional Consideration

With HHS developing information for performance reporting and obtaining additional project evaluations, the Congress may want to consider requiring HHS to report on REACH program effectiveness and project results in several years, after the projects funded in the first 3 years of the program have completed their evaluations by the end of 2002. Once HHS has reported, the Congress may also wish to consider whether the REACH program should continue indefinitely or whether the program should have an end date after a sufficient number of demonstration projects.

Agency Comments and Our Response

The Department of Health and Human Services provided written comments on a draft of this report. These comments are reprinted in appendix VIII, along with our responses. HHS generally agreed with our recommendations.

However, while HHS agreed with our third recommendation that it ensure that REACH funds are used for activities directly related to the home energy needs of low-income households, it made two comments related to this recommendation. First, HHS disagreed with our assessment that language in its program announcements on holistic program strategies may have been misconstrued to encourage non-energy-related activities. We continue to believe that HHS should review the language of its program announcement, as it plans to do, to ensure that the REACH program does not fund activities that are not directly related to the home energy needs of low-income households. The language in the program announcement is used as criteria for reviewing and selecting grant proposals, as well as for providing guidance to applicants for grants. Our recommendation concerns only REACH program funds; grantees would not be precluded from using other sources of funds for such activities or from coordinating with other social service efforts. Second, HHS stated that the term “residential energy” is understood to include all household energy use, not just home heating and cooling. However, we note that the authorizing legislation does not define the term “residential energy.” Therefore, we believe that HHS should apply the definition of home energy in the authorizing legislation—namely, a source of heating or cooling in residential dwellings. We have changed the wording of our recommendation to make this more clear.

HHS suggested that our matters for congressional consideration include the possibility of expanding the REACH program, as well as the options of continuing it or setting an end date for the program. We have not incorporated this comment because the effectiveness of the REACH program has not yet been determined and because we believe that HHS should be required to report to the Congress in 2003 on the REACH program’s effectiveness.

HHS also made a number of technical comments, which we have incorporated as appropriate.

Scope and Methodology

We reviewed all of the REACH grants that have been awarded since the program was first funded in fiscal year 1996. However, in reviewing project evaluations, we focused only on states’ projects, because tribal organizations are not required to do project evaluations. We also reviewed documents about the LIHEAP program and interviewed LIHEAP officials in order to better understand the context and purpose of the REACH program. Specific actions that we took to accomplish each of our objectives are listed as follows: To obtain information about grant amounts, recipients, and project activities, we reviewed REACH program summary documents and grant proposals; interviewed state officials responsible for most of the state grants awarded in the first 2 years of the program; and attended the REACH program’s annual conference, where we heard presentations by grantees.

To obtain information about REACH program goals, objectives, and performance measures, we reviewed the authorizing legislation, HHS performance plans, requirements and guidance related to the Government Performance and Results Act of 1993, and REACH program announcements that request grant proposals. We also interviewed program officials.

To analyze the results of state evaluations, we reviewed the project design and implementation of the six completed evaluations. We assessed the adequacy of key aspects of project design and implementation since these factors determine the confidence that can be placed on an evaluation’s findings. By determining whether the evaluation reports contained critical methodological flaws, we ascertained whether the reported findings were so qualified as to preclude their use in assessing the REACH project’s effectiveness. In addition, we summarized these evaluations, contacted the state officials responsible for these six projects to discuss the evaluations, and discussed efforts to improve evaluations with REACH program officials and the program’s consultants.

To identify the REACH program’s communications efforts, we interviewed program officials and reviewed the material made available on the REACH program’s pilot Web site and through REACH program conferences. We did not assess the communications efforts of REACH grantees.

We conducted our review from December 2000 through July 2001 in accordance with generally accepted government auditing standards.

We are sending copies of this report to the Secretary of HHS, the Director of the Office of Management and Budget, appropriate congressional committees, and other interested parties. We will also make copies available to others upon request.

If you or your staff have any questions about this report, please call me on (202) 512-3841. Other key contributors to this report are listed in appendix IX.

Appendix I: REACH Project Activities

Table 4 summarizes the activities planned for the 53 Residential Energy Assistance Challenge Option (REACH) projects that have been funded since the beginning of the program: 28 state and 25 tribal organization projects. (The fiscal year 1999 North Carolina grant is not included in the table because it was being redesigned at the time of our review because the state had not deregulated utilities as had been expected when the project was originally being designed.) Most project plans included multiple activities. Several activities used by only one or a few projects are not included below.

Appendix II: Review of California’s REACH Project Evaluation Report

Goals and Measures

The primary goal of the California REACH project was to help low-income households reduce their energy use by providing energy conservation and other services. Specific project goals follow: Dwellings would become more energy-efficient. Health and safety risks would be mitigated. Families would become knowledgeable about energy use and conservation.

Families would make and keep energy conservation goals. Families would reach a stable level of energy consumption. Families would reduce their demand for Low-Income Home Energy Assistance Program (LIHEAP) payment assistance.

Eligible recipients would enroll in utility-supported rate discount programs.

Families that needed additional help in other areas would receive assistance.

Families would become stable and develop problem solving skills and coping abilities.

Project Design and Implementation

Project Assumptions

The evaluation report did not clearly identify major project assumptions, other than recognizing an expected participant attrition rate of 20 to 25 percent. To compensate for expected attrition, the project design reflected plans for increasing the participant base.

Evaluation Design

The project was designed to determine which combinations of services (described below) were the most effective in reducing energy use of low- income households. The project was designed to use test groups and control group comparisons. It was to use six test groups (one for each of six combinations of services), with corresponding subgroups located at three separate locations. Services were to be provided to REACH participants by four different community-based organizations. A seventh group—the control group—was not to receive services other than energy assistance payments.

Utility bill data and other information were to be collected for all seven groups before and after project implementation. The results of the REACH groups were to be compared and evaluated.

Services to Be Provided

The evaluation report identified five services to be provided by community-based organizations through case management: (1) basic and (2) enhanced energy conservation education, (3) basic and (4) enhanced home weatherizing, and (5) family intervention (including referral to other agencies when needed). The control group was to receive only energy assistance payments.

In addition to the above services that would be evaluated, the community- based organization was to provide a variety of other services, such as outreach, eligibility determinations, residential assessments, and family assessments.

REACH Participant and Control Group Selection

Eligible project participants were households that received energy assistance payments, had lived at the same residence for the past year, and were not expected to move in the current year. In addition, households were required to have metered energy—electricity, gas, or propane—for their individual residence and could not have received previous home weatherizing services. Project participation was further restricted to households that spoke limited English, had a high energy burden, received public assistance for children, and/or were unable to pay current utility bills. The evaluation report stated that participant eligibility was verified using the standard process for determining eligibility for energy assistance payments.

The evaluation report indicated that households were to have been randomly selected and assigned to the various test and control groups on the basis of the services they were to receive.

Evaluation Findings and Reported Limitations

Findings

According to the evaluation report, there was no statistically significant reduction in the use or cost of energy among the test groups. The report found that the average monthly household energy use and cost actually increased slightly during the project. Overall, average monthly electrical and natural gas use was also found to have increased. The results of the individual test groups, however, were reported to be mixed. For example, 6 groups reduced their electrical use, 7 groups reduced their natural gas use, and 10 groups reduced their total energy cost.

The evaluation report also concluded that data were insufficient to determine whether the project had helped increase the regularity of utility bill payments. In addition, the evaluation report noted that according to the results of an energy awareness survey, participant knowledge of energy conservation measures had increased slightly. The report did not specifically assess the project’s impact on the other eight project goals.

Reported Limitations

Data collection problems were identified by the evaluation report as a major problem that could affect conclusions about the project’s outcomes. None of the community-based organizations provided all of the data needed for the evaluation. One community-based organization’s work was not included in the evaluation because it did not collect any follow-up data. The information provided by the other three community-based organizations was incomplete because they either lost or did not completely collect the data. The evaluation report also noted that the project did not collect data on the regularity of utility bill payments, and the report provided no reason for this.

The evaluation report attributed these data collection problems to a lack of controls over the data collection activities of the service providers— noting that procedures for data collection were limited, and the service providers had no incentives to collect follow-up data. The report also said that community-based organization staff turnover and the lack of training in data collection requirements contributed to the problem and recommended that in the future final award payments be withheld until all the data are provided.

In addition, the evaluation report noted that energy use data provided by the community-based organization was not adjusted for changes in weather and concluded that this could have significantly limited the evaluation’s findings and conclusions.

Our Observations

The effectiveness of the California REACH project cannot be determined on the basis of the information presented in the project evaluation report because of limitations in the evaluation’s implementation and analysis. The report also provided insufficient information to determine the validity of the findings and did not provide other information needed to assess the project’s overall effectiveness.

Analytical Problems

Several key issues compromised this REACH evaluation effort. For example, the evaluation did not compare the test groups with the control group as called for in the project design. Without a control group there is no assurance that changes found in the test groups did not result from factors other than the combination of services provided. Moreover, while the report noted that the project’s impact on utility bill payments could not be assessed because of insufficient data, it did not identify or explain the effect of attrition and the loss of data on the statistical validity of its findings. The substantial household attrition that occurred and the resulting loss of data could have further compromised the validity of the project results because households that completed the project may differ from those that dropped out in ways likely to affect the outcome.

In addition, as noted in the evaluation report, energy use data were not adjusted for weather conditions, thereby further compromising the statistical validity of the findings on energy use. For example, if temperatures were significantly colder in winter or warmer in summer, a corresponding increase in energy use would also be expected. If this occurred during the period after energy conservation services were provided, these services could have been responsible for stabilizing household energy use or reducing the amount of energy that might otherwise have been consumed. Such a finding could not have been reported because the effects of changes in weather had not been taken into account when measuring results.

Other Shortcomings

In addition to not including all of information necessary to assess the validity of its findings, the evaluation report did not provide other important information needed to assess the overall performance of the project. For example, the report did not address the project’s performance regarding one of the project performance goals identified in the authorizing legislation and the Department of Health and Human Services (HHS) program announcement—increasing the contributions energy suppliers make to reducing the energy burden of eligible households. Finally, the evaluation report did not provide a complete discussion of the project design, including information on project assumptions, hypotheses to be tested in terms of measurable objectives, or lessons learned.

Appendix III: Review of Maryland’s REACH Project Evaluation Report

Goals and Measures

The goals of the Maryland REACH project were to help participants (1) reduce their electric bills, (2) improve their budgeting skills, and (3) increase their education or work opportunities with the ultimate goal of preparing them for self-sufficiency once the project was completed.

Project Design and Implementation

Project Assumptions

The evaluation report did not clearly identify major project assumptions.

Evaluation Design

The project was designed to use two groups of low-income households—a test group and a control group—to assess of services provided. Data on participants in the test group were to be collected (1) from participant surveys—before and after the project—and (2) from utility statements for the year before the project, when available, and for the year during project implementation. Participant data before project implementation were to be compared with participant data collected during project implementation so that the effect of the services could be evaluated. Utility bill statements were also to be obtained for the control group receiving financial assistance for energy bills, so that a comparison could be made between the test group and the control group. The evaluation report also noted that energy use data were to be compared by “seasons” to control for the effect of weather.

Services to Be Provided

Services were to be provided by a community-based organization using a case management approach. Each household was to receive four caseworker contacts. During the first contact the caseworker would perform an initial assessment of household energy use, income, education, and occupation. During the second contact, the family assessment was to be updated. The caseworker would also obtain an application for energy assistance from the household and provide energy conservation counseling. During additional contacts, the family assessment would continue to be updated and energy conservation counseling provided. Referrals to public assistance programs for help with rent or mortgage payments, utility bills, home weatherizing, or job training would be made when necessary. Other services would include a newsletter and workshops on energy conservation. (The newsletter was eventually cancelled because it was not being read, and the workshops were cancelled because of low attendance.)

In addition, program participants were often referred to other agencies providing various kinds of assistance, and more than one referral could be made for each participant. There were 152 referrals made for a variety of social assistance services, including employment and education. Other referrals were made for home weatherization assistance (86), such as instruction on energy conservation; energy assistance payments (73); and mortgage or rent assistance (20).

REACH Participant and Control Group Selection

The local energy assistance office initially identified eligible households, and later participants were recruited from energy assistance applicants and referrals from other agencies. A questionnaire was used to establish eligibility. Over a 25-month period, a total of 124 participants entered the program and 92 participants were dropped from the program, primarily because they could not be contacted.

The control group was selected from households receiving energy assistance payments. Its members did not receive the REACH project services noted above. The evaluation report noted that the size of the control group was adjusted monthly to compensate for changes in the number of project participants.

Evaluation Findings and Reported Limitations

Findings

The evaluation report compared participant pretest and posttest data on energy use and found that while households reduced their use of electricity, the overall reduction was not statistically significant. However, the report noted that a statistically significant reduction was seen in winter. The average monthly energy use was reported to have decreased by 61 kilowatt hours, from 1,275 kilowatt hours to 1,214 kilowatt hours. The average cost of household monthly utility bills was reported to have decreased by $16, from $137 to $121.

The evaluation report also found that the use of electricity by the control group had increased by 55 kilowatt hours, and that the difference between the two groups was statistically significant. As a result, the evaluation report concluded that participating households were able to stabilize their use of electricity while the control group households were not. The report did not assign a dollar cost savings to the difference between the groups.

Utility bills were also used to assess the impact of program services on arrearages. According to the evaluation report, the number of late payments and termination notices decreased for households participating in the project, but neither decrease was statistically significant. Although the number of termination notices significantly increased for the control group, the number of actual terminations decreased, while the number of actual terminations increased for participating households. According to the evaluation report, this difference was statistically significant and suggested that the project goal of improving budgeting skills was not accomplished.

The evaluation report also analyzed monthly utility bill payments and balances carried over from month to month to assess the ability of households to manage their personal finances. The evaluation report found a slight increase in electricity bill payments that was not statistically significant but found a reduction in monthly balances that was statistically significant. Neither difference was found to be significant when compared with the control group.

On the basis of these findings, the evaluation report concluded that the project achieved the following: Households received $60,000 in assistance for energy use and rent or mortgage assistance.

A total of 363 referrals were made to agencies providing assistance. Indications implied that households reduced or stabilized their energy use. Households received fewer termination notices. Households reported less difficulty in paying utility bills and in conserving energy.

The report, however, also concluded that the project did not meet some expectations: Over half of the households entering the project were dropped because they could not be contacted later in the project.

Newsletters and workshops were discontinued because of a lack of interest.

Participants did not decrease the amount of arrearages in their utility bills. A decrease was not achieved in the termination of utility services.

Reported Limitations

According to the evaluation report, the attrition rate of households resulted in difficulty obtaining adequate information for the project evaluation. In addition, survey responses were obtained in two ways: 29 households completed surveys, and 10 additional responses were obtained through telephone interviews.

Our Observations

The effectiveness of the Maryland REACH project cannot be determined on the basis of information presented in the project evaluation report. The assessment has limitations that affect the validity of conclusions that can be made regarding reducing energy use and other goals. For example, the report does not present information on the comparability of the test and control groups. In addition, there was a lack of information on the number of months of utility data obtained for REACH and control group members. The report did not address the possible effects of participant attrition on the comparability of these groups. Moreover, energy use data were not adjusted for the effects of weather. The report also had other shortcomings that precluded an overall assessment of the project’s effectiveness.

Analytical Problems

The report did not provide sufficient information on the test and control groups to allow an assessment of their comparability. Discussion of factors that could affect comparability of the test and control group was incomplete in several respects. For example, the report did not provide sufficient information on the number of months of utility bills available for both groups. Estimates based on data for a few months can be more easily influenced by factors unrelated to the project services than those based on data for an entire year. Although the report stated there was a statistically significant difference in energy use between the groups for the year, it did not discuss the possible reasons a statistically significant difference was not observed in winter when the control group also showed a decrease in energy use.

The validity of the conclusions regarding energy use could also be compromised by attrition in the test and control groups because households completing the project may not be representative of the target population as a whole. The report, however, did not identify the effect of attrition and the resulting loss of data on the validity of its findings.

In addition, energy use data were not adjusted for possible changes in the weather from preproject implementation to postproject completion. The decrease in winter energy use noted in the report could have occurred because of a relatively milder winter during the project. The seasonal comparison of the test and control groups actually showed no significant statistical difference between the groups during the project.

Other Shortcomings

The evaluation report did not provide other important information needed to assess the overall effectiveness of the project. For example, the report did not address the project’s performance in terms of the performance goal stated in the authorizing legislation and the HHS program announcement—increasing energy supplier contributions to reducing the energy burden of eligible households. The evaluation report also did not provide a complete discussion of the project design, including information on the project’s assumptions, hypotheses to be tested in terms of measurable objectives, or lessons learned.

Appendix IV: Review of Massachusetts’ REACH Project Evaluation Report

Goals and Measures

The overall goal of the Massachusetts REACH project was to reduce homelessness by fostering self-sufficiency. In that context, the grant application identified three main project goals: reducing household energy costs, increasing the regularity of household energy bill payments, and increasing contributions by energy suppliers toward reducing households’ energy burden.

The evaluation report also identified five objectives for the REACH project participants: Increase awareness of energy consumption. Increase awareness of the ability to shop for the least expensive energy supplier.

Learn budgeting. Make regular payments to energy suppliers. Negotiate arrearage forgiveness with energy service providers.

Project Design and Implementation

Project Assumptions

The evaluation report did not—except in the calculation of energy burden—clearly identify major project assumptions. Assumptions related to the calculation of household energy burden were discussed in some detail (see below).

Evaluation Design

The project was designed as a single group of REACH participants receiving a variety of services. The study stated that financial constraints prevented using a control group. Participant data were to be collected both before and after the project using a standard data collection instrument, questionnaires, or both. Utility bills were to be obtained from energy suppliers. The information collected would then be used to assess changes in household self-sufficiency, knowledge of energy use, the effect of specific interventions, and changes in energy burden by comparing pre- REACH data with post-REACH data.

Services to Be Provided

The project focused on assisting households through case management services. Caseworkers were to provide financial counseling and energy education and make referrals to other public assistance agencies for services dealing with home energy conservation, employment, training, day care, and improving language skills. Energy assistance payments were to be the only immediate financial assistance provided—although 148 of 164 households (over 90 percent) reported receiving other public assistance. Caseworkers also were to facilitate household access to an arrearage forgiveness program offered by a major utility company. The evaluation report noted that, during implementation, caseworkers provided intake to over 460 households that were either homeless or at risk of being homeless, and maintained contact with over 350 of these households.

REACH Participant and Control Group Selection

Project participants were selected from low-income households eligible to receive energy assistance payments, and caseworkers recruited clients from assistance programs for the homeless. The report did not describe the selection procedure. As noted above, the project did not use a control group.

Evaluation Findings and Reported Limitations

Findings

According to the evaluation report, project results were difficult to quantify, and the true measure of success was the range of services provided in response to participants’ needs. However, information gathered by case managers showed no change in self-sufficiency by project participants. The evaluation report addressed project performance in terms of the three main project goals.

The evaluation report calculated the energy burden to households using a formula developed to measure energy costs, or debt, as a portion of income. On the basis of the results of the calculation, the evaluation report found that the energy burden—the proportion of household income represented by energy-related debt—decreased during the project from $345 (105 percent) to $252 (64 percent). However, the evaluation report noted that complete data—both pre- and postproject data for the same households—were available for only five project participants. The report also stressed that the calculation was based on the unlikely assumption that income would remain the same over time and on unreliable energy bill data.

In addition, the evaluation report stated that, although the goal of reducing energy costs appeared to have been achieved, the lack of posttest follow- up data prevented a determination of whether the objectives of increased awareness of energy use and energy supplier options were accomplished.

Increasing the regularity of household energy bill payments The evaluation report assumed that participant utility bills would be available to provide a record of payments. However, only electric bills were available, limiting payment analysis to the 22 households using electricity to heat their homes. Of these households, follow-up data existed for only two, according to the evaluation report, making assessment of the outcome impossible.

Increasing contributions by energy providers toward reducing the households’ energy burden The evaluation report included information provided by the project staff on the arrearage forgiveness program offered by a major public utility company to households receiving energy assistance payments and indicated that caseworkers processed 45 applications from project participants for this program.

The evaluation report also identified lessons learned from experience with the project and from interviews with case managers. Lessons learned included the need for additional guidance from HHS on the conduct of evaluations, involving evaluators in project design, obtaining access to utility bills from utility companies, and thorough training for project staff involved in data collection.

Reported Limitations

The evaluation report noted that in many cases data were either unreliable or missing, making an assessment of project outcomes impossible. The report attributed this data problem to several factors, including the transient nature of participating households, lack of training for caseworkers, and failure to involve the evaluator early in the program design.

The report also cited shortcomings in the assumptions used to calculate and analyze household energy burden that adversely affected the assessment of project performance. In addition, the report stated that data records were sometimes constructed by case managers according to their memory of events occurring weeks or even months earlier. The evaluation report also noted that a scale used to measure self-sufficiency was never tested for validity or reliability.

Our Observations

The effectiveness of the Massachusetts REACH project cannot be determined using the information presented in the project evaluation report. Performance could not be assessed and reported because of the lack of data. Data were missing primarily because of attrition— participants leaving the project before completion. In addition, the project used a pretest/posttest project design that attempted to measure the impact of delivering a variety of services but did not include a control group. A comparison group is necessary to help ensure that the effects observed resulted from services provided and not other variables and to evaluate the effectiveness of the services compared with energy assistance payments alone. The report had additional shortcomings that would preclude an overall assessment of the project’s effectiveness.

On the plus side, the evaluation report discussed project performance in terms of the performance goals set out in the authorizing legislation and the REACH program announcement. The report also discussed the difficulty encountered in measuring project performance because of data collection problems and described the limits on statistical analysis due to insufficient data. The report also identified issues relating to the calculation of energy burden that could adversely affect the analysis. In addition, the report provided a description of lessons learned that could benefit the design of future projects.

Analytical Problems

Statistical estimates of project performance could not be made because of the lack of data. Problems with households leaving the project before completion, accompanied by the inability to obtain utility bills from all energy providers, resulted in insufficient information to assess the effect of services provided on either household energy use or the regularity of utility payments. Attrition (leaving the project before completion) raises the potential for biased results if those remaining in the project systematically differ from those who drop out in ways that are likely to affect the outcome. Concerns regarding data quality also arise from the construction of records by case managers on the basis of memory months after events occurred.

However, even if sufficient data had been available for statistical analysis, failure to include a control group in the project design would have limited conclusions regarding the effect of project services on household energy use and changes in the regularity of utility bill payments. When a control group is used, the credibility of the identified cause of observed outcomes is generally greater because external factors have been taken into account.

Additional analytical problems identified in the evaluation report include assumptions implicit in the energy burden calculation and the use of a scale to measure changes in self-sufficiency that was not tested for either reliability or validity. These shortcomings further limit findings or conclusions that can be made about the project.

Other Shortcomings

While the evaluation report clearly identified certain problems encountered in assessing performance and the resulting limitations of its findings, it did not provide or address the lack of other information needed to assess overall project performance. For example, the report did not address the project’s performance in terms of the performance goal stated in the authorizing legislation and the HHS program announcement— increasing energy suppliers’ contributions to reducing the energy burden of eligible households. The report noted that the information needed to assess this goal would be provided by the project staff in a separate report.

Appendix V: Review of Michigan’s REACH Project Evaluation Report

Goals and Measures

The Michigan REACH project was comprised of two subprojects with different goals. One subproject’s primary goal was the reduction of household energy use. The other subproject’s primary goal was to educate households on energy deregulation, provide consumer advocacy, and explore the feasibility of bulk energy purchases.

The evaluation report identified eight performance measures that were to be used to evaluate the impact of services on households: reducing utility bills, increasing the regularity of utility bill payments, increasing earned income, reducing reliance on energy assistance programs, increasing knowledge of energy conservation methods, increasing understanding of utility bills, reducing energy-related safety problems, and maintaining or increasing the availability of affordable housing.

Project Design and Implementation

Project Assumptions

The evaluation report did not clearly identify major project assumptions.

Evaluation Design

The project was divided into two subprojects with different goals for different providers of education services. One subproject was to focus on improving energy conservation and developing life skills, such as budgeting. The other subproject was to focus on increasing knowledge about energy deregulation. Both subprojects, however, would use a pre- and posttest evaluation design—collecting data before and after project implementation—to evaluate the effectiveness of the services provided. A control group that did not receive services was to be created from eligible households to permit comparison with households that received services.

In the energy conservation and life skills education subproject, household energy use data would be collected by seven service providers—located in different areas—for the year before and the year after services were provided. The evaluation report indicated that these data were to come from utility bills obtained from energy suppliers. In addition, participants would also be mailed a questionnaire after project completion to obtain their views on the project.

In the energy deregulation education subproject, households would complete a questionnaire before and after project implementation to determine changes in their knowledge about energy deregulation.

Services to Be Provided

The energy conservation and life skills education subproject was designed to help low-income households develop plans identifying changes directed at achieving an immediate reduction in energy use. Education would be provided through workshops. Other workshops would also be provided that addressed broader issues, such as preparing household budgets and finding employment. This subproject initially targeted 1,100 households at various locations throughout the state.

The other subproject was designed to prepare low-income households for the effect of utility deregulation through education and to serve as a consumer advocate. This subproject was to consist of placing informational articles on energy deregulation in newsletters and in pamphlets that were delivered to participating households.

REACH Participant and Control Group Selection

The evaluation report stated that the subprojects had a target population of households at or below 150 percent of the federal poverty level. The report also stated that households in the deregulation education subproject were participants in the Head Start Program. The report did not further explain the selection criteria or process other than to note that the control group would be selected from eligible households.

Evaluation Findings and Reported Limitations

Findings

According to the evaluation report, there was no significant difference before and after the project in natural gas use by participants at the three locations for which data were available. However, on the basis of the results of two locations reporting pre- and postproject data, the report found the project effect on electricity use mixed: a significant reduction was reported at one location but not at the other.

On the basis of participant responses to the postproject survey, the evaluation assessed the results, or outcomes, of four performance measures in qualitative terms: Increased regularity of utility bill payments—most participants described this workshop education component as helpful.

Reduced reliance on energy assistance programs—about one-third of the participants indicated that their use of emergency energy assistance had decreased. Increased knowledge of energy conservation methods—the majority of the participants felt that the energy workshop was helpful. Increased understanding of utility bills—most participants found the educational information helpful.

Also, on the basis of participant responses to the surveys, before and after the project, the evaluation report concluded that there was no change in household understanding of utility deregulation.

Reported Limitations

The evaluation report noted that the data results cited on natural gas use should be used with caution because of the small number of households for which data were available. Three locations provided data on natural gas use, while two locations provided data on electricity use. Data on gas use were adjusted for weather, but data on electricity use were not adjusted. The report also noted that the control group was not valid because some households may have also received weatherization services. In addition, the evaluation report stated that changes were made to the project in the second year, but no postproject data were available to include in the final evaluation report.

Regarding the survey of participants, the evaluation report noted that the assessment of perceptions of energy conservation and life skills workshops was based on 299 surveys returned out of 740 surveys mailed— a response rate of 40 percent.

According to the evaluation report, it was not possible to measure the effect of the project on earned income, reductions in housing safety problems, and availability of affordable housing.

Our Observations

The effectiveness of the Michigan REACH project cannot be determined because of limitations in design and implementation. For example, the lack of data on participant energy use, along with other analytical problems, such as the lack of a control group and failure to adjust energy use data for weather conditions, compromised the validity of the findings on energy use. As a result, a valid assessment of the effect of project activities on energy use or regularity of participant utility bill payments cannot be made. In addition, there were limited months available on which to base pre- and postintervention estimates, ranging from 2 to 4 months. Moreover, the indication of a decrease in electric usage for one location was based on data that were not adjusted for effects of changes in weather. In addition, the report had other shortcomings that precluded an assessment of the project’s overall effectiveness.

Analytical Problems

The first Michigan subproject focused on changing participant behavior through education. The inability to obtain quality data on participants’ energy use from the service providers and the local energy supplier— especially data on postintervention energy use—meant that conclusions were based on relatively few records. For example, postintervention data were not available on gas use for four of the seven locations and not available for electricity use for five locations. Moreover, the postintervention data did not cover an entire year—data were available for only 2 to 4 months.

Overall attrition was also an issue. The evaluation report states that 1,028 participants were served by seven locations from October 1997 to September 1999 but that pre- or postintervention fuel consumption data were usable for 570 participants. Matched pre- and postintervention fuel data were available for 93 participants using natural gas and 116 participants using electricity. These data come from three of the seven locations. These limited data restrict inferences that could be drawn from the originally intended locations.

Additionally, attrition occurred within the locations for which pre- and postintervention data were available. One of the three locations served 200 participants and obtained 34 useable natural gas records and 43 electric records. The second location served 156 participants and obtained 40 useable natural gas records and 73 electric records. The third location served 150 participants and obtained 19 useable natural gas records and no usable electricity data. This level of attrition raises the issue of bias because participants that remain in the project may differ from those dropping out in ways that are likely to affect the outcome. The high attrition rate also reduces the precision of any statistical estimate of change in fuel usage.

Even if sufficient data had been available for statistical analysis, not having a valid control group for comparison greatly limited the ability to make a valid assessment of project effects. Similarly, failure to adjust energy use data for weather conditions prevented a valid assessment of participant energy use.

Other Shortcomings

In addition to not including all of the information needed to assess the statistical analysis and validity of its findings, the evaluation report did not provide other important information needed to assess the project’s overall effectiveness. For example, the report did not address the project’s performance in terms of the project performance goal provided in the authorizing legislation and the HHS program announcement: increasing energy suppliers’ contributions to reducing the energy burden of eligible households. The report also did not provide a complete discussion of the project design, including critical project assumptions, hypotheses to be tested in terms of measurable objectives, or lessons learned from implementing the design.

Appendix VI: Review of Nebraska’s REACH Project Evaluation Report

Goals and Measures

The primary goal of the Nebraska REACH project was to increase the economic self-sufficiency of low-income families by increasing household energy efficiency. The project sought to decrease participants’ household utility bills by reducing energy use. A second project goal was to increase participants’ knowledge of energy conservation practices and personal finance. Additional project goals included increasing the environmental comfort and health and safety of the participating households.

Project Design and Implementation

Project Assumptions

The Nebraska REACH project was to focus on changing the behavior of the target population—low-income households—as the best way of achieving project goals. This approach was based on the assumptions that low-income households were most often renters who frequently moved and that the housing available to them was generally substandard. Given these premises, it was assumed that the only way to reduce energy use in this population would be to change household behavior through teaching energy-efficient practices. It was also assumed that it would have been too expensive to weatherize substandard housing, and upgrading the structure would have benefited the household only temporarily, until its next move.

Evaluation Design

According to the evaluation report, the Nebraska REACH project provided a unique opportunity to evaluate a social services project using a “true experimental design.” The project was designed to use two groups of low- income households—a test group and a control group—to evaluate the primary hypothesis that changing behavior could reduce energy use. Data collected on the two groups, both before and after project implementation, would be analyzed using a standard computer program—the Princeton Scorekeeping Method (PRISM) software—to determine the statistical significance of the results. The PRISM software uses data from monthly utility bills to produce a weather-adjusted index of energy use for both the test and control groups. The program also compares the differences in energy use among households in the test group and between households in the test group and control group. Members of both the test group and the control group would be required to sign forms authorizing the release of their utility bills to the project.

In addition, knowledge of energy conservation practices and personal financial management obtained from the workshops would be measured using pre- and postknowledge tests and a third test given 6 months after project completion.

Services to Be Provided

Households were to receive four related services: (1) a home energy audit, (2) instruction in energy conservation practices and personal financial management, (3) case management, and (4) provision of basic home weatherizing materials. Home energy audits would include an interview and inspection by a certified energy specialist to assess the household’s energy efficiency and form the basis of the action plan. Workshops providing instruction on energy conservation and personal finance would be available to participants each month, and training manuals would be distributed providing detailed instruction on these issues. Case management would include developing a household action plan, preparing family assessments, and conducting monthly progress reviews during home visits. Materials such as weather stripping, along with instruction on installation, would also be provided. In addition to these services, project participants would receive between $200 and $350 in vouchers to pay utility bills. Control group members would receive $50 in vouchers for utility bills.

REACH Participant and Control Group Selection

Potential applicants were recruited through advertisements and social service agency referrals. Case managers determined project eligibility using four criteria: participants had to (1) have income at or below the poverty level, (2) have resided at their current address for at least 1 year, (3) not be planning to move, and (4) need assistance in paying utility bills. The control group was randomly selected from every third eligible applicant. There were no significant differences between the two groups in either education or other demographic characteristics. During implementation, the project provided services and cash assistance to 439 participants and cash assistance and goods to 202 control group members.

Evaluation Findings and Reported Limitations

Findings

Separate analyses were done for natural gas and electricity use using the PRISM computer program. PRISM provided normalized annual consumption data (pre- and postproject) and normalized annual savings separately for both gas and electricity.

The PRISM analysis indicated a statistically significant decreased use of natural gas by REACH project participants compared with those in the control group. Program participants had normalized annual savings of $48.12 per household more than the control group—or a savings of 122 hundred cubic feet.

The result of the PRISM analysis for electricity use revealed a different use pattern. Although the analysis showed no change in electricity use by project participants, the analysis revealed an annual increase in electricity use of 2,716 kilowatt hours by the control group. The evaluation report stated that this effect was consistent with the hypothesis that electricity use would be lower for project participants—an annual savings of $149.38 per household.

On the basis of these findings, the evaluation report concluded that project participants had achieved greater economic self-sufficiency through a reduction in utility costs—an average of $197.50 each year—compared with the nonparticipant control group. However, the report also noted that an actual reduction in utility use for project participants was confirmed only for natural gas and not for electricity.

In addition, given the results of the knowledge tests administered to participants, the report concluded that the instruction provided during the project resulted in a significant increase in knowledge about energy conservation practices and personal financial management.

The evaluation report attributed the project’s ability to implement the project design as planned and to obtain sufficient data to perform the statistical analysis to the community-based organization’s experience, expertise, and knowledge of the target population and to the early involvement of the evaluator in the project’s design. Cash incentives provided to the test and control groups also appeared to have reduced participant attrition. In addition, administrative controls designed to ensure access to utility bills seemed to have played a role in obtaining sufficient data for analysis.

Reported Limitations

The evaluation report noted several factors that greatly reduced the number of households whose energy use data could be used in the PRISM analysis. For example, frequent changes in residence eliminated many households from the analysis. In addition, frequent changes in household size and composition, along with utility shutoffs, made much of the data too unreliable for use in the PRISM analyses; only a small percentage of the database was considered appropriate for analysis.

According to the evaluation report, analysis was limited to determining the overall effect of the project, not the effect of specific services or combination of services (i.e., identifying which services were most effective in achieving a reduction in energy use.)

Our Observations

The effectiveness of the Nebraska REACH project can be assessed with some qualification, given the information presented in the evaluation report. The project’s design and implementation allowed statistically valid conclusions to be made about the effect of project services on participant energy use. The report did not address the two other performance goals of reducing utility bill arrearages and increasing contributions by energy suppliers and did not provide other information needed to fully assess the project’s overall effectiveness.

Analytical Problems

The evaluation report did not provide an assessment of potential bias due to attrition, which may result if those remaining in the project systematically differ from those who dropped out, and its possible effect on the analyses and conclusions.

Other Shortcomings

Although the evaluation report provided a statistical analysis regarding the project’s main hypothesis, the report did not address two of the project performance goals identified in the authorizing legislation and the HHS program announcement. Specifically, the report did not address the performance goal of increasing the regularity of utility bill payments or the performance goal of increasing contributions by energy suppliers. Finally, the evaluation report also did not state the hypotheses in terms of measurable objectives. The report did not provide a discussion of lessons learned or best practices that could be valuable to other projects. For example, if the report had discussed the strategies used to successfully collect energy use data, this information could have been useful to other projects. This lack of information is often cited as a principle reason for not being able to measure project performance.

Appendix VII: Review of Oregon’s REACH Project Evaluation Report

Goals and Measures

The primary goal of the Oregon REACH project was to help low-income households manage their energy costs more effectively. The evaluation report identifies four related project goals: reducing household energy use; reducing household energy cost; increasing the regularity of utility payments, reducing arrearages in utility bills, or both; and eliminating health and safety risks related to energy use.

The long-term objectives of the project were described as sustained reduced energy use, overall improvement in economic self-sufficiency, and eventual elimination of reliance on energy assistance payments.

Three primary measures were used in the evaluation report to assess the impact of the services provided by the project: Reducing household energy use—75 percent of participating households will reduce energy use by 15 percent.

Reducing household energy burden—no operational objective stated. Reducing arrearages in utility bills—75 percent of participating households will reduce arrearages in utility bills, and 50 percent of households will not incur new arrearages for 6 months.

Other measures used in the evaluation report addressed program activities, such as completing action plans and enrolling participants in social service programs.

Project Design and Implementation

Project Assumptions

The Oregon REACH project was based on the premise that providing services to low-income households—such as information on energy conservation practices and personal financial management—would result in changes in their behavior that would reduce their energy costs and utility bill arrearages. The evaluation report identified three general assumptions that guided development of the project: Coordinating services provided by organizations within the community would be more effective than uncoordinated assistance.

Household knowledge of energy conservation, as well as assistance in weatherizing homes, is necessary to reduce reliance on energy assistance payments.

Households that participated in the project would be willing and able to make changes in their behavior and personal financial management that could affect their energy costs and the regularity of their utility payments.

Evaluation Design

According to the evaluation report, the Oregon REACH project was conceived as a “quasi-experimental” design. The project was designed as an experiment to use two test groups and a control group to determine the effect of the services provided. The report noted that the inclusion of a control group was one of the distinctive features of the design.

A different combination of services would be provided to the two test groups. One test group would receive a complete set of services— including home weatherizing and heating system repairs—as needed. The second test group would receive all of the project services, as needed, with the exception of home weatherizing and heating system repairs. Households within each group would receive different combinations of services on the basis of need and availability of the service in the community. The control group would not receive any of the project services.

Data on energy use would be collected for all three groups, when possible, for the year preceding the project, the year of project implementation, and the year after project completion. Members of each group would be required to sign forms authorizing the release of their utility bills to the project. These data would be used to evaluate energy use, utility arrearages, and energy burden by performing a statistical analysis of the differences among groups.

Both participants and caseworkers would be surveyed after project implementation to obtain their views on the project’s implementation and usefulness. An incentive payment of $20 would be given to households to complete the survey.

Services to Be Provided

Services were to be provided to participating households by 13 community-based organizations across the state. Services would be provided through case management that focused on working with households to develop an action plan for reducing their energy use according to an assessment of their needs. Case workers would also provide instruction to households in energy conservation practices and personal financial management, facilitate negotiation with energy suppliers to develop payment plans for reducing arrearages, and make referrals to other organizations providing social services.

Other services to be provided would include financial assistance to help pay for utility bills and reduce arrearages, home energy audits, and home weatherizing assistance. Project funds were also to be used to replace water heaters, furnaces, and thermostats and to provide carbon monoxide detectors and heating repairs, as needed. An emergency payment of $200 was to be allowed for especially needy households.

REACH Participant and Control Group Selection

Households were selected from those already receiving energy assistance. Households also had to have utility bill arrearages equal to one-half the energy assistance payment, an energy burden greater than 15 percent of income, and an energy-related health or safety risk. In addition, participants were selected on the basis of their motivation and the priorities of local communities. According to the report, the control group was selected from energy assistance recipients at each participating community-based organization. The sampling procedure was not discussed.

Evaluation Findings and Reported Limitations

Findings

According to the evaluation report, the Oregon REACH project was largely successful in achieving two of its primary goals: reducing energy use and reducing arrearages in household energy bills while increasing the regularity of payments to energy providers. The report noted that the project strongly supported the assumption that coordinated services effectively reduce energy use, energy costs, and energy burden for low- income households. In addition, the report stated that its analysis confirmed that the services helped households reduce arrearages and increase the regularity of utility bill payments. The evaluation report assessed the effect of services provided to households completing the project. The evaluation report stated that participant attrition was about 8 percent in the last half of the first year and 16 percent in the second year of the project.

The evaluation report found that both test groups reduced their electricity use by 11 percent for the year after project implementation. Of the 173 households receiving services, 58 (or 33.5 percent) used less energy, and 40 (or 23 percent) achieved a reduction of at least 15 percent. This was 52 percent short of the project goal of 75 percent of households achieving the 15-percent reduction. In addition, households were found to have reduced their energy burden by 2.5 percent.

The evaluation report also found that both test groups reduced arrearages in their utility bills as a result of participation in the project and that this difference was statistically significant. According to the report, the project goal that 50 percent of households not incur new arrearages for 6 months was met, and the number of households with arrearages decreased from 59 percent (102 participants) to 36 percent (63 participants). Moreover, the evaluation report found that households receiving the additional services of weatherizing and repairs achieved a slightly larger reduction in their energy bill arrearages than the test group not receiving those services— $77 compared with $55—but noted that the difference was not statistically significant.

In the exit surveys, most households indicated that the project had helped make their homes healthier, safer, and more comfortable and energy- efficient. Similarly, in the staff survey, case workers indicated satisfaction with the project.

As a result of all these findings, the evaluation report concluded that the Oregon REACH project greatly assisted low-income households in achieving a greater degree of energy self-sufficiency.

Reported Limitations

The evaluation report did not specifically identify limitations other than to note that the difference in the energy use among the groups before project implementation was statistically significant, suggesting that the households may not have been assigned to the test and control groups in a random fashion. The control group had the lowest level of energy use both before and after project implementation.

The evaluation report also noted that not all households received the full benefits because they did not complete the project and that it was difficult to contact some participants because they did not keep appointments or did not have telephone service. Two hundred and twenty four households were listed as members of the test groups, and fewer households were used in the various analyses. Data were not available for participants entering the project in the second year.

Our Observations

The effectiveness of the Oregon REACH project cannot be determined on the basis of the information presented in the project evaluation report. Although the study did have a control group, the omission of a comparison of the test groups with the control group is a shortcoming in the analyses: Only the energy consumption analysis compared the control group directly with the REACH groups.

The assessments of both energy burden and change in arrearages used two REACH test groups but did not compare changes between the two groups. Instead, they assessed changes from pre-REACH to post-REACH project points within each of the groups separately.

The study did not assess the impact of the project by comparing the change in energy use of the two REACH groups to the change in energy use of the control group.

The report also noted that participants might not have been assigned to groups in a random manner, leaving the possibility that the changes observed could be the result of factors other than the services provided. Finally, the report had other shortcomings that precluded an overall assessment of the project’s effectiveness.

Analytical Problems

The evaluation report did not identify important limitations of the analysis or their effect on the conclusions. As noted, failure to compare the test groups and the control groups for assessment of changes in energy use and arrearages limits the usefulness of the findings. The report also did not address the effect of the criteria and procedure for selecting both the test and control groups, as well as the attrition rates from these groups. These factors also adversely affect the report’s analysis and conclusions. For example, assignment to the test and control groups appears not to have been on a random basis. Test group selection resulted in a group of households with the highest energy use, whereas control group selection resulted in a group of households with the lowest energy use. Such assignment issues weaken the conclusions that can be made about the effect of REACH services on energy use. In addition, the report did not state whether energy use data were adjusted for weather.

Other Shortcomings

In addition to not including all of the information necessary to assess the statistical analysis and validity of the project’s results, the evaluation report did not provide other important information needed to assess the project’s overall effectiveness. For example, the report did not address one of the project performance goals stated in the authorizing legislation and the HHS program announcement—increasing energy suppliers’ contributions to reducing the energy burden of eligible households. In addition, the report did not provide information on the dollar amount of the expected reduction in energy use or the amount of financial assistance given to households. Finally, the evaluation report did not provide a discussion of lessons learned. A discussion of best practices—the techniques, procedures, and controls—used to ensure sufficient data for analysis might have been useful to future projects.

Appendix VIII: Comments From the Department of Health and Human Services

The following are GAO’s comments on the Department of Health and Human Services’ letter dated July 23, 2001.

GAO Comments

1. The technical comments have been incorporated as appropriate.

Several more substantive technical comments are addressed individually below. 2. The authorizing legislation defines home energy as “a source of heating or cooling in residential dwellings.” While HHS argues that residential energy commonly refers to all household energy use, for the purposes of this review, we used the narrower definition stated in the legislation. We have also changed the wording of our third recommendation, concerning the use of REACH program funds, to clarify that REACH program funds should be used for activities directly related to the home heating and cooling needs of low-income households. We do not dispute that a broad range of services is authorized in the legislation, but these services should relate to home heating and cooling. 3. We are not including the possibility of expanding the program in our matters for congressional consideration because the effectiveness of the REACH program has not yet been determined and because we believe that HHS should be required to report to the Congress in 2003 on the REACH program’s effectiveness. 4. While coordination among programs is addressed in a general way in the REACH program announcements, we believe that HHS should more specifically articulate its view of the REACH program’s purpose and relationship to LIHEAP in the Administration for Children and Families’ performance plan. We also note that a performance plan is more readily accessible to the Congress, the public, and other agencies than the program announcements intended for potential grant applicants. 5. Our draft report did not state that the use of randomized control groups is the only acceptable design, so we have not made any wording changes. Although there are legitimate questions concerning using randomized control group designs in social service programs, such designs have sometimes been effectively used. For instance, when resources are not sufficient to provide services to all who are eligible, random assignment to control groups can be feasible and ethical and provide convincing results. When randomized control groups are not feasible, evaluations can be designed to use nonequivalent control groups, and, with suitable caveats, such designs may yield defensible results. Regardless of the design used, evaluations of project results must be able to distinguish between results that are most likely due to the project services and results that may be due to other, external factors. 6. We have added language indicating that the Office of Community Services is aware of the evaluation challenges that we cite. 7. Pages 14 and 15 of our report describe the legislative authority and restrictions on the use of LIHEAP funds, as well as the need to use much of LIHEAP’s funding for direct assistance with energy bills. We also report that, as allowed by law, 25 states plan to use a portion of their fiscal year 2001 LIHEAP funds for activities such as energy education and budget counseling and 44 states plan to provide weatherization services. As HHS’ comments note, opportunity exists to continue some of the activities tested under REACH grants through states’ LIHEAP funds. We have made minor wording changes to more clearly recognize that not all REACH activities may fall within the legislative and funding constraints of the LIHEAP program. 8. Coordinating with other programs is not the same as spending REACH grant funds for non-energy-related activities. We believe that HHS’ planned efforts to clarify language in its program announcement should help ensure that REACH funds are used only for activities directly related to the home energy needs of low-income households and should provide a clearer basis for reviewing and selecting proposals.

Appendix IX: GAO Contact and Staff Acknowledgments

GAO Contact

Acknowledgments

In addition, Julie Gerkens, Kathleen Gilhooly, Curtis Groves, Rachel Hesselink, Judy Pagano, and Don Pless made key contributions to this report.