DOE Weatherization Program: A Review of Funding, Performance, and Cost-Effectiveness Studies
Released: 2012-01-11T00:00:00

Background

The DOE Weatherization Assistance Program enables low-income families to permanently reduce their energy bills by making their households more energy efficient. DOE program guidelines specify that a variety of energy efficiency measures are eligible for support under the program. The measures include insulation, space-heating equipment, energy-efficient windows, water heaters, and efficient air conditioners.

Statutory Authority

The program was created under Title IV of the Energy Conservation and Production Act of 1976 ( P.L. 94-385 ). The statute specifies that the program's primary purpose is

to increase the energy efficiency of dwellings owned or occupied by low-income persons, reduce their total residential energy expenditures, and improve their health and safety, especially low-income persons who are particularly vulnerable such as the elderly, the handicapped, and children.

The 1973 oil crisis caused rapid increases in energy prices, which caused major economic dislocations for the nation. As one result, the program was designed to save imported oil and cut heating bills for low-income households. This included senior citizens living on fixed incomes and Social Security, who were especially hard hit by rising energy bills.

The Department of Health and Human Services (HHS) operates a Low-Income Home Energy Assistance Program (LIHEAP), which was designed to help pay energy bills for low-income households. Since the inception of LIHEAP in 1981, up to 15% of funds could be used for weatherization. In 1990, the statute was amended to allow states to use up to 25% of funds for weatherization, without requiring a waiver from HHS.

Program Structure

Types of Weatherization Services

At first, weatherization providers emphasized low-cost measures, such as covering windows with plastic sheets and caulking and weatherstripping windows and doors. Many of these activities involved emergency and temporary measures. With the accumulation of experience over time, the emphasis shifted to more permanent and more cost-effective measures. The range of qualified measures expanded to include storm windows and doors, attic insulation, space heating and water heating systems, furnace and boiler replacements, and cooling efficiency measures. The cooling efficiency measures include activities such as air conditioner replacements, ventilation equipment, and screening and shading devices.

DOE recounts that the use of home energy audits was key to adapting portfolio of measures:

In the 1990s, the trend toward more cost-effective measures continued with the development and widespread adoption of advanced home energy audits. This proved to be a key advance for weatherization service providers since it required every home to be comprehensively analyzed before work began in order to select the most cost-effective measures and the best approach. This custom analysis of every home has become the hallmark of weatherization and ensures each client receives the most cost-effective treatment.

Eligible Groups

DOE's Weatherization Program is one of the largest energy efficiency programs in the nation. It is implemented in all 50 states, in the District of Columbia, in the U.S. trust territories, and by Native American Tribes. Vulnerable groups are targeted, including the elderly, people with disabilities, and families with children. A high priority is given to households with an elderly or disabled member. In FY2000, 49% of the weatherized households were occupied by an elderly resident or by a person with a disability.

Low-Income Population and Energy Cost Burden

The American Recovery and Reinvestment Act of 2009 (Recovery Act, P.L. 111-5 ; §407a) revised the program guidelines to raise the low-income eligibility ceiling from 150% to 200% of the poverty level. Low-income households have lower total energy use and smaller bills than the non-low-income population. However, those bills represent a higher proportion of total income. For 2009, Oak Ridge National Laboratory (ORNL) estimated the average energy cost burden at about 10% of income for low-income households compared to about 3.3% for non-low-income households. DOE elaborates further:

Low-income households have lower average residential energy usage and lower residential energy bills than the non-low-income population, but this difference is not in proportion to household income. The average income of low-income households as provided in the 2005 RECS and adjusted for inflation was estimated at $18,624 compared to $71,144 for non-low-income households. In 2009 the group energy burden of low-income households, defined as average residential energy expense divided by average income, was estimated to be 10 percent of income for low-income households compared to 3.3 percent for non-low-income households.... Households that actually received energy payment assistance, estimated at just over 5 million in 2005, had an even higher energy burden of 11.5 percent of income.

Allocation Formula and Intergovernmental Administration

DOE employs a formula to allocate funding to the states and territories. Each state and territory, in turn, decides how to allocate its share of the funding to local governments and jurisdictions. Funds made available to the states are allocated to local governments and nonprofit agencies for purchasing and installing energy efficiency materials, such as insulation, and for making energy-related repairs.

Funding Use Breakdown

The law directs DOE to reserve funds for national training and technical assistance (T&TA) activities that benefit all states and Native American Tribes. DOE allocates funding for T&TA activities at both the state and local levels. The total funding for national, state, and local T&TA was originally limited to 10% of an annual congressional appropriation. The Recovery Act allowed the T&TA share to increase temporarily to 20%.

The remaining funds comprise the total allocation to state programs. The program allocation consists of two parts: the base allocation and the formula allocation. The base allocation for each state is fixed, but the amount differs for each state. The fixed base was computed so that a revised formula would not cause large swings from previous allocations, which could disrupt a state's program operations. Appendix A provides a history of total annual funding.

In FY2010, a new program account for "Innovations in Weatherization" was funded. The new activity was designed to demonstrate new ways to increase the number of low-income homes weatherized and lower the federal cost per home for residential retrofits, while also establishing a stable funding base. Partnerships with traditional weatherization providers such as non-profits, unions, and contractors is the focus. The partners are expected to leverage financial resources, with a goal of $3 of non-federal contributions for each $1 that DOE provides.

Distribution Factors

The distribution of total formula allocations across the states is based on three factors: the relative size of the low-income population, climatic conditions, and residential energy expenditures. The low-income population factor is the share of the nation's low-income households in each state expressed as a percentage of all U.S. low-income households. The climatic conditions factor is obtained from the heating and cooling degrees for each state, treating the energy needed for heating and cooling proportionately. The residential energy expenditure factor is an approximation of the financial cost burden that energy use places on low-income households.

Funding Allocation Priorities and Proportional Funding Cuts

In the event of funding cuts below a minimum threshold level, DOE program rules specify how cuts would be carried out. The rule provides funding according to four priority levels. In descending order, the priorities are: national training and technical assistance (TTA) activities, TTA for state and local levels, base allocation to states, and—if the funds remaining after TTA exceed a threshold of $209.7 million—then a formula is used to spread the remainder among the states. Table 1 illustrates these priorities. In the other case—if the funds remaining after the total TTA allocation are less than $209.7 million—then there is no formula allocation and the base allocation is reduced proportionally.

Funding History: Selected Highlights20

Funding continuity has been elusive for this program. Over its history, WAP program funding has followed an up-and-down pattern, framed by occasional funding spikes and proposals to eliminate program funding and operations.

Funding Trend Highlights

The chart in Figure B -1 of Appendix B, and the data in Table A -1 of Appendix A, show the variation in the program's historical funding trend, calibrated in constant 2010 dollars. Table A -1 shows that the accumulated congressional appropriations for the 32-year period from FY1977 through FY2008 reached a sum of nearly $8.7 billion. For FY2009, the Recovery Act made a special one-time appropriation of $5.0 billion, which added about 57% to the sum of all previous appropriations through the end of FY2008.

Figure B -1 shows the alternating pattern of support, characterized by occasional agreements—and periods of marked differences—between administration and congressional funding viewpoints. A few observations on this history:

Funding Range. Except for the FY2009 Recovery Act, single-year program funding has ranged between a high of about $500 million in FY1979 to a low of about $150 million in FY1996 (constant FY2010 dollars); Funding Trend. Except for the FY2009 Recovery Act appropriation, program funding has been on a long-term downtrend since FY1979 (constant FY2010 dollars); Past Requests for Termination. In 8 out of 34 years (from FY1978 through FY2011), the administration requested zero funding (program termination): FY1982, FY1983, FY1984, FY1987-FY1990, and FY2009; Administration-Congress Agreements. In 7 years of the 35-year funding history from FY1977 through FY2011, Congress approved nearly the same funding (less than 6% difference) level as the administration requested: FY1979, FY1980, FY1981, FY1985, FY2001, FY2006, and FY2010; Request-Appropriation Percent Differences. Including the 8 years of zero funding requests, in 20 out of 35 years, the final congressional appropriation differed (plus or minus) from the administration request by more than 20%; Request-Appropriation Differences Larger than $50 Million. In 20 out of 35 years (since FY1977), the final congressional appropriation differed (plus or minus) from the administration request by more than $50 million; Appropriation Exceeded Request by $200 Million. In 10 out of 35 years (since FY1977), Congress appropriated a funding level that exceeded the administration request by more than $200 million (in 2010 dollars): FY1982-FY1984, FY1987–FY1992, and FY2009.

Funding History Highlights, FY1977 Through FY2008

Presidents Ford, Carter, and Reagan Administrations

Congress started the Weatherization Program in FY1977, during the Ford Administration. Prodded by the second oil import crisis, funding was driven rapidly upward during the Carter Administration through FY1980. For FY1981, the Carter Administration requested $428.2 million (in constant 2010 dollars), which the 96th Congress approved. In January 1981, the outgoing Carter Administration issued a DOE FY1982 budget request that sought $402.8 million (in constant 2010 dollars) for the Weatherization Program.

Shortly after coming into office in January 1981, the incoming Reagan Administration reversed the previous trend by requesting that $26.2 million be rescinded from the FY1981 appropriation. The 97th Congress approved the rescission, bringing program funding down to $376.6.0 million (in constant 2010 dollars) . In March 1981, the Reagan Administration issued an FY1982 request for zero funding, noting that it planned to restructure WAP as a block grant program under the Department of Housing and Urban Development. In DOE's FY1983 request, the Reagan Administration again proposed to terminate the Weatherization Program and other DOE grant programs:

The Energy Conservation Grants account consolidates financial and technical assistance programs carried out under the Energy Conservation appropriation which are proposed for termination in FY1983 in support of the Administration's proposal to dismantle DOE. These programs include State and Local Assistance.... The budget reductions are in response to the fact that motivated by rising energy cost and federal tax policies, individuals, businesses and other institutions have undertaken major conservation efforts. The President's economic recovery program, in conjunction with oil price decontrol and increasing natural gas prices will accelerate this trend. Public awareness of energy conservation benefits and the high level of private investment in energy conservation clearly show that the State/Local grant programs do not warrant further federal support.

Despite the zero request, Congress approved nearly $473 million in 2010 dollars for FY1983. The Reagan Administration sought zero funding again in FY1984, FY1987, FY1988, and FY1989. Following FY1983, appropriations trailed downward steadily—and stood at about $258 million in 2010 dollars in FY1989.

Presidents George H. W. Bush and Clinton Administrations

In parallel to the Reagan Administration requests for zero funding in FY1987, FY1988, and FY1989, the George H.W. Bush Administration sought zero funding for FY1990 and only relatively small amounts for FY1991 ($22 million, in 2010 dollars) and FY1992 ($35 million, in 2010 dollars).

In its first request, the Clinton Administration sought a major increase for FY1994 ($332 million, in 2010 dollars), and sustained the request at about that level for FY1995. Congress approved most of those two requests. For FY1996, the Administration came back with a request at nearly the same level (about $306 million, in 2010 dollars), but Congress approved only about $150 million (in 2010 dollars)—less than half the request. In real dollar terms, this was the lowest appropriation since FY1977. For FY1997 through FY2001, the Administration's requests hovered in a range from about $190 million to $200 million, in 2010 dollars. Congress generally approved $20 million to $40 million less than those requests, in 2010 dollars. For FY2001, Congress approved about $187 million (in 2010 dollars), which was just about $1 million less than was requested by the outgoing Clinton Administration.

President George W. Bush Administration

Bush Administration's 2001 Initiative to Increase Funding

In May 2001, President George W. Bush's National Energy Policy Development (NEPD) Group released the National Energy Policy Report. In regard to improving national energy security, that report stated:

Energy security also requires preparing our nation for supply emergencies, and assisting low-income Americans who are most vulnerable in times of supply disruption, price spikes, and extreme weather.

The report cited the cost-effectiveness of the program:

Currently, each dollar spent on home weatherization generates $2.10 worth of energy savings over the life of the home; with additional economic, environmental, health, and safety benefits associated with the installation and resulting home improvements. Typical savings in heating bills, for a natural gas heated home, grew from about 18 percent in 1989 to 33 percent today.

As one part of the plan to assist low-income people with high energy costs, the report indicated that the Bush Administration was committed to increasing support for DOE's program over the long-term. Specifically, the NEPD Group recommended that:

... the President increase funding for the Weatherization Assistance Program by $1.2 billion over ten years. This will roughly double the spending during that period on weatherization. Consistent with that commitment, the FY2002 Budget includes a $120 million increase over 2001.

As Table A -1 and Figure B -1 show, the Congress responded to the President's funding initiative by approving a nearly 50% increase for FY2002—raising funding from about $187 million to about $277 million, in constant 2010 dollars. The requests—and appropriations—climbed steadily for the next four years, reaching $261 million in FY2006. After that, the requests turned sharply downward and appropriations dropped slightly, standing at $232 million in FY2008.

Bush Administration's FY2009 Request to Terminate Funding

The FY2009 request observed that a 2003 assessment had rated the program as "moderately effective," and that:

the program coordinates effectively with other related government programs in its efforts to meet interrelated Departmental goals and achieves its goals of a favorable benefit-cost ratio and other performance goals, based on internal programmatic assessments.

DOE also noted that a new evaluation of the program's benefits and costs was underway.

However, in a reversal of its previous requests, the George W. Bush Administration requested that funding for the Weatherization Program be terminated in FY2009. The main rationale given for program termination was that the energy efficiency technology programs operated by DOE's Office of Energy Efficiency and Renewable Energy (EERE) provided a much higher benefit-to-cost ratio than that for the Weatherization Program. Specifically, the narrative for the EERE in DOE's FY2009 Congressional Budget Request stated that:

In FY2009, Weatherization Assistance Program funds are redirected to R&D programs which deliver greater benefits. EERE's Energy Efficiency portfolio has historically provided approximately a 20 to 1 benefit to cost ratio. In comparison, Weatherization has a benefit cost ratio of 1.53 to 1.

President Barack Obama Administration: Recovery Act Expands Program

Under the Obama Administration, the American Recovery and Reinvestment Act of 2009 (Recovery Act, P.L. 111-5 ) identified the program as one avenue to help address the recession and provided the program with a major one-time increase—$5 billion to weatherize an estimated 600,000 homes. This objective aimed to help achieve the President's goal of weatherizing one million homes per year. In addition to the funding increase, some amendments to the original authorizing law were enacted. The amendments allowed more cost-effective measures to be installed in more homes. One amendment raised the ceiling on cost per dwelling from $2,500 to $6,500.

Program Appeared "Shovel Ready"

The Recovery Act aimed to stimulate the U.S. economy, create jobs, and make infrastructure investments in energy and other areas. During congressional consideration of the Recovery Act, and following its enactment, the conventional wisdom was that DOE's Weatherization Program was about as close to meeting the definition of "shovel ready" as virtually any program in DOE's portfolio. Specifically, the Recovery Act weatherization effort had the following attributes: an existing programmatic infrastructure, including processes and procedures, which had been in place for many years. DOE's Inspector General observed that:

The techniques for weatherization tasks were well known and comparatively uncomplicated, and the requisite skills were widely available; performance metrics were relatively easy to establish and understand; the potential benefits for low income citizens were easily recognized; and, the potential beneficial impact on energy conservation was obvious.

With that well-entrenched program structure, there was a strong expectation that the $5 billion in Recovery Act funds would have a prompt and easily measurable impact on job creation and economic stimulation.

New Program Requirements

The Recovery Act modified the program to incorporate a new labor requirement for "prevailing wage" and to embrace new training requirements.

Prevailing Wage

The Recovery Act required that state and local recipients of weatherization funds ensure that laborers be paid at least the "prevailing wage" as determined under the Davis-Bacon Act. This requirement had not been previously applied to weatherization program activities. As a result, grantees lacked information on which to base wage rates. Many grantees chose not to begin work until the prevailing wage rates were formally established by the Department of Labor (DOL). Even after DOL's work was complete, additional delays occurred while grantees prepared guidance for sub-recipients on how to apply the wage rates. Thus, the delivery of Recovery Act-funded weatherization activities did not reach full momentum until after guidance was completed in October 2009.

Additional Training

To ensure successful implementation of program criteria under the Recovery Act, DOE required that program personnel at state and community action agencies receive additional training. The aim of the training was to ensure that recipients understood Davis-Bacon Act requirements, new income eligibility requirements, increased allowable costs per unit, and monitoring of work performed by sub-recipients. As with other state and local activities, recession-driven budget shortfalls and staff furloughs delayed many required state training initiatives.

Recession Impeded State Capacity

Recession-driven state budget shortfalls caused certain states to administer hiring freezes that applied to all employees regardless of the source of their funding, including those tasked with weatherization-related work. In some other states, progress was impacted because personnel involved with the program were subjected to significant state-wide furloughs. Further, the approval of state budgets was delayed in states such as Pennsylvania, as legislators deliberated over how to address overall budget shortfalls. Lacking staff, states were unable to perform required implementation tasks necessary to handle the large infusion of Recovery Act funding for DOE's weatherization program. Without adopted budgets, states did not have any spending authority. As a result, many states were not able to obligate or expend any weatherization program funds.

Major Implementation Delays

DOE had taken a number of proactive steps to foster timely implementation of the program. In spite of those efforts, grantees had made little progress in weatherizing homes by the beginning of 2010. As of February 2010, the one-year anniversary of the Recovery Act, only a small percentage of Recovery Act weatherization funds had been spent and few homes had actually been weatherized. Only $368.2 million (less than 8%) of the total award of $4.73 billion had been drawn by grantees. With the low spending rates, state and local grant recipients fell significantly short of goals to weatherize homes.

The lack of progress by state and local grantees to implement the weatherization program funding alarmed the DOE Inspector General (IG). In early 2010, the IG found that the nation had not realized the potential economic benefits of the $5 billion in Recovery Act funds allocated to the program. In particular, the job creation impact of what many considered to be one of DOE's most "shovel ready" projects had not materialized. Further, the IG observed that modest income residents had not enjoyed the benefits of reduced energy use and better living conditions that had been promised as part of the Recovery Act weatherization effort.

The IG's report found:

Department officials worked aggressively with the states and other responsible agencies to mitigate these challenges.... However, as a practical matter, program challenges, such as those identified in this report, placed the Recovery Act-funded Weatherization Program "on hold" for up to nine months.

Thus, the Recovery Act goals proved to be much more difficult to achieve than originally envisioned. The IG report noted:

The results of our review confirmed that as straight forward as the program may have seemed, and despite the best efforts of the Department, any program with so many moving parts was extraordinarily difficult to synchronize. In this case, program execution depended on the ability of the Federal government (multiple agencies, in fact), state government, grant sub-recipients and weatherization contractors, working within the existing Federal and state regulatory guidelines, to respond to a rapid and overwhelming increase in funding. Further, anticipated stimulus impact was affected by certain conditions and events clearly outside of Departmental control including state budget difficulties; availability of trained and experienced program staff; and, meaningful changes in regulatory requirements.

Thus, despite the assumption that the program was "shovel ready," the IG uncovered several key administrative and intergovernmental barriers to delivery of Recovery Act funded weatherization services.

Performance Assessments and Evaluation Studies of Cost-Effectiveness

This part of the report reviews a selection of key studies that examined Weatherization Program management processes, performance, and economic (energy and non-energy) impacts. Performance assessments and evaluation studies are related activities within a family of research on performance measurement and program evaluation. Both activities support the Government Performance and Results Act (GPRA) and the Office of Management and Budget's (OMB's) concerns for federal program management. In general, a performance assessment employs anecdotal data that is collected over a shorter time-frame, with a focus on management responsibilities and budget processes. In contrast, an evaluation study employs survey research methods to gather comprehensive data that is collected over a longer time-frame, with a focus on revealing the net impacts and cost-effectiveness of program operations.

Background

GPRA Performance Assessment Requirements

The Government Performance and Results Act of 1993 (GPRA, P.L. 103-62 ) established requirements for federal agencies to conduct regular performance assessments based on general methods of management by objectives and strategic planning. The law directs OMB to lead GPRA implementation, in cooperation with federal agencies. OMB describes this responsibility in Part 6 of its Circular A-11:

GPRA provides the foundation for performance planning, reporting, and budgeting for Federal agencies. GPRA requires agencies to prepare strategic plans, related performance budgets/annual performance plans, and annual performance reports (31 U.S.C. §1115). The legal requirements for an annual performance plan are met by a performance budget. The annual performance report requirement (APR) will be fulfilled by either the annual performance and accountability report (PAR) or by the congressional budget justification for agencies that choose to produce a separate annual financial report (AFR) and APR.

Among more specific GPRA purposes, Circular A-11 identifies a key GPRA purpose to:

Improve Congressional decision-making by providing more objective information on achieving statutory objectives, and on the relative effectiveness and efficiency of Federal programs and spending....

Over the years, DOE has integrated its responses to GPRA requirements into its strategic plan, annual performance report, annual performance and accountability report, and annual budget request. DOE notes that it has, since 2002, been working with OMB to assess its programs with the Performance Assessment Rating Tool (PART). Evidently, PART had become the main vehicle for reporting of GPRA assessment requirements. PART is described in greater detail in the next section of the report.

Circular A-11 defines program assessment:

A determination, through objective measurement and systematic analysis, of the manner and extent to which Federal programs achieve intended objectives.

It also defines program evaluation:

Program and practice evaluation is an assessment, through objective measurement and systematic analysis, of the manner and extent to which programs or practices achieve intended results. Program and practice evaluations should be performed with sufficient scope, quality, and independence. Although agencies may cite rigorous evaluations commissioned independently by organizations such as the Government Accountability Office, Office of the Inspector General, or other groups, these evaluations should not completely supplant rigorous evaluations commissioned by the agencies themselves.

DOE's Approach to Assessment and Evaluation

DOE's Weatherization Program has been assessed through a combination of assessment and evaluation methods. DOE's GPRA-driven program assessment tools—strategic plan, annual performance plan, performance and accountability report, and performance budget—tend to be quite broad, focused on the larger agency missions of R&D, science, and defense. Those documents tend to mention the program incidentally, mainly citing achievements in terms of annual number of units weatherized. For a more in-depth examination of program operations, DOE has a history of directing the Oak Ridge National Laboratory (ORNL) to conduct evaluation impact studies.

Choice of Studies for Review

In 1993, DOE published a major evaluation report, the first "scientific" study of program performance, impact, and cost-effectiveness. That report established impact evaluation studies as a key feedback component that promoted program design changes and improvements. However, its cost, complexity, and lag time led DOE to subsequently rely on aggregations of available state-level evaluations. Also, in 1993, GPRA stimulated the emergence of a second track of less in-depth performance assessments that had a shorter-term budget focus. Both tracks of analysis—performance assessments and evaluation studies—have the same general goals of providing feedback that can be used to improve program operations. So both types of studies were selected for review.

Selected Performance Assessments

2003 IG Performance Audit

A performance audit is an assessment tool used to examine the performance and management of a program against objective criteria. It is designed to facilitate oversight and may serve several objectives, including the assessment of program effectiveness and results. From late 2002 through early 2003, DOE's Office of the Inspector General (IG) conducted a performance audit of the program. The audit period included Program Year 2000 (PY2000), PY2001, and planned activities for PY2002. The total DOE budget for the audit review period was $518 million. The purpose of the audit was to "determine whether the program was properly administered and was achieving its goals." The audit was conducted in accordance with Government Auditing Standards for performance audits.

The IG observed that the program has a long-established structure for transferring funds to state and local agencies and that improvements over the years had made use of the funds more efficient and effective. As further background, the IG observed that, in addition to DOE funds, state and local agencies also obtain funding for weatherization from the Department of Health and Human Services' (HHS) Low-Income Home Energy Assistance Program (LIHEAP) and other programs funded by utilities, states, and other sources.

Local agencies report to DOE annually through state weatherization offices on expenditures, number of units completed, and other performance-related measures. For PY2001, the IG noted that " about 900 agencies received funding that ranged from as low as a few thousand dollars to as high as $4 million." The IG found two local government reporting issues: administrative costs and number of households weatherized.

Regarding the first concern, it identified instances where local agencies reported administrative-type expenses as program operating costs:

Specifically, we observed that certain organizations inappropriately charged expenses such as administrative staff, office rent, and administrative supplies as direct program costs and thus understated total administrative costs. Public laws and federal regulations limit the amount of weatherization grant funds that may be used for administrative purposes. If local agencies continue to under report administrative costs, states could ultimately exceed statutory limitations for administrative expenses over the life of the grant.

DOE advised the IG that it is often difficult for local agencies to operate within the limits for administrative costs. The IG concluded that the DOE Program Office should work more closely with states and local agencies to ensure that administrative costs are minimized and that the costs incurred are reported accurately and consistently.

Regarding the concern about local agency reporting on the number of households weatherized, the IG found that:

... data regarding the number of households weatherized was not reported on a consistent basis. For example, data reported by some states related strictly to the number of homes weatherized using Department funds. Other states, however, combined the results of weatherization efforts funded by the HHS LIHEAP with those completed with Departmental funds. Merging performance data for such states distorts program results and could make it appear program efficiencies and energy savings are greater than that actually achieved with available funding.

A recent OMB program assessment found that DOE had addressed the IG's concern about distorted efficiency (benefit-cost) measures:

Average cost per home employed in the calculation of benefit/cost ratios now reflects total costs (including non-DOE funds) expended per unit in the states whose evaluations were used in the energy savings estimates. This substantially raises the per-unit investment from a DOE-only level making the benefit/cost ratio more conservative.

2003 OMB Assessment with PART Method

PART Background

The Performance Assessment Rating Tool (PART) was developed by the Office of Management and Budget (OMB) in 2002 as a key component for implementing GPRA and the President's Management Agenda (PMA), particularly the Budget and Performance Integration initiative. PART was designed to assess program planning, management, and performance against quantitative, outcome-oriented goals. It can inform funding and management decisions aimed at improving program effectiveness. As a diagnostic instrument for evaluating efficiency and effectiveness, PART aims to help managers identify, anticipate, and rectify performance problems.

PART was designed to provide a basis for DOE and OMB to agree upon meaningful annual and long-term targets. The PART assessment process aims to unify annual performance targets and goals with long-term goals. PART was designed to be an iterative process, capable of tracking the evolution of program performance over time through periodic reassessments. OMB's recommendations to foster program improvement are central to the PART process. Program offices track actions taken to implement PART recommendations and report those actions to OMB semi-annually.

According to OMB, the ongoing cycle of reviewing and implementing PART recommendations, coupled with the use of performance data from assessments and periodic reassessments, signify the perception of PART as an integral process for planning and budget decision-making, as distinguished from a set of one-time program evaluations.

PART assessments help inform budget decisions and identify actions to improve results. Agencies are held accountable for implementing PART follow-up actions and working toward continual improvements in performance.

PART asks a series of questions that cover four areas:

program purpose and design, performance measurement, evaluation, and strategic planning, program management, and program results.

To earn a high PART rating, a program must use performance assessment to manage, justify its resource requests on expected (projected) performance, and continually improve efficiency. All of those activities are goals of the Budget and Performance Integration Initiative. The PART performance rating scale is shown below:

PART Report on DOE Weatherization Program

The summary of a PART report on the DOE Weatherization Program found that it was performing at the "moderately effective" level. The assessment defines the moderately effective rating as follows:

In general, a program rated Moderately Effective has set ambitious goals and is well-managed. Moderately Effective programs likely need to improve their efficiency or address other problems in the programs' design or management in order to achieve better results.

The PART assessment cited three findings to explain the "moderately effective" rating. First, it noted that the program met its annual performance target for the number of homes weatherized. Second, OMB stated that the program lacked an assessment of performance that was current, comprehensive, and independent. It noted that the program reported on "internal assessments" that showed a favorable benefit-cost ratio. However, OMB found that those assessments relied in part on old data and were conducted by Oak Ridge National Lab (ORNL), which it says is not an "independent" source. Third, OMB noted that the DOE Inspector General identified issues with the way state and local agencies report on program management. DOE responded that it was conducting an independent analysis of the cost-effectiveness of the program and addressing the IG's audit recommendations.

The detailed PART assessment report presented trend data for key performance measures, including number of homes weatherized, cost per home, and the overall benefit-cost ratio. Also, it described and rated strategic program features, such as goals, objectives, design, targets, and budgeting.

The PART Assessment defined the benefit-cost ratio as a long-term measure of program efficiency. OMB stated that the benefit-cost ratio represents the value of energy saved, divided by program costs. The ratio depended in part on estimated long-term energy prices at the time of the assessment and on an average energy savings per household of 29.1 million British thermal units (MBtu). PART reported that estimates of the benefit-cost ratio for energy savings under various price scenarios ranged from 1.19 to values greater than 2.00, never measuring less than 1.00.

PART assessment question number 4.3 asked:

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program performance goals each year?

The response was:

Benefit-cost ratio rose from 1.06 in 1989 to 1.79 in 1996, and then declined to 1.51 and 1.30 in 1999 and 2002, respectively. These estimates depend largely on EIA estimated long-term energy prices.

PART question number 2.6, on strategic planning, asked:

Are independent and quality evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

The response touched upon several aspects of the evaluation strategy, including frequency of studies, measurement of non-energy benefits, and independence of the evaluator. It stated:

The program does not conduct annual evaluations on a national basis because of the high cost of such evaluation and the limited amount of change that occurs in program activities from year to year. The program has contracted with Oak Ridge National Laboratory (ORNL) to devise evaluation methodologies and report periodically on program results based on state grantee-level performance evaluation. ORNL has also conducted selective evaluation activities designed to inform program management of performance characteristics in areas in which the program performance has been below average (hot climate zones) or in areas in which there has been growing strategic program interest but little evaluation data. The latter includes base load electric measures as well as non-energy benefits. To assure independence, the program should consider using an alternative contractor in future assessments, or at least having future ORNL reports assessed by a third party.

As evidence of its evaluation activities, DOE's response included references to several ORNL program metaevaluations and other evaluation studies. Some key ORNL studies are described in the next section.

2011 GAO Performance Audit of Recovery Act Funding

The Government Accountability Office (GAO) defines a performance audit broadly:

Performance audits provide objective analysis so that management and those charged with governance and oversight can use the information to improve program performance and operations, reduce costs, facilitate decision making by parties with responsibility to oversee or initiate corrective action, and contribute to public accountability.

According to GAO, a performance audit may embrace a wide variety of objectives, including:

an analysis of the "relative cost-effectiveness of a program or activity," and/or a determination of the "current status or condition of program operations or progress in implementing legislative requirements."

Thus, by GAO's definition, a performance audit may include activities with the same purposes as those of an impact and/or process evaluation study.

The Recovery Act directed GAO to conduct bimonthly reviews and reporting on selected states and localities' use of the funds it provided, including the funds for the DOE Weatherization Program. GAO describes these bimonthly review as performance audits. In addition to the bimonthly audits, GAO's response to the directive has included one performance audit that covered only the Weatherization Program.

For example, in December 2009, GAO published a performance audit report on Recovery Act funds used by states and localities. The DOE Weatherization Program was one of many programs covered. For the weatherization component, GAO covered 16 states and 24 localities. GAO found that, as of the end of FY2009, states had outlaid about 2% ($113 million) of weatherization funds and had completed about 1% (7,300) of the targeted number of homes to be weatherized. Many contracts between states and local agencies had been delayed due to concerns about staff capacity and new labor requirements.

As another example, in May 2010, GAO published another audit report on Recovery Act funds that covered DOE Weatherization Program funds. As of the mid-point in FY2010, states had outlaid about 13% ($659 million) of weatherization funds and had completed about 13% of the targeted number of homes to be weatherized. GAO made several observations about the status of program implementation and offered some recommendations for program improvement.

In January 2011, the GAO initiated a performance audit focused solely on Recovery Act funding for the DOE Weatherization Program. The report was released in late December 2011. The study aimed to examine: (1) status of funds, (2) implementation challenges facing recipients, (3) achievement of energy and cost savings, and (4) changes in the quality of employment data reported by recipients. To address the four objectives, GAO conducted a web survey of all 58 recipients and received 55 responses. GAO also interviewed officials from DOE and selected national associations, conducted site visits in seven states, and conducted telephone interviews with two other state agencies. Selected findings, as of the end of FY2011, included: (1) recipients had spent about $3.46 billion (73%) of the $4.75 billion allocated, (2) recipients reported that implementation challenges were declining, (3) the average cost per home was about $4,900, about 563,000 homes had been weatherized, and DOE projected it would exceed the original target of 607,000 homes by the deadline set for the end of March 2012; and (4) GAO found that the quality of employed data reported by recipients had improved.

Selected Evaluation Studies

Several DOE reports, prepared primarily by staff of the Oak Ridge National Laboratory (ORNL), have assessed the energy cost savings and non-energy benefits of the DOE Weatherization Program. In particular, a comprehensive evaluation of the PY1989 program was published in 1993.

1993 National Evaluation Report

In 1993, DOE issued a report entitled National Impacts of the Weatherization Assistance Program in Single Family and Small Multifamily Dwellings . The reporting project was an exhaustive five-year, first-of-its-kind, comprehensive evaluation study of the energy savings and cost-effectiveness of the program during PY1989. A letter accompanying the report noted that the program is "the nation's largest residential conservation program, and one of its oldest." The letter captures the highlights:

The report evaluates the national and regional energy savings and cost effectiveness of the program in single-family dwellings, mobile homes, and small (2-unit to 4-unit) multifamily dwellings. It is based upon a representative national sample of nearly 15,000 dwellings weatherized by 368 local weatherization agencies, and a control group of homes waiting to be weatherized by the same agencies.

The report estimated an overall program benefit-cost ratio of 1.09 for the entire program. Adding estimates of non-energy benefits—such as comfort, employment, reduced environmental impacts, and preservation of affordable housing—yielded a societal benefit-cost ratio of 1.72.

Three Metaevaluations (1997, 1999, 2005)

After the 1993 National Evaluation report, DOE relied upon "metaevaluations" of state evaluation studies as the main tool to assess program energy savings and cost effectiveness. All of the metaevaluations were conducted by ORNL. Compared with the 1993 National Evaluation, metaevaluations became a low-cost way for DOE to update energy savings and cost-effectivness. This section reviews three key metaevaluations, covering program operations from 1990 through 2005.

1997 Metaevaluation (Covers 1990-1996)

DOE asked ORNL to prepare the 1997 metaevalution. The study involved locating, assembling, and summarizing the results of all state-level evaluations that had become available since 1990. ORNL found 17 state-level evaluation studies from 15 states that it used to cover program operations during the period from 1990 through 1995. Energy savings were expected to be higher than those found in the 1993 National Evaluation, which was based on 1989 data. Typical savings ranged from 18% to 24%, which was greater than the range from 12% to 16% found in the 1993 National Evaluation. Key reasons for the expected improvement included some advances in weatherization procedures, such as the use of advanced audits and blower-door directed air sealing. The study concluded that program performance had improved significantly over that seven-year period.

The 1997 metaevaluation study used three perspectives to estimate cost-effectiveness. The program perspective compared energy benefits to total program costs. The installation perspective compared energy benefits to installation costs. The societal perspective compared the summed value of energy and non-energy benefits to total costs. ORNL reported that the metaevaluation calculations employed the same procedures and assumptions used in the 1993 National Evaluation. The values of 1.79 (program perspective), 2.39 (installation perspective), and 2.40 (societal perspective), respectively, are shown in Table 3 .

The study found that the synthesis of state-level evaluations offered a reasonable characterization of national program performance:

Although national level evaluation efforts are sometimes needed to definitively demonstrate program performance, reviews of state-level evaluations provide useful, and inexpensive, benchmarks of progress during the years between such large-scale national assessments.

1999 Metaevaluation (Covers 1996-1998)

As a follow-up to the 1996 metaevaluation, ORNL performed another metaevaluation, which was published in 1999. The report synthesized results from 10 individual evaluation studies of state weatherization efforts that were completed between April 1996 and September 1998.

The stated objectives of the 1999 study were (1) to identify average energy savings of households in the states that provided information; (2) to identify key variables that explain the magnitude of those weatherization-induced savings; and (3) to use the findings from the state studies to estimate average household energy savings that could be expected nationwide.

The study reported that program-induced savings continued at a higher rate than those found in the 1993 national evaluation. However, the study found no dramatic changes were made in the structure or practices of the program during the period after the 1996 study. Thus, no major increases in savings were expected above those found in the 1996 study.

To make the benefit-cost ratios in the 1999 study comparable to those in the 1996 metaevaluation and the 1993 national evaluation, ORNL used the same assumptions and procedures. In particular, the average measure lifetime was assumed to be 20 years and the discount rate used was 4.7%. Cost-effectiveness was calculated for the weatherization program nationwide. As in past evaluations, three different benefit-cost perspectives (program, installation, and societal) were examined.

Because higher average national energy savings were estimated in the 1996 and 1999 metaevaluations, the benefit-cost ratios for those years were higher than the ones reported in the 1993 National Evaluation. The benefit-cost ratios for the three perspectives are shown in Table 3 .

2005 Metaevaluation (Covers 1993-2005)

The most recent metaevaluation, published in 2005, estimated average household energy savings at 30.5 million British thermal units (MBtu) per year with a benefit/cost ratio from the program perspective estimated at 1.34 with 2003 prices. Costs were based on those derived from the state evaluations themselves.

Like the previous metaevaluations, ORNL undertook this one to update the findings from the 1993 national evaluation. This metaevaluation was based on data from 38 evaluation studies of weatherization programs in 19 states, published between 1993 and 2005.

The overall method of the 2005 metaevaluation paralleled that of the previous metaevaluations, but ORNL describes some improvements:

1. Expanded Range of Time and Geography. The overall similarity of results across the various studies led ORNL to include findings from a 2003 study that covered all post-1992 state-level studies. ORNL found that this approach increased sample size, improved the ability to cover all major U.S. climate regions, and added to the statistical rigor of the results. 2. States as the Unit of Analysis. ORNL aggregated data from the 38 studies by state. This aspect contrasts with previous metaevaluations, which had treated multiple studies of the same state as separate empirical observations. 3. Studies Combined into a Single State Value. For each relevant variable (e.g., energy savings per gas-heated household), a single average value was used to represent the entire state. ORNL says this procedure prevents overall study results from being skewed due to the presence of multiple studies with similar results in a single state. 4. Weighted Average Used to Account for Multiple Studies. Where there were multiple studies, the average values computed in each study were used to determine a weighted average for each variable, with the weighting done by sample size.

Under this revised approach, those states that represent a larger part of the national program contribute more heavily to the analysis and findings. This is appropriate because the purpose of the study is to estimate energy savings nationwide.

Table 3 shows the benefit-cost values in the various ORNL metaevaluations. The similarity among the values is not surprising because those meta analyses used data from many of the same state-level studies.

ORNL states that all three of the metaevaluations, including the 2005 study, focused primarily on energy savings in homes heated by natural gas because the large majority of state-level studies addressed that fuel. ORNL found a statistically significant difference between the natural gas savings per household in the 2005 metaevaluation and the savings reported in the 1993 national evaluation. However, the 2005 study noted that the other two-thirds of the states were not included in the metaevaluation, and thus did not provide data for the study, which raises the possibility that the sample examined may not fully represent the nation as a whole.

ORNL found that actual benefit-cost values are likely to be higher than those reported for natural gas savings, as shown in Table 3 . This was due to the fact that reported expenditures also included the cost of installing measures to reduce baseload electricity use, but only the benefits of natural gas savings were used in those benefit-cost calculations.

Overall, the study found significant program improvements since 1989 (as reported in the 1993 national evaluation) and a generally higher benefit-cost ratio. However, it also acknowledged the shortcomings of the metaevaluation method and called for a new national evaluation:

While the metaevaluations performed over the last decade have consistently shown higher natural gas savings per household than those reported in the national evaluation, there is a need to corroborate those findings through a rigorous examination of Weatherization Program efforts nationwide. Even the current metaevaluation is based on studies performed in only a third of the states, and those may not be fully representative of the entire Weatherization Program. Also, the value for preweatherization energy consumption, which is a major input for the calculation of national savings, is based on 1989 data. In addition, while state-level evaluations have put a strong emphasis on gas-heated houses, few studies have been conducted on electrically heated dwellings. And it is important to note that the biggest recent change to the Weatherization Program – the addition of baseload measures such as highly efficient refrigerators, water heaters, and light bulbs – has barely been addressed by state-level studies. For all these reasons, there is a strong need for a new national evaluation to thoroughly explore the current operations and achievements of the Weatherization Program across the entire nation.

DOE has engaged APPRISE, a nonprofit analytical organization, to conduct the second national evaluation. The status of plans for that evaluation are described in the section below, entitled "Evaluation Plan for the Regular Program."

2010 Technical Memorandum (FY2010 Projected Data)

The Recovery Act provided a large infusion of funding and prescribed major new requirements for the program. In response to concerns about these major changes, ORNL prepared a technical memorandum to update estimates from the metaevaluations. The memo only attempted to make preliminary projections based on an audit; it did not attempt to generate empirical data from an impact evaluation study. Specifically, the memo recounted results from an engineering modeling approach that used the National Energy Audit Tool (NEAT) to estimate the performance of the program by using typical homes in each state. These homes varied by building type, primary heating fuel and prices, and local weather conditions.

ORNL employed the Weatherization Assistant residential audit package to estimate annual energy savings under new parameters set by the Recovery Act. The estimated annual savings for heating and cooling projected for FY2010 were 29.0 MBtu per household. In comparison, the 2005 metaevaluation estimated savings at 30.5 MBtu over the period from 1993 through early 2005.

The Recovery Act raised the average per-unit investment cost ceiling from $2,500 to $6,500. The likely increase in investment per unit was expected to increase total energy savings. However, the higher funding level was accompanied by a shift in formula allocation that allowed proportionally more funding for states in warmer climates.

ORNL projected a 2010 benefit-cost ratio of 1.80 (from the program perspective). Most of the increase relative to the 2005 benefit-cost ratio of 1.54 was attributed to the change in energy prices from 2003 to 2010. Costs were based on the Recovery Act ceiling of $6,500 for average costs. However, the ceiling for cost-effective expenditures varied from state to state, based on the requirement that the savings-to-investment ratio (SIR) reach one (1.0) or greater for each of three factors: the last measure installed, energy intensity, and local energy prices.

Traditionally, the program benefit-cost ratio has been expressed in terms of estimated national energy cost savings per dollar of investment for homes heated with natural gas, with cost savings based on national average residential natural gas prices. ORNL's estimates in the memo differ somewhat, due to reliance on state-by-state energy savings and investment numbers used to generate the energy savings estimates. This aspect allowed the dollar savings estimates to vary by region, by fuel type, by housing type, and with the adjusted average cost ceiling provided under the Recovery Act. The projected 2010 societal benefit/cost ratio (all benefits divided by all costs) is 2.51. This is almost identical to the societal benefit/cost ratio of 2.53 computed in the 2005 metaevaluation.

ONRL reflected on the limits of the data used to make estimates, noting again the need for a new national evaluation:

The methods used here are more complex and hopefully better at reflecting current reality than those used in previous years. Nonetheless, one needs to keep in mind the limitations of available input data that constrain an analysis such as this. There is no nationally available data on energy-related housing characteristics of weatherized households, nor is there data on the variations in the cost of measures installed from one locale to another. There is also no way of knowing at this point the exact characteristics of the housing stock and housing types that will actually get weatherized with the greatly expanded revenues available. Making these estimates more precise requires populating the methodology with more accurate data that will flow from the National Evaluation effort.

ONRL noted that the memo's estimates were derived from a different methodology than that used in the metaevaluations. There are several reasons that a change in method was needed. First, the metaevaluation estimates are dated and do not reflect recent changes in program operations that materially impact household savings. These include a major change in the program's average cost ceiling, from $2,500 to $6,500, an expansion in allowable measures to include electricity measures such as refrigerator replacement and lighting changeout, and a major increase in program funding impacting the allocation of resources among different regions and climate zones.

Second, the 2005 metaevaluation results describe only homes heated with natural gas and do not reflect the diversity of heating fuels used in treated homes nor do the results reflect potential cooling savings. The method used for the metaevaluations reflected only savings in single-family homes and those savings were never adjusted for variations in the treated housing stock. Finally, the estimates in the metaevaluations were based on national average energy prices and were not varied to reflect the diversity of energy prices weighted by the location of the weatherization work being performed.

ONRL states that the methodology used to prepare statistics for the memo corrects for many of the above-noted deficiencies. However, it acknowledges that the findings do not present a statistically valid representation of the program's performance and results:

There are too many assumptions and uncertainties incorporated in it to allow that to be the case. Much of this is the result of a lack of up-to-date information on program operations, particularly regarding measures installed and their cost as well as the energy-related characteristics of the homes weatherized. Rather, the results should be treated as the best currently available estimate that can serve until more rigorous results are provided by the new National Evaluation.

Current Plans for Assessment and Evaluation Studies

Three activities are underway to assess and evaluate DOE's Weatherization program: a DOE (Deloitte) strategic assessment, a DOE (APPRISE) impact study for PY2007 through PY2008, and a DOE (APPRISE) impact evaluation for the Recovery Act period of PY2009 through PY2011. Table 4 provides a brief overview of the goals and target dates for the three activities. Descriptions of the activities follow.

Plan for a Strategic Assessment (Deloitte)

In November 2008, DOE's Weatherization Assistance Program Office announced that it had contracted with Deloitte Limited Liability Partnership (LLP) to perform a strategic assessment of the program. Deloitte is tasked with conducting a fundamental analysis of the program's objectives, impact metrics, market delivery vehicles, and finance mechanisms. The study aims to identify fundamental improvements in program design and delivery, in contrast to the more traditional evaluation of program benefits and costs that ORNL has conducted in the past.

Evaluation Plan for Regular Program (APPRISE, 2007-2008 Data)

DOE acknowledges that OMB's PART review and the IG's report have highlighted the need for an updated comprehensive evaluation of the program to ensure that objectives are being met and that estimates of energy savings, bill reductions, program costs, and program benefits are valid. Since the first national evaluation was published in 1993, there have been several changes made to program policies, procedures, technologies, and techniques. DOE notes that it is important to assess how well these changes have worked. For example, there is a need to assess how well new policies—such as whole-house weatherization—are impacting the program.

In response to the growing need for updated and valid program data, the program moved forward in FY2005 to begin planning another national evaluation. With a committee comprised of state and local weatherization staff, the program office started planning the evaluation and identifying areas where the program might be improved. DOE states that the 1993 national evaluation brought about major program changes such as computerized audits, less use of door and window measures, and increased emphasis on hot climate opportunities. DOE anticipates that the second national evaluation will have a similar effect on future program policy and performance.

In February 2007, ORNL released a plan for a second national retrospective evaluation, which would use data collected during PY2006. ORNL says that the program has been changed by findings from the 1993 study, a strategic planning process, and other initiatives that had caused the program to change significantly. The program has incorporated new funding sources, management principles, audit procedures, and energy efficiency measures.

ORNL is supervising a competitively selected independent contractor team to conduct the evaluation. The team is led by the Applied Public Policy Research Institute for Study and Evaluation (APPRISE). The study will assess program performance for PY2007 and PY2008, the period immediately preceding the Recovery Act. Performance measures will include program costs and benefits. In order to conduct a comprehensive savings analysis, the design includes collection of a full year of post-weatherization billing data. The evaluation is expected to be completed by Fall 2012.

Design for the APPRISE Second National Retrospective (Impact) Evaluation

DOE's central evaluation question is: How much energy did the weatherization program save in 2007 and 2008? DOE plans to use well-known analytical approaches to answer this question. Electricity and natural gas billing histories will be collected pre- and post-weatherization for a sample of weatherized homes (the treatment group) and a sample of comparable homes that were not weatherized (the comparison group). The comparison group for the 2007 treatment group will be homes weatherized in 2008. The comparison group for the 2008 treatment group will be homes weatherized in 2009. A national sample of 400 local weatherization agencies will be selected to provide information on about one-third of their weatherized homes. Billing history data will be normalized using three different analytic methods, including the Princeton Scorekeeping Method (PRISM).

DOE will also address an important complementary evaluation question: How cost-effective were those energy savings? In principle, all measures should be cost-effective, because each one is required to meet the savings-investment ratio (SIR) test. Evaluations are conducted to compare expectations with measured values. DOE says it will collect cost information associated with each weatherized home included in the treatment sample. Total projected energy cost savings over the lifetime of the measures will be divided by total costs to estimate a benefit-cost ratio for the program.

The study will attempt to quantify nonenergy benefits, which will be sorted into three categories—utility benefits, occupant benefits, and societal benefits. Utility providers benefit because weatherization reduces arrearages and service shutoffs. Occupants benefit because weatherization makes homes more comfortable, healthier, safer, and more valuable. Society benefits because weatherization curbs greenhouse gas emissions, reduces other forms of pollution, and conserves water. Further, it increases local employment and it increases local economic activity caused by the multiplier effect.

Design for APPRISE Process Evaluation

DOE intends that the Second National Retrospective Evaluation also examine program processes. To do that, it plans to survey all program grantees and subgrantees to collect data about program operations, training and client education activities, and quality assurance procedures. That survey data will be used to provide a program snapshot for PY2008. Case studies and a field study will be incorporated into the process evaluation design to furnish additional insights on program implementation.

Evaluation Plan for Recovery Act (APPRISE, 2009-2011 Data)

The Recovery Act committed $5 billion over two years to an expanded weatherization program. This large sum greatly heightened interest in the program, the population served, its energy and cost savings, and its cost-effectiveness. Solid answers to many of the questions about the program and its performance require a comprehensive evaluation. DOE is funding an evaluation by APPRISE in order to ascertain program effectiveness and to improve program performance. The evaluation project has a budget of $19 million. GAO reviewed the plan's design for evaluating energy cost-effectiveness and found it methodically sound.

DOE observes that the weatherization program has changed considerably. In particular, it notes that there are four key differences between the pre-Recovery Act program and the post-Recovery Act program. First, to expend $5 billion and to increase weatherization activity by 300%, a greatly expanded weatherization workforce was recruited, trained, organized, and sent into the field. To support that expansion, the percentage of program funds allowed for training and technical assistance was raised from 10% to 20%. Second, all states and U.S. territories received large funding increases, and some states and local agencies struggled with weatherization budgets several times larger than previous ones. Faced with that major program expansion, many states and local agencies chose to implement innovations in program delivery and management. Third, DOE also has made provisions to set aside substantial sums to support innovations in program funding and design.

Fourth, to accommodate the Recovery Act funding expansion, major changes were incorporated into the program:

The definition of the threshold for a low- income household was raised from 150% to 200% of the federal poverty income guidelines; The ceiling on average cost per unit—that is, the average amount of money that grantees can spend to weatherize their homes—was increased from $2,500 to $6,500; and Wages for weatherization workers were subjected to Davis-Bacon Act prevailing wage requirements for the first time.

DOE/ORNL says that many features of the 2009-2011 evaluation will parallel those of the retrospective evaluation for 2007-2008, as described in the previous section. Analyses of energy savings and cost-effectiveness will be similar, although billing histories may be collected to also assess the effects of certain program innovations.

The main difference between the retrospective evaluation of 2007-2008 and the evaluation of 2009-2011 will be the process evaluation component. DOE says that funding and associated provisions have had a significant impact on the weatherization community and program operations. Thus, it plans two special process evaluation studies as part of the Recovery Act evaluation project. One would focus on Davis-Bacon requirements and the other would address changes in the composition of the national weatherization community.

In May 2011, ORNL published a plan for the Recovery Act evaluation that includes details about both the impact and the process components of the assessment. The schedule for the evaluation activities covers the period from the second quarter of calendar year (CY) 2011 through the first quarter of CY2014.

Impacts of Davis Bacon Requirement

Congress directed that weatherization projects funded with Recovery Act appropriations must follow Davis-Bacon labor rules. The Department of Labor is responsible for identifying prevailing wages in the construction industry. These wages are identified for a set of construction industry jobs and are estimated for each county in the United States. Two possible questions DOE has identified for this part of the evaluation research are:

(1) What were the actual, monetary administrative costs for complying with Davis-Bacon? (2) Overall, how did application of Davis-Bacon influence the cost-effectiveness of WAP?

Impacts on Administration, Training, and Employment

DOE observes that the large flow of Recovery Act funds into low-income weatherization changed the national weatherization network. The weatherization labor force grew larger and new stakeholders were drawn into the network. The increased funding had both positive and negative effects on long-standing leveraging relationships. DOE is considering several possible research questions. Four of the questions are:

(1) How have relationships between state weatherization offices and local weatherization agencies changed? (2) Did the Recovery Act change the way that local agencies use contracts to procure weatherization services? (3) What approaches did local agencies and/or contractors use to recruit and train new qualified, reliable, and trustworthy weatherization crew members, and how effective were these approaches? (4) Did the expanded weatherization workforce find work opportunities in the energy efficiency field outside of the DOE weatherization program?

Other Expected Results

The evaluation design is expected to produce a variety of new information and findings about the program. Basic results are expected to include estimated energy savings for 2007 through 2011, along with estimates of cost-effectiveness and non-energy benefits. The design aims to uncover insights into energy savings attributable to particular measures and the strengths and weaknesses of computer audits, versus priority lists, as a way to select appropriate weatherization measures. Also, the design is expected to produce program operational data for two levels of average home investments ($2,500 and $6,500) and two levels of home eligibility (150% and 200% of the poverty level). Findings are expected to yield some indication of the impact on program benefits attributable to changing the federal grant formula in a way that distributes more funds to states in warmer climates.

Issue of Evaluation Independence and Objectivity

The OMB Inspector General has criticized DOE for depending almost exclusively on "in-house" ORNL staff to conduct evaluation studies. The core issue is whether an "in-house" evaluation effort can be sufficiently objective or neutral in its conduct of research. This issue is part of a quite spirited debate over federal program evaluation that goes far beyond the DOE weatherization program. A key assumption of the debate is that a separation of evaluation duties from program implementation duties would increase the "independence" of the evaluating organization. In response to criticism regarding ORNL, DOE engaged a competitive process to retain an outside contractor to conduct the second national evaluation (PY2007-PY2008) and the evaluation of the Recovery Act period (PY2009-PY2011). As noted previously, the competition led to the selection of the APPRISE contracting firm.

Debate Over the Use of Evaluation Contractors

A recent debate between professors Charles Metcalf and David Reingold focused on the extent to which the use of "outside" contractors to conduct federal program evaluations can significantly improve research independence and objectivity. In a separate study, Professor Jacob Klerman continues the dialogue, adding further perspectives to the debate. All three professors suggest ways to strengthen the independence of evaluations conducted with contractors.

Metcalf Argues: Use of Contractors Improves Evaluation Independence

Charles Metcalf speaks from experience as both a professor and an evaluation contractor. He says the central issue for evaluation independence is control over research processes and reporting. Metcalf acknowledges that outside pressures can affect a contractor's objectivity. In particular, he contends that client (agency/program staff) authority to accept or reject the evaluation findings and report—and to time its release—can give it significant control over evaluation content. Metcalf argues that the independence and objectivity of contractor evaluations are promoted by four main conditions: maturity of the contractor organization, contractor reputation and longevity, formal agency-contractor agreements, and research transparency.

First, he claims that, over the past 40 years, the methods and practices of federal program evaluation have matured to the point where major evaluation contractor organizations have become generally objective and independent:

Mathematica Policy Research (MPR), and others such as Abt Associates, MDRC, and RAND ... have established objectivity and independence as core values, and by and large have successfully defended these values in their conduct with funding organizations. Our track record establishes that government supported policy research can be and is independent, with proper safeguards.

Second, Metcalf claims that the ability of a contractor organization to establish a reputation for objectivity enables it to enjoy longevity. In turn, this longevity helps the contractor to be independent from any single administration:

If we define the client to be the current administration, the contractor and often the contract outlive the current client. For contract research institutions that regard themselves as doing policy research for a sequence of clients with differing policy agendas, the institutions' reputation for objectivity (as well as the quality of their work) is their primary currency in maintaining their longevity.

Third, Metcalf argues that the agency's ability to influence content is largely constrained by the evaluation contract:

For contract research, the researcher and his or her employing institution are collecting data and conducting research on an issue defined and paid for by a client. In this environment, two questions must be answered: (1) Who owns the research product? and (2) Who controls the content and the dissemination of the research product? These questions are typically defined by the contract, and the rights no longer lie intrinsically with the institution nor, by extension, with the researchers it employs to conduct the research.

Further, he claims that there is an increased tendency for researchers and clients to agree explicitly, in advance, on the research methodology and the scope of reports. This process tends to protect the client (agency) from unexpected surprises, and protects the researcher (contractor) by allowing less "wiggle room" for the client to reinterpret results.

Fourth, Metcalf contends that research transparency is a key factor for contract evaluation independence. In this regard, he draws a parallel to academic research:

[T]he basic guardian of both quality and objectivity in the conventional world of scholarly research is the required openness to the examination of others. If you don't adhere to this rule, your research is presumptively not credible.

Metcalf argues that openness in the various phases of federal program evaluations is guaranteed by the government's competitive procurement process and by the Freedom of Information Act (FOIA). In procurement, the agency makes the request for proposals (RFP) publicly available. After the contract is awarded, the public can submit a FOIA request to obtain both the contract document (including the statement of the work) and the winning proposal:

Regardless of internal contract restrictions, the contractor has the implicit nonexclusive right to request its own report under the FOIA, and to release it freely without restriction. Thus, while the government owns the research products it purchases through contract, the FOIA protects the researcher's right to disseminate his or her results—and, indeed, the right of anyone else to disseminate the results.

Metcalf further claims that, in practice, the right of evaluation contractors to disseminate research results through conference participation and publication parallels the rights of researchers in academic institutions. Also, in parallel to academic peer review, he notes that third parties often have a role in evaluations:

First, we have already stressed the importance of the ultimate availability of research reports to the public, time delays notwithstanding. Second, most large-scale studies engage advisory panels and other forms of peer review at both the design and report stages.

Thus, he observes that third party involvement is an integral aspect of transparency.

While acknowledging that threats to independence persist, Metcalf contends that the contracting environment has had a positive effect on evaluation. He concludes that the practice of using external evaluation contractors has shown an ability to enhance research credibility and independence.

Reingold Counter-Argues: With Contractors, Threats to Independence Remain

Professor Reingold starts from the same assumptions as Professor Metcalf. Reingold says separation of the evaluation activity is assumed to establish evaluator independence which—along with quality research techniques—allows program evaluation to serve an accountability function:

[U]nlike other services frequently contracted out by governmental and nongovernmental organizations, the rationale for contracting out program evaluation rests largely in its ability to establish independence by creating a clear division of labor between those conducting evaluations and those responsible for managing programs being evaluated.

He elaborates further that the process for selecting a contractor is vital to independence:

[T]he perception of independence is typically established via the selection process used to identify and hire the person or organization responsible for designing and implementing the evaluation. For many organizations that purchase evaluation services, independence is established by selecting someone who is external to the organization or program being evaluated and who is free from financial self-interest or ideology that may bias the evaluation to produce a desired outcome, and where the evaluator has control (or ownership) over the evaluation design and its intellectual property (data and findings).

Reingold argues that all of the "safeguards" to independence that Metcalf identified are overwhelmed by three main factors: (1) agency funding control gives it influence over evaluation content, (2) contractor organizations have financial self-interests in accommodating agency wishes, and (3) their mutual self-interests undermine the competitiveness of the selection process.

First, Reingold argues that an agency's control over evaluation funding allows it to retain a strong influence over evaluation content:

[W]hen purchasing evaluation services via contracts, the purchasing agency has total control over all aspects of the services to be provided ... [thus] ... [i]t is inconsistent to argue that contracted program evaluation can be independent when the agency purchasing the evaluation owns (or controls) the research question and design, the method of data collection, the strategy of analysis, the data, the final report, and the rules governing the dissemination of results.

Second, Reingold contends that contractor organizations have a financial self-interest in accommodating agency wishes:

[E]valuation contractors view their relationships with those responsible for running programs and funding evaluations as clients where satisfaction with the deliverable will influence the likelihood of securing additional business in the form of more evaluation contracts.

In other words, he argues, independence can be lost when experts that are expected to produce independent facts also have a substantial incentive to acquiesce to a client's interests.

Third, Reingold claims that self-interests may begin to erode the contractor selection process, which underpins the independence sought by separating evaluation activity from program management:

When program staff are allowed to strongly influence the selection of a firm to conduct an evaluation of their program, they frequently select firms that will "work with them" to produce a "helpful" evaluation. This is code for selecting a firm that will tailor and implement an evaluation design that will likely provide positive findings for the program under study.

Over time, this threat to independence can deepen. Ultimately, he says, the complementary self-interests of agency and contractor may undermine the competitive foundation of the selection process:

When a firm has established itself as a reliable partner that will cooperate with the demands of a particular client, the selection process has virtually eliminated any real competition that would allow for a truly independent evaluation.

In sum, Reingold argues that threats to independence and objectivity can enter from both sides—a program's interest in self-protective control and a contractor's interest in preserving a business relationship. He concludes that:

Unfortunately, there are signs this arrangement [the effort to achieve greater independence through external contracting] is not working.

To address the problem, Reingold suggests two policy options to improve the independence of evaluation contracting. One option is to substitute grants for contracts, as the grants confer greater independence to the recipient. However, grants typically allow less oversight than contacts, which means that quality could suffer. Another option is to expand third party involvement. This option is described in the last section of the report.

Klerman Contributes to the Debate

Since the Metcalf-Reingold debate, there has continued to be a focus on means to reduce threats to evaluation independence. For example, Professor Jacob Klerman echoes some points from both Metcalf and Reingold in describing his view of the challenges to contractor evaluation independence. Klerman stresses two key threats to independence: the mutually reinforcing self-interests of funder and contractor and the self-interests of third parties. Also, he offers two ways to improve independence by modifying formal contract agreements.

First, Klerman characterizes the main threat to independence as an "inherent tension" between funder and contractor. Beyond stated goals for quality and objectivity, he claims that both parties bring "unstated goals" involving pre-set policy opinions and organizational survival interests that can color an evaluation project's process and findings. He echoes Reingold's argument that self-interests pose a key threat to independence:

[E]valuation is sometimes politics (or business) by other means. For the funder, showing that the current administration's programs "work" (and those of the previous administration did not "work") yields political benefits. Sometimes the concerns are quite direct. A negative evaluation result might cause shrinking of a program, loss of budget for the funder's organization, in the extreme even loss of jobs for the funder's lead evaluation staff. For the contractor, this is a business. The "right answer"—that is, the answer that the funder "wants to hear"—now is likely to lead to more evaluation business in the future; the "wrong answer" now is likely to lead to less evaluation business in the future.

Second, Klerman proposes that when seeking a balance point between funder and contractor over evaluation content, one option is to involve a third party. However, "[e]ven nominating some third party leaves open the question of the third party's policy and organizational goals." From his perspective, all three parties to an evaluation—funder, contractor, and third party—are propelled by self-interests that pose threats to independence. However, Klerman raises the question of whether there may be an "appropriate balance" of these interests that could actually reinforce independence.

In conclusion, Klerman—like Metcalf—claims that independence may be improved by augmenting, or otherwise improving, formal contract agreements. He describes one strategy for improvement:

The challenge is to devise contractual mechanisms that assure satisfaction of the stated goal—a high-quality and independent evaluation. Ideally, a contract would give the funder enough authority to handle standard contractual issues while preventing either side from deviating from the stated goals to further policy or organizational goals.

As a starting point for this strategy, Klerman suggests that a clear separation be established between an official evaluation report and a contractor's own publication of analysis from the underlying evaluation. He recommends that:

The funder would retain almost unfettered rights to the official contract report. Those rights would include the right never to officially release the document but not the right to change the contractor's text while continuing to list the contractor and its individual staff members as authors. The contractor would retain clearly defined rights to publish any findings from the evaluation.

Klerman's main strategy for improving independence encompasses 19 approaches to modify evaluation contracts, which are grouped into three categories: (1) approaches applied when a funder retains the right to specify the final report text, specify the report release, and control discussion of the process; (2) approaches that limit the funder's right to specify the final text, and (3) approaches that limit the funder's right to specify report release.

Table 5 summarizes the main points of the Metcalf-Reingold debate, including the points contributed by Klerman.

The Limits of Independence: Is There an Optimum?

Agencies of other national governments are also concerned about program evaluation independence. An example is the United Kingdom's Department for International Development (DFID). An assessment of DFID's evaluation function addressed many of the same issues of control over the research process and findings that were the focus of the Metcalf-Reingold debate. The study observed that independence is important, a critical aspect of evaluation credibility and reliability:

To be sure, independence on its own does not guarantee quality (relevant skills, sound methods, adequate resources and transparency are also required) but there is no necessary trade-off between independence, quality or credibility. Indeed, evaluation quality without independence does not assure credibility. Furthermore, in open and accountable working environments, evaluation independence induces credibility, protects the learning process and induces program managers and stakeholders to focus on results. Thus, evaluation independence, quality and credibility are complementary characteristics that together contribute to evaluation excellence.

The DFID assessment of evaluation function suggests that the quest for independence—if taken too far—could lead to dysfunction. The report suggests that there may be limits to evaluation independence:

Optimum independence is not maximum independence. Accurate and fair evaluations combine intellectual detachment with empathy and understanding. The ability to engage with diverse stakeholders and secure their trust while maintaining the integrity of the evaluation process is the acid test of evaluation professionalism. This is why diminishing returns set in when evaluation independence assumes extreme forms of disengagement and distance. Independence combined with disengagement increases information asymmetry, ruptures contacts with decision makers and restricts access to relevant sources of information. It leads to isolation, a lack of leverage over operational decision making and a chilling effect on learning. Thus, the basic challenge of evaluation governance design consists in sustaining full independence without incurring isolation.

That finding appears to be a more elaborate expression of a key conclusion that Klerman makes in regard to achieving an "appropriate balance" of the three potential parties to a contract evaluation:

[N]o single party—not the funder, not the contractor, and not some third parties—will always be without bias. Different approaches shift the relative power of the three groups. Furthermore, different approaches have different costs—in dollars, in the formality of the funder–contractor relationship, and in time to complete the study. The appropriate balance will vary with the particular situation, and, in particular, with the prevalence of problematic behaviors on all sides—funder, contractor, and third parties.

As noted previously, Klerman proposes several means to modify the client-contractor relationship to address the inherent tension and reduce—but not eliminate—the threats to objectivity and independence.

Implications of the Independence Debate for DOE Evaluations

DOE Addresses Threats to Independence

DOE is aware of the concern about contractor independence for impact evaluations. It has published guidance for program staff that addresses the issue:

Evaluation should be conducted by outside independent experts. This means that, even if staff commission a study (fund an evaluation contractor) that contractor should have some degree of independence from the program office that is being evaluated. Also, the contractor should have no real or perceived conflict of interest. Although the program staff person may work with the contractor during the consultation phase to clarify information needs and discuss potential evaluation criteria and questions, the staff should establish some line of separation from the contractor for much of the remainder of the evaluation study—i.e., put up a firewall after the initial consultation period is concluded.

Also, in order to address remaining concerns about outside contractor independence, DOE guidance further recommends the establishment of a third party group of experts to critique the contractor's work:

It is further highly recommended that a panel of external evaluation experts who are not part of the contractor's team be assembled to serve as reviewers of the contractor's work. This would include the Evaluation Plan, and the draft and final reports. Having the work of the contractor's team itself evaluated helps ensure the evaluation methodology is sound and the study is carried out efficiently. It also sets up a second firewall that raises the credibility of the study to even a higher level (important for those who remain skeptical of evaluation studies commissioned by the program being evaluated).

Further Considerations for DOE's Active Evaluations

DOE has addressed the first point of the independence debate. It separated the evaluation activity from the program activity by choosing outside contractors to perform a strategic assessment and two major evaluation studies. Additional comments and questions may focus on the other three points of the independence debate:

• Contractor Selection. All three debaters stressed that the selection process is critical to evaluation independence. Was the selection of Deloitte and APPRISE competitive? What objective indicators should be used to gauge the competitiveness of the selection process? Is there an optimum degree of selection process competitiveness? • Transparency. Metcalf argued that transparency helps guarantee evaluation independence. Will this dimension be fulfilled by the public release of the three final reports? Otherwise, what objective indicators should be used to gauge transparency for each evaluation? Is there an optimum degree of transparency? • Third Party Involvement. Klerman described the possibility of an "appropriate balance" that included a third party. Related to the ongoing evaluation of the Recovery Act period (2009-2011), the DOE IG issued a progress report and GAO has contributed a major performance audit. Is that sufficient third party involvement? Otherwise, what objective indicators should be used to gauge the sufficiency of third party involvement for each evaluation? Is there an optimum degree of third party involvement?

Some Policy Options That May Enhance Independence

The debate between Metcalf and Reingold produced some other ideas for establishing independence of the evaluation function. One notable strategy suggests using other (third party) organizations to play a more central role in managing the program evaluation process. Reingold suggests:

Ideally, Congress, the GAO, and perhaps the National Academy of Sciences [National Research Council] could play a more active role in screening and selecting evaluation plans and firms.

Three variations on this strategy, in increasing order of dependence on the outside agency are:

(1) Have the DOE Office of the Inspector General (DOE IG) or GAO or Congressional Budget Office (CBO) ensure that the third party contractor evaluation work is performed in an independent manner.

(2) Move the contractor (evaluator) selection process from the agency to the Office of the Inspector General, GAO, CBO, or the National Research Council (NRC).

(3) Transfer full responsibility for conduct of the evaluation to the DOE IG, GAO, CBO, or NRC.

Appendix A. DOE Weatherization Program: Historical Data

Appendix B. DOE Weatherization Funding Chart

Figure B -1 , below, shows the entire history of annual requests and appropriations for the DOE Weatherization Assistance Program (WAP). Requested amounts for FY1977, FY1978, and FY1979 were not available. A final FY2012 appropriation figure had not yet been determined at the time this report was prepared.