Department of Energy: Further Actions Are Needed to Improve DOE's Ability to Evaluate and Implement the Loan Guarantee Program
Released: 2010-07-12T00:00:00

Letter

DOE Has Broadly Indicated the Program’s Direction but Is Not Well Positioned to Evaluate Progress

DOE has broadly indicated the direction of the LGP but has not developed all the tools necessary to evaluate progress. DOE officials have identified a number of broad policy goals that the LGP is intended to support, including helping to ensure energy security, mitigate climate change, jumpstart the alternative energy sector, and create jobs. Additionally, through DOE’s fiscal year 2011 budget request and a mission statement for the LGP, the department has explained that the program is intended to support the “early commercial production and use of new or significantly improved technologies in energy projects” that “avoid, reduce, or sequester air pollutants or anthropogenic emissions of greenhouse gases, and have a reasonable prospect of repaying the principal and interest on their debt obligations.”

To help operationalize such policy goals efficiently and effectively, principles of good governance identified in our prior work on GPRA indicate that agencies should develop associated performance goals and measures that are objective and quantifiable. These performance goals and measures are intended to allow comparison of programs’ actual results with the desired results. Each program activity should be linked to a performance goal and measure unless such a linkage would be infeasible or impractical.

DOE has linked the LGP to two departmentwide performance goals: “Double renewable energy generating capacity (excluding conventional hydropower) by 2012.” “Commit (conditionally) to loan guarantees for two nuclear power facilities to add new low-carbon emission capacity of at least 3,800 megawatts in 2010.”

DOE has also established nine performance measures for the LGP (see app. II).

However, the departmentwide performance goals are too few to reflect the full range of policy goals for the LGP. For example, there is no measurable performance goal for job creation. The performance goals also do not reflect the full scope of the program’s authorized activities. For example, as of April 2010, DOE had issued two conditional commitments for energy efficiency projects—as authorized in legislation—but the energy efficiency projects do not address either of the performance goals because the projects are expected to generate little or no renewable energy and are not associated with nuclear power facilities. Given the lack of sufficient performance goals, DOE cannot be sure that the LGP’s performance measures are appropriate. Thus, DOE lacks the foundation to assess the program’s progress, and more specifically, to determine whether the projects it supports with loan guarantees contribute to achieving the desired results.

DOE Has Taken Steps to Implement the LGP but Has Treated Applicants Inconsistently and Lacks Mechanisms to Identify and Address Applicants’ Concerns

As the LGP’s scope and authority have increased, the department has taken a number of steps to implement the program for applicants. For example, DOE has substantially increased the LGP’s staff and in-house expertise, and applicants we interviewed have commended the LGP staff’s professionalism. DOE officials indicated that, prior to 2008, staffing was inadequate to review applications, but since June 2008, the LGP’s staff has increased from 12 federal employees to more than 50, supported by over 40 full-time contractor staff. Also, the LGP now has in-house legal counsel and project finance expertise, which have increased the program’s capacity to evaluate proposed projects. In addition, in November 2009, the Secretary named an Executive Director, reporting directly to the Secretary, to oversee the LGP and to accelerate the application review process.

Other key steps that DOE has taken include the following: DOE has identified a list of external reviewers qualified to perform legal, engineering, financial, and marketing analyses of proposed projects. Identifying these external reviewers beforehand helps to ensure that DOE will have the necessary expertise readily available during the review process. DOE officials said that the department has also expedited the procurement process for hiring these external reviewers.

DOE developed a credit policies and procedures manual for the LGP.

Among other things, the manual contains detailed internal policies and procedures that lay out requirements, criteria, and staff responsibilities for determining which proposed projects should receive loan guarantees.

DOE revised the LGP’s regulations after receiving information from industry concerning the wide variety of ownership and financing structures that applicants or potential applicants would like to employ in projects seeking loan guarantees. Among other things, the modifications allow for ownership structures that DOE found are typically employed in utility-grade power plants and are commonly proposed for the next generation of nuclear power generation facilities.

DOE obtained OMB approval for its model to estimate credit subsidy costs. The model is a critical tool needed for the LGP to proceed with issuing loan guarantees because it will be used to calculate each loan guarantee’s credit subsidy cost and the associated fee, if any, that must be collected from borrowers. (We are evaluating DOE’s process and key inputs for estimating credit subsidy costs in other ongoing work.)

Notwithstanding these actions, the department is implementing the program in a way that treats applicants inconsistently, lacks systematic mechanisms for applicants to appeal its decisions or for applicants to provide feedback to DOE, and risks excluding some potential applicants unnecessarily. Specifically, we found the following: DOE has treated applicants inconsistently. Although our past work has shown that agencies should process applications with the goals of treating applicants fairly and minimizing applicant confusion, DOE’s implementation of the program has favored some applicants and disadvantaged others in a number of ways. First, we found that, in at least five of the ten cases in which DOE made conditional commitments, it did so before obtaining all of the final reports from external reviewers, allowing these applicants to receive conditional commitments before incurring expenses that other applicants were required to pay. Before DOE makes a conditional commitment, LGP procedures call for engineering, financial, legal, and marketing reviews of proposed projects as part of the due diligence process for identifying and mitigating risk. If DOE lacks the in-house capability to conduct the reviews, external reviews are performed by contractors paid for by applicants. In one of the cases we identified in which DOE deviated from its procedures, it made a conditional commitment before obtaining any of the external reports. DOE officials told us this project was fast-tracked because of its “strong business fundamentals” and because DOE determined that it had sufficient information to proceed. However, it is unclear how DOE could have had sufficient information to negotiate the terms of a conditional commitment without completing the types of reviews generally performed during due diligence, and proceeding without this information is contrary to the department’s procedures for the LGP.

Second, DOE treats applicants with nuclear projects differently from applicants proposing projects that employ other types of technologies. For example, DOE allows applicants with nuclear projects that have not been selected to begin the due diligence process to remain in a queue in case the LGP receives additional loan guarantee authority, while applicants with projects involving other types of technologies that have not been selected to begin due diligence are rejected (see app. III). In order for applicants whose applications were rejected to receive further consideration, they must reapply and again pay application fees, which range from $75,000 to $800,000 (see app. IV). DOE also provided applicants with nuclear generation projects information on how their projects ranked in comparison with others before they submitted part II of the application and 75 percent of the application fees. DOE did not provide rankings to applicants with any other types of projects. DOE officials said that applicants with nuclear projects were allowed to remain in a queue because of the expectation that requests would substantially exceed available loan guarantee authority and that the applications would be of high quality. According to DOE officials, they based this expectation on information available about projects that are seeking licenses from the Nuclear Regulatory Commission. DOE officials also explained that they ranked nuclear generation projects for similar reasons—and also to give applicants with less competitive projects the chance to drop out of the process early, allowing them to avoid the expense involved in applying for a loan guarantee. However, all of the solicitations issued through 2008 initially received requests that exceeded the available loan guarantee authority (see app. V), so nuclear projects were not unique in that respect. In addition, applicants with coal-based power generation and industrial gasification facility projects paid application fees equivalent to those paid by applicants with nuclear generation projects but were not given rankings prior to paying the second application fee (see app. IV). To provide EERE applicants with earlier feedback on the competitiveness of their projects, DOE instituted a two-part application for the 2009 EERE solicitation—a change from the 2008 EERE solicitation. DOE officials stated that they made this change based on lessons learned from the 2008 EERE solicitation. While this change appears to reduce the disparity in treatment among applicants, it remains to be seen whether DOE will make similar changes for projects that employ other types of technologies.

Third, DOE has allowed one of the front-end nuclear facility applicants that we contacted additional time to meet technical and financial requirements, including requirements for evidence that the technology is ready to move to commercial-scale operations, but DOE has rejected applicants with other types of technologies for not meeting similar technical and financial criteria. DOE has not provided analysis or documentation explaining why additional time was appropriate for one project but not for others.

DOE lacks systematic mechanisms for applicants to appeal its decisions or provide feedback to DOE. In its solicitations, DOE states that a rejection is “final and non-appealable.” Once a project has been rejected, the only administrative option left to an applicant under DOE’s documented procedures is to reapply and incur all of the associated costs. Nevertheless, DOE said that, as a courtesy, it had rereviewed certain rejected applications. Some applicants did not know that DOE would provide such rereviews, which appear contrary to DOE’s stated policy and have been conducted on an ad hoc basis.

DOE also lacks a systematic mechanism for soliciting, evaluating, and incorporating feedback from applicants about its implementation of the program. Our past work has shown that agencies should solicit, evaluate, and incorporate feedback from program users to improve programs. Unless they do so, agencies may not attain the levels of user satisfaction that they otherwise could. For example, during our interviews with applicants, more than half said they received little information about the timing or status of application reviews. Applicants expressed a desire for more information about the status of DOE’s reviews and said that not knowing when a loan guarantee might be issued created difficulties in managing their projects—for example, in planning construction dates, knowing how much capital they would need to sustain operations, and maintaining support for their projects from internal stakeholders.

According to DOE officials, the department has reached out to stakeholders through its Web site, presentations to industry groups and policymakers, and other means. DOE has also indicated that it has changed the program to make it more user-friendly, based on lessons learned and applicant feedback. For example, unlike the 2008 EERE solicitation, the 2009 EERE solicitation includes rolling deadlines that give applicants greater latitude in when to submit their applications; a simplified part I application that provides a mechanism for DOE to give applicants early feedback on whether their projects are competitive; and delayed payment of the bulk of the “facility fee” that DOE charges applicants to cover certain program costs. While DOE said that these changes were based, in part, on feedback from applicants, because DOE has no systematic way of soliciting applicant feedback, the department has no assurance that the views obtained through its outreach efforts are representative, particularly since the means that DOE uses to obtain feedback do not guarantee anonymity. The department also has no assurance that the changes made in response to feedback are effectively addressing applicant concerns.

DOE risks excluding some potential applicants. Even though the Recovery Act requires that applicants begin construction by the end of fiscal year 2011 to qualify for Recovery Act funding, DOE has not yet issued solicitations for the full range of projects eligible for Recovery Act funding under section 1705. DOE has issued two solicitations specific to the Recovery Act for the LGP, but neither invites applications for commercial manufacturing projects, which are eligible under the act. While DOE has announced that it will issue an LGP solicitation for commercial manufacturing projects, it has given no date for doing so. The 2009 EERE solicitation provided an opportunity for some manufacturing applicants to receive Recovery Act funding, but because DOE combined the Recovery Act’s requirements with the original section 1703 requirements, applicants with commercial manufacturing projects were excluded. DOE officials told us that they combined the requirements to ensure that projects that are initially eligible under section 1705 but that fail to start construction by the deadline can remain in the LGP under section 1703.

Conclusions

DOE has made substantial progress in building a functional program for issuing loan guarantees under Title XVII of EPAct; however, it may not fully realize the benefits envisioned for the LGP until it further improves its ability to evaluate and implement the program. Since 2007, we have been reporting on DOE’s lack of tools necessary to evaluate the program and process applications and recommending that the department take steps to address these areas. While DOE has identified broad policy goals and developed a mission statement for the program, it will lack the ability to implement the program efficiently and effectively and to evaluate progress in achieving these goals and mission until it develops corresponding performance goals. As a practical matter, without such goals, DOE will also lack a clear basis for determining whether the projects it decides to support with loan guarantees are helping achieve the desired results, potentially undermining applicants’ and the public’s confidence in the legitimacy of those decisions. Such confidence could also be undermined by implementation processes that do not treat applicants consistently—unless DOE has clear and compelling grounds for disparate treatment—particularly if DOE skips steps in its review process prior to issuing conditional commitments or rereviews rejected applications for some applicants without having an administrative appeal process. Furthermore, while DOE has taken steps to increase applicants’ satisfaction with the program, it cannot determine the effectiveness of those efforts without systematic feedback from applicants that preserves their anonymity.

Recommendations for Executive Action

To improve DOE’s ability to evaluate and implement the LGP, we recommend that the Secretary of Energy take the following four actions: Direct the program management to develop relevant performance goals that reflect the full range of policy goals and activities for the program, and to the extent necessary, revise the performance measures to align with these goals.

Direct the program management to revise the process for issuing loan guarantees to clearly establish what circumstances warrant disparate treatment of applicants so that DOE’s implementation of the program treats applicants consistently unless there are clear and compelling grounds for doing otherwise.

Direct the program management to develop an administrative appeal process for applicants who believe their applications were rejected in error and document the basis for conclusions regarding appeals.

Direct the program management to develop a mechanism to systematically obtain and address feedback from program applicants, and, in so doing, ensure that applicants’ anonymity can be maintained, for example, by using an independent service to obtain the feedback.

Agency Comments

We provided a draft of this report to DOE for review and comment. In its written comments, DOE stated that it recognizes the need for continuous improvement to its Loan Guarantee Programs as those programs mature but neither explicitly agreed nor disagreed with our recommendations. In one instance, DOE specifically disagreed with our findings: the department maintained that applicants are treated consistently within solicitations.

Nevertheless, the department stated that it is taking steps to address concerns identified in our report. Specifically, DOE pointed to the following recent or planned actions: Performance goals and measures. DOE stated that, in the context of revisions to its strategic plan, the department is revisiting the performance goals and measures for the LGP to better align them with the department’s policy goals of growing the green economy and reducing greenhouse gases from power generation.

Consistent treatment of applicants. DOE recognized the need for greater transparency to avoid the perception of inconsistent treatment and stated that it will ensure that future solicitations explicitly describe circumstances that would allow streamlined consideration of loan guarantee applications.

Appeals. DOE indicated that its process for rejected applications should be made more transparent and stated that the LGP continues to implement new strategies intended to reduce the need for any kind of appeals, such as enhanced communication with applicants including more frequent contact, and allowing applicants an opportunity to provide additional data at DOE’s request to address deficiencies DOE has identified in applications.

While these actions are encouraging, they do not fully address our findings, especially in the areas of appeals and applicant feedback. We continue to believe that DOE needs systematic mechanisms for applicants to appeal its decisions and to provide anonymous feedback.

DOE’s written comments on our findings and recommendations, along with our detailed responses, are contained in appendix VI. In addition to the written comments reproduced in that appendix, DOE provided technical comments, which we incorporated as appropriate.

We are sending copies of this report to the appropriate congressional committees, the Secretary of Energy, and other interested parties. This report also is available at no charge on the GAO Web site at http://www.gao.gov.

If you or your staffs have any questions concerning this report, please contact me at (202) 512-3841 or ruscof@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix VII.

Appendix I: Scope and Methodology

To assess the extent to which the Department of Energy (DOE) has identified what it intends to achieve through the Loan Guarantee Program (LGP) and is positioned to evaluate progress, we reviewed and analyzed relevant provisions of Title XVII of the Energy Policy Act of 2005 (EPAct), the American Recovery and Reinvestment Act of 2009 (Recovery Act); DOE’s budget request documents; and Recovery Act planning information, as well as other documentation provided by DOE. We discussed strategic planning and program evaluation with cognizant DOE officials from the LGP office, the Office of the Secretary of Energy, the Office of the Chief Financial Officer, and the Credit Review Board (CRB) that is charged with coordinating credit management and debt collection activities as well as overall policies and procedures for the LGP. As criteria, we used the Government Performance Results Act (GPRA), along with our prior work on GPRA.

To evaluate DOE’s implementation of the LGP for applicants, we reviewed relevant legislation, such as EPAct and the Recovery Act; DOE’s final regulations and concept of operations for the LGP; solicitations issued by DOE inviting applications for loan guarantees; DOE’s internal project tracking reports; technical and financial review criteria for the application review process; minutes from CRB meetings held between February 2008 and November 2009; applications for loan guarantees; application rejection letters issued by DOE; and other various DOE guidance and procurement documents related to the process for issuing loan guarantees. We interviewed cognizant DOE officials from the LGP office, the Office of the Secretary of Energy, the Office of the Chief Financial Officer, the Office of Headquarters Procurement Services, and program offices that participated in the technical reviews of projects, including the Office of Electricity Delivery and Energy Reliability, the Office of Energy Efficiency and Renewable Energy, the Office of Nuclear Energy, and the National Energy Technology Laboratory (NETL). In addition, we interviewed 31 LGP applicants and 4 trade association representatives, using a standard list of questions for each group, to obtain a broad representation of views that we believe can provide insights to bolster other evidence supporting our findings. We selected the applicants and trade associations using a mix of criteria, including the amount of the loan guarantee requested and the relevant technology. As criteria, we used our prior work on customer service. We did not evaluate the financial or technical soundness of the projects for which applications were submitted.

We conducted this performance audit from January 2009 through July 2010 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Performance Measures for the LGP

DOE has developed the following nine performance measures for the LGP: percentage of projects receiving DOE loan guarantees that have achieved and maintained commercial operations; contain the loss rate of guaranteed loans to less than 4 percent; contain the loss rate of guaranteed loans to less than 11.81 percent in fiscal year 2009 (11.85 percent for fiscal years 2010 and 2011) on a long-term portfolio basis; newly installed generation capacity from power generation projects receiving DOE loan guarantees; average cost per megawatthour for projects receiving DOE loan forecasted greenhouse gas emissions reductions from projects receiving loan guarantees compared to ‘business as usual’ energy generation; forecasted air pollutant emissions (nitrogen oxides, sulfur oxides, and particulates) reductions from projects receiving loan guarantees compared to ‘business as usual’ energy generation; average review time of applications for Section 1705 guarantees; and percentage of conditional commitments issued to qualified applicants relative to plan.

Appendix III: Application Review Process

Appendix III: Application Review Process Required for projects with estimated total costs exceeding $25 million.

Required for projects with estimated total costs exceeding $25 million.

Required for projects with estimated total costs exceeding $25 million.

Required for projects with estimated total costs exceeding $25 million.

Appendix IV: Standardized Fees Associated with Obtaining a Loan Guarantee, by Solicitation

Appendix IV: Standardized Fees Associated with Obtaining a Loan Guarantee, by Solicitation $600,000 ½ of 1% of guaranteed 600,000 ½ of 1% of guaranteed 600,000 ½ of 1% of guaranteed 2008 Energy efficiency, renewable energy, and advanced transmission and distribution technologies (EERE) 1% of guaranteed amount $375,000 + 0.75% of guaranteed amount $1,625,000 + 0.50% of guaranteed amount 1% of guaranteed amount $375,000 + 0.75% of guaranteed amount $1,625,000 + 0.50% of guaranteed amount 600,000 ½ of 1% of guaranteed 2009 Commercial technology renewable energy generation projects under the Financial Institution Partnership Program (FIPP)

Appendix V: Loan Guarantee Amounts Available and Amounts Applicants Sought for Technology Categories Targeted in Solicitations

Coal-based power generation and industrial gasification facilities Energy efficiency, renewable energy, and advanced transmission and distribution technologies (EERE)

Electric power transmission infrastructure projects Commercial technology renewable energy generation projects under the Financial Institution Partnership Program (FIPP)

The 2006 mixed solicitation invited applications for all technologies eligible to receive loan guarantees under the Energy Policy Act of 2005 except for nuclear facilities and oil refineries.

Appendix VI: Comments from the Department of Energy

The following are GAO’s comments on the Department of Energy’s (DOE) letter dated June 17, 2010.

GAO Comments

1. DOE appears to concur with the spirit of our recommendation. Best practices for program management indicate that DOE should have objective, quantifiable performance goals and targets for evaluating its progress in meeting policy goals DOE has identified for the LGP. Such goals and targets are important tools for ensuring public accountability and effective program management. 2. Our finding about inconsistent treatment of LGP applicants is based on information obtained from applicants corroborated by documents from DOE. In the instance we identified in which DOE made a conditional commitment before obtaining any of the required external reports, the external reviewers were not fully engaged until after DOE had negotiated the terms of the conditional commitment, which is contrary to DOE’s stated procedures and provided an advantage to the applicant. Other applicants who received conditional commitments before completion of one or more of the reports called for by DOE’s due diligence procedures also had a comparative advantage in that they were able to defer some review expenses until after DOE had publicly committed to their projects. We continue to believe that DOE should revise the process for issuing loan guarantees to treat applicants consistently unless there are clearly established and compelling grounds for making an exception. 3. We agree that there may be grounds for treating applicants differently depending on the type of technology they employ but do not believe that DOE has adequately explained the basis for the differences among the solicitations. For example, DOE’s response does not address the possibility that lack of ranking information for fossil energy projects, combined with the knowledge that the solicitation was significantly oversubscribed, could have factored into applicants’ decisions to drop out of the process, especially given the relatively high fees associated with submitting part II of the application. 4. We disagree that DOE’s current process for rereviewing rejected applications is working. As we state in our report, some applicants did not know that DOE would provide rereviews. While we are encouraged by DOE’s efforts to reduce the need for appeals, we believe that an administrative appeal process would allow DOE to better plan and manage its use of resources on rejected applications. 5. We applaud DOE’s efforts to reach out to stakeholders and to use lessons learned to improve procedures and increase efficiencies and effectiveness. However, we continue to believe that DOE needs a systematic mechanism for applicants to provide anonymous feedback, whether through use of a third party or other means that preserves confidentiality. Several applicants we interviewed expressed concern that commenting on aspects of DOE’s implementation of the LGP could adversely affect their current or future prospects for receiving a loan guarantee. Systematically obtaining and addressing anonymous feedback could enhance DOE’s efforts to improve procedures and increase efficiencies and effectiveness.

Appendix VII: GAO Contact and Staff Acknowledgments

Staff Acknowledgments

In addition to the individual named above, Karla Springer, Assistant Director; Marcia Carlsen; Nancy Crothers; Marissa Dondoe; Brandon Haller; Whitney Jennings; Cynthia Norris; Daniel Paepke; Madhav Panwar; Barbara Timmerman; and Jeremy Williams made key contributions to this report.