National Laboratories: Better Performance Reporting Could Aid Oversight of Laboratory-Directed R&amp;D Program
Released: 2001-10-15T00:00:00

Letter

Background

The Congress has long recognized the value of allowing laboratories to conduct a certain amount of discretionary research. The current LDRD program grew out of legislation enacted in 1977 that authorized the use of a reasonable amount of laboratory funds to conduct employee-suggested, research and development (R&D) projects selected at the discretion of the laboratory directors. DOE’s implementation of its authority to conduct discretionary research evolved over the years. For example, in 1983, DOE Order 5000.1A formally established a discretionary R&D program called Exploratory Research and Development (ER&D); in 1992, DOE Order 5000.4A established the current LDRD program, which includes the previously defined ER&D program and other discretionary work, and further memorialized its long-standing policy of allowing its multi-program national laboratories the discretion to conduct self-initiated, independent R&D; and in 1997, DOE revised its LDRD program direction in DOE Order 413.2 providing clearer guidance on how LDRD funds may and may not be used.

Each of DOE’s nine multi-program national laboratories has an LDRD program. Funding for LDRD projects comes from existing program budgets. Historically, this has been accomplished by allowing each laboratory to assess its program’s budgets at a set rate of up to 6 percent and accumulate that money into an overhead account for its LDRD program.

DOE’s field offices oversee each laboratory’s LDRD program by approving the laboratory’s spending plans and making sure that projects comply with guidelines. DOE also approves each laboratory’s processes and procedures for selecting, reviewing, and tracking LDRD projects and requires annual reports from each laboratory. DOE conducts periodic reviews of the laboratories’ management that encompass the LDRD program.

Laboratories Have Spent Over $2 Billion on LDRD Projects Since Fiscal Year 1992

DOE’s nine multi-program laboratories have invested over $2 billion on LDRD projects since 1992, when the LDRD program was created. DOE’s three large defense laboratories account for a majority of all LDRD spending. Most LDRD funding is invested in research supporting the laboratories’ strategic plans and maintaining the skills and competencies necessary to carry out laboratory missions. In addition, laboratory managers told us they believe that LDRD projects help to attract new scientists and encourage others to explore cutting-edge science projects in order to maintain the “vitality” of the laboratories. The managers believe that LDRD projects also help to identify new mission areas consistent with DOE’s overall mission.

As shown in table 1, DOE’s nine multi-program national laboratories have spent over $2 billion on LDRD projects since fiscal year 1992.

DOE’s three largest multi-program national laboratories—Lawrence Livermore in California, and Los Alamos and Sandia in New Mexico— account for nearly three-quarters of all LDRD spending. These laboratories concentrate on national security issues and in recent years spent near the maximum amount authorized by the Congress for LDRD projects—no more than 6 percent of their budgets, except for fiscal year 2000 when the Congress limited the amount to 4 percent. By contrast, DOE’s other laboratories generally spend less than 4 percent of their budgets on LDRD projects. (See table 2.)

Each of the nine multi-program national laboratories established separate but similar LDRD categories of funding, using these as guides to selecting proposals. The number of categories ranged from one to five. In most laboratories, the largest category contained projects that aligned most closely with the laboratory’s strategic missions, such as the principal missions of national security at the defense laboratories and fundamental science at the Lawrence Berkeley National Laboratory. These types of LDRD projects tended to be larger and were expected to have nearer-term results. The second largest category was generally directed at building scientists’ skills and strengthening laboratory competencies. Generally, the laboratories target the smallest amount of funding to projects that are the highest risk and most cutting-edge as shown in the following examples: The Lawrence Livermore National Laboratory has three main categories of funding. Strategic Initiatives projects represent 27 percent of all LDRD funds, focus on research addressing national needs in support of the laboratory’s strategic vision, and are larger multidisciplinary projects. Exploratory Research projects received 67 percent of the funds, support the strategic vision and competencies building of programs and directorates across the laboratory, and are smaller than the Strategic Initiatives projects. Laboratory-wide projects received about 6 percent of the funds, are designed to encourage creativity of individual scientists in the pursuit of innovative research, and are funded at a maximum of $180,000. A category of funding that receives less than 1 percent of the laboratory’s LDRD funds— Definition and Feasibility Study projects—provides the seeds for new research ideas and are usually funded for less than 6 months and $50,000.

The Los Alamos National Laboratory has two categories of LDRD projects. Directed Research projects received about two-thirds of the funds, support the laboratory’s strategic plan, are typically multidisciplinary, and generally cost $1 million or more. Exploratory Research projects received about one-third of LDRD funds, are usually smaller and the most innovative, and generally cost $250,000 or less.

The Pacific Northwest National Laboratory has three categories for LDRD projects. Laboratory-level projects received about two-thirds of the laboratory’s LDRD funds and are for projects that directly align with the laboratory’s primary research areas, are generally multiyear and multidisciplinary, and cost from $100,000 to $250,000. Division- level projects received about one-third of the LDRD funds, are aimed at developing new ideas in a particular mission area, have intermediate and near-term mission relevance, and cost from $80,000 to $100,000. Level VI projects, which received a total of about $500,000 of the laboratory’s LDRD budget, are intended to support highly innovative ideas, and typically cost less than $60,000 each.

DOE and laboratory officials believe that the innovative nature of LDRD projects helps attract new scientists who can contribute to maintaining the vitality of the laboratories. Those officials focusing on national security issues believe that the LDRD program helps attract scientists who can eventually perform national security research work. They believe that because nuclear weapons science is not taught in colleges and must be taught within the defense laboratories, LDRD projects—and the scientists they attract—are vital for national security in the long term. For example, postdoctoral students represent a major source of future research staff at the laboratories, and most of them are hired to work on LDRD projects. Sixty-two percent of Sandia’s postdoctoral staff hired between 1996 and 1999 worked on LDRD projects.

DOE’s Laboratory Operations Board, comprising internal managers and external consultants, reported, in January 2000, that LDRD programs are vital in recruiting and retaining the best scientific talent into the laboratories. According to the Board’s report, from 1993 through 1998, 41 percent of LDRD-funded postdoctoral staff at Lawrence Livermore National Laboratory—a defense program laboratory—were subsequently hired by the laboratory. Officials from nondefense program laboratories also told us that LDRD projects are important for attracting and maintaining scientific talent in their laboratories. These laboratories, however, spend less on LDRD than defense program laboratories for a number of reasons, including that they conduct more basic science work as a primary mission within their regular programs.

LDRD Projects Met DOE’s Guidelines

All of the randomly selected LDRD projects we reviewed at the five laboratories we visited met DOE’s guidelines for selection. Additionally, DOE’s and the laboratories’ management controls were adequate to reasonably ensure that approved projects would likely meet DOE’s project-selection guidelines. DOE’s guidelines specify that LDRD projects must be in the forefront of science and technology and should include at least one of the following: Advanced study of hypotheses, concepts, or innovative approaches to scientific or technical problems.

Experimentation and analyses directed toward “proof of principle” or early determination of the utility of new scientific ideas, technical concepts, or devices.

Conception and preliminary technical analyses of experimental facilities or devices.

In addition, DOE’s guidelines generally require that LDRD projects should not last longer than 36 months, be supplemented by non-LDRD funds, be used to perform or supplement funding for DOE’s program work, or be used to fund construction for scientific projects beyond the preliminary phase of the research.

Projects Met DOE’s Guidelines

All LDRD projects we reviewed met DOE’s guidelines. These projects were new projects that were proposed for fiscal year 2000 funding. Most of these projects tested or analyzed a new or untested concept and were consistent with the laboratory’s strategic missions, as shown in the following examples: A Los Alamos project has a goal of advancing the state of fundamental simulation theory so that sophisticated simulation tools can be developed for use in decision-making in complex national security environments, such as critical national infrastructure analysis and military engagements. The project involves developing complex integrated simulation tools that will advance fundamental research in the areas of mathematical foundations of simulation, issues in implementing and computing for large simulations, statistical methods for simulation-based studies, and principles for simulation-based assisted reasoning. The project’s results are primarily targeted to have relevance in mobile communications, regional population mobility and transportation infrastructure, electrical power distribution networks and markets, epidemiological impacts on populations, and threat identification and targeting in urban terrain. In the project’s first year, among other things, demonstrations will focus on mobile telecommunications, transportation systems, and epidemiological impacts. The project is being done under Los Alamos’ Directed Research category and supports the laboratory’s strategic goals in threat reduction, high-performance computing, and modeling and simulation. The project was proposed for 3 years; $600,000 was approved for first-year funding.

An Argonne National Laboratory project is designed to fabricate magnetic wires of 20 nanometers (a nanometer is one-billionth of a meter) down to atom scale and study their static and dynamic magnetic properties. This project complements Argonne’s mission in the materials science area and could help define a new research direction for the laboratory. The ultimate goal is to create a new generation of miniaturization in electronics, including memories, transistors, logic elements, and sensors. The physical size of a magnetic system may affect its magnetic properties; this project proposes to study this phenomenon and make major inroads in understanding the fundamental issues of low-dimensional magnetic systems. These issues require a basic understanding of magnetic thin films and multilayers used in computing today as well as a deeper understanding of one- dimensional nanotechnology and synthesis of materials in this environment. The project managers plan to develop samples unprecedented in the study of lower-dimensional systems to better explore fundamental questions about the next-generation magnetism research. This project is being done under Argonne’s category of funding for more innovative projects—the Director’s Competitive Grants Program. The project was proposed for 2 years; $65,000 was approved for first-year funding.

A Sandia National Laboratories project aims to develop new scientific tools for addressing the threat of biological terrorism, which is consistent with Sandia’s national security mission. Currently, the ability to initially detect people exposed to a released agent relies on the outward appearance of symptoms, such as lethargy and fever. The goal of this proposed LDRD project is to show that earlier detection, based on cellular-level changes in the body through blood analysis, could be accomplished. The project also aims to develop techniques and models to detect and analyze infection without waiting for external symptoms. The results could reduce disease detection time from days to hours. The development of a rapid, highly sensitive screening mechanism would also have widespread application in the fight against other infectious diseases. This 9-month project costs $100,000 and falls under Sandia’s Development Reserve category, which is used for urgent science and technology needs or technical work related to development of a new program.

Adequate Management Controls Exist to Reasonably Ensure Compliance With LDRD’s Guidelines

DOE and laboratory management controls were adequate to reasonably ensure that projects approved would likely meet DOE’s project-selection guidelines. The key controls in place included using DOE’s guidelines to control and conduct the project-selection process, using individuals in the review and selection process with the appropriate skills and knowledge to evaluate the proposed projects, substantially segregating duties among individuals to help ensure that no one individual is likely to control the project-selection decision in a way that will violate LDRD’s guidelines, and ensuring appropriate DOE oversight and review of the results of the process.

All laboratories used DOE’s LDRD Order 413.2 as the primary guidance to review and select projects. Individuals involved in the review and selection of the projects had the requisite background and experience to provide credible review. Those individuals had wide-ranging scientific backgrounds—usually a Ph.D. in scientific research and practical experience in basic scientific research. When the subject matter of a project proposal was outside the knowledge base of the review team, the laboratories generally contracted with outside experts to provide reviews and recommendations on the merits of that proposal. In general, each laboratory established review panels comprising individuals from across the laboratory, which provided for diverse opinions to ensure that various points of view were brought to bear on the selection decision. In general, the review panels consisted of managers from directorates having knowledge in the project subject area, other subject matter experts, and managers from the LDRD program. Finally, DOE’s field offices, which are responsible for overseeing each laboratory, annually review the laboratories’ recommendations for projects to be funded and forward recommendations to headquarters for approval. While DOE’s reviews of proposed projects have resulted in clarifications and minor revisions in the proposals’ documentation, those reviews have rarely resulted in not funding proposed projects.

All laboratories we reviewed have separate and somewhat different review and selection processes linked to their distinct categories of funding for LDRD projects, but key elements of these processes are very similar. For example, the laboratories we visited initiate their annual LDRD selection process by asking research staff to propose potential projects, called “calls for proposals.” These calls ask for proposals that generally fit into a particular category of funding in the LDRD program. Reviewers for the individual categories of funding review those proposals and either reach consensus or vote outright on where each proposal should be ranked in terms of recommending it for funding. That recommendation is then generally given to the laboratory director, who selects the projects to be funded. The projects recommended for funding are given to DOE’s field offices for review and comment and ultimately forwarded to DOE’s headquarters for approval.

Opportunities Exist to Improve LDRD’s Performance Reporting

The LDRD program could improve its performance reporting. Each laboratory issues an annual LDRD report that includes performance reporting, but those reports do not use a common set of performance indicators. Additionally, the reports present performance information in varying formats, making it difficult to focus on the most relevant performance information. Laboratory managers told us there is no consensus on which performance indicators to use when reporting the results of their LDRD projects, nor is there an agreed upon reporting format. While the reports describe the accomplishments of individual laboratories, taken together, the laboratories’ reports do not provide aggregate performance information that DOE managers and the Congress could use to readily assess the overall value of the program.

The different performance indicators reported in each of the laboratories’ annual LDRD reports make it difficult to readily assess overall program performance for DOE’s LDRD program. Table 3 provides a summary of the performance information included in the annual LDRD reports published by the nine multi-program national laboratories in our review and demonstrates the lack of uniformity in reporting the LDRD program’s results across the laboratory complex.

In general, the laboratories maintain more detailed performance information than they report in their annual reports, but laboratory officials do not agree on a set of performance indicators that should be reported on for the program. Some pointed out that there is a significant difference between different types of publications. Refereed publications, for example, must go through an expert review process before they can be published. Also, certain publications have higher levels of difficulty and achievement and, therefore, significance. The same issue surrounds the tabulation of awards as performance indicators. Likewise, symposia, as well as other potential measures, carry different degrees of significance. Many suggested that success stories are the best measures of a project’s performance, particularly for basic research whose ultimate value may not be evident for a long time. Furthermore, they told us that projects viewed as unsuccessful with respect to their direct proposed goals might in fact have answered critical questions that paved the way for major breakthroughs in science.

In addition, we found that differences in how performance information is presented in the laboratories’ annual LDRD reports also make it more difficult to assess the overall value of the program. As indicated in table 3, we found that while some laboratories present performance information for individual projects, other laboratories present performance information in a summary fashion. Two contrasting performance-reporting styles can be found in Sandia National Laboratories’ and Lawrence Livermore National Laboratory’s annual LDRD reports. Sandia’s report provides an appendix entitled “Project Performance Measures,” which lists LDRD projects and catalogues outputs of the projects using 11 quantitative performance indicators and several qualitative indicators. In contrast, Lawrence Livermore’s LDRD report provides an appendix listing publications resulting from individual LDRD projects and describes—in summary format rather than on a project-by-project basis—several other quantitative performance indicators, including patents, awards, and permanent staff hired.

While the laboratories’ annual LDRD reports describe the accomplishments of individual laboratories, taken together, the laboratories’ reports do not provide aggregate performance information that DOE managers and the Congress could use to readily assess the overall value of the program. Aggregate, more-uniform performance reporting on the LDRD program could aid DOE managers, the Congress, and others in their oversight of the program.

Conclusions

In general, LDRD project-selection and review processes in place at DOE’s multi-program national laboratories are adequate to reasonably ensure compliance with DOE’s project-selection guidelines. Our review of randomly selected LDRD projects at laboratories found that they met DOE’s guidelines. However, our observations of the performance- reporting practices for the LDRD program lead us to conclude that performance reporting for the program could improve. By reporting aggregate, more-uniform performance information for the LDRD program as a whole, DOE managers and the Congress could more readily assess the overall value of the program.

Recommendation for Executive Action

To improve the Congress’s ability to make informed decisions on the value of the LDRD program, we recommend that the Secretary of Energy develop and annually report aggregate, more-uniform performance information for the LDRD program. This recommendation will require DOE’s National Nuclear Security Administration and the Office of Science, which are both accountable for laboratory performance, to work together and develop performance indicators that can be used to demonstrate accomplishments across all the laboratories.

Agency Comments

We provided a draft of this report to DOE for review and comment. According to representatives of the Office of Science responsible for the LDRD program, DOE agreed with our findings, conclusions, and recommendation. DOE also provided a number of clarifying comments, which we incorporated, as appropriate, in this report.

As arranged with your offices, unless you publicly announce its contents earlier, we plan no further distribution of this report until 15 days after the date of this letter. At that time, we will send copies to the Secretary of Energy and the Director, Office of Management and Budget. We will also make copies available to others on request.

Appendix I: Scope and Methodology

To determine how much the Department of Energy’s (DOE) multi-program national laboratories have spent on Laboratory Directed Research and Development (LDRD) projects since 1992 (when the LDRD program was created), we reviewed program information, including annual reports, budgets and other financial information provided by DOE and laboratory officials for the nine DOE multi-program national laboratories. These laboratories are Argonne National Laboratory, Brookhaven National Laboratory, Idaho National Engineering and Environmental Laboratory, Lawrence Berkeley National Laboratory, Lawrence Livermore National Laboratory, Los Alamos National Laboratory, Oak Ridge National Laboratory, Pacific Northwest National Laboratory, and Sandia National Laboratories.

Although DOE’s Ames Laboratory has a LDRD program, we excluded it from our review because Ames is not a multi-program national laboratory.

To determine if LDRD projects met DOE’s selection guidelines, we reviewed the procedures and processes for selecting LDRD projects at all nine of DOE’s multi-program national laboratories. We also tested the internal controls for project selection at five of those laboratories and the respective DOE offices responsible for oversight of the program, and randomly selected approved LDRD projects at those five laboratories. The five laboratories were Argonne National Laboratory, Lawrence Berkeley National Laboratory, Lawrence Livermore National Laboratory, Los Alamos National Laboratory, and Sandia National Laboratories. These laboratories include three of the largest multi-program national laboratories and represent 83 percent of DOE’s LDRD expenditures for the period reviewed. We also interviewed DOE’s field office officials responsible for the oversight of the program in the Albuquerque, Chicago, Idaho, and Oakland Operations offices. In addition, we interviewed officials responsible for the LDRD program in DOE’s headquarters Office of Science, Office of Defense Programs, and Office of Environmental Management.

To test the internal controls of the program, we evaluated the processes and procedures used to select LDRD projects. The internal control tests were designed to determine if adequate management control was built into the LDRD program to provide reasonable assurance that projects approved through the program comply with DOE’s guidelines for the LDRD program. We performed the internal control tests by examining the processes and procedures to ensure that the (1) people involved in the selection of the LDRD projects used the same guidance and selection criteria, (2) individuals involved in the selection of the projects had the appropriate skills and knowledge to evaluate the proposed projects, (3) duties in the project-selection process were segregated substantially among individuals so that no one individual would be likely to control the project-selection decision in a way that would violate LDRD guidelines, and (4) DOE oversight activities were adequate. To accomplish this, we obtained from the respective DOE officials and laboratory management officials, documentation and interview information on guidance provided to the LDRD project review and selection personnel on how to select LDRD projects for funding at each laboratory. We then obtained documentation on the LDRD processes and procedures for reviewing and selecting projects for funding at each of the five laboratories. This information included documentation on how proposed projects originate to final selections or other dispositions. We obtained documentation and interview information on which individuals participate in each phase of the process, their roles, and their backgrounds.

Using random number tables, we selected five projects from each of the five selected multi-program national laboratories’ projects approved for funding for fiscal year 2000— a total of 25 projects. Because each laboratory had more than one category of LDRD funding and to enable us to review projects within each of those categories, we randomly selected at least one project from each category of funding at each laboratory. We determined if each selected project met DOE’s project-selection guidelines and, to complement our internal control tests, we examined the elements of the processes, qualifications of reviewers, segregation of duties, and DOE’s oversight. For each of these projects, we reviewed the project proposal files, including documentation reflecting individual reviewers’ recommendations on the disposition of each case. We also interviewed the scientists who proposed each project and the laboratory officials responsible for reviewing the projects for selection to better understand the technical nature of the research and how that research meets DOE’s guidelines for LDRD projects. Interviews with selection officials also focused on determining if individuals involved in the selection of the projects had the appropriate skills and knowledge to evaluate the proposed projects and if the duties in the process were segregated so that no one individual would be likely to control the project-selection decision in a way that would violate LDRD’s criteria. We also interviewed DOE officials in headquarters and the field offices involved in the oversight process through which the projects were selected. While we cannot project the results of our analysis of LDRD projects to the universe of those projects, our analysis provides a snapshot of how internal controls were being applied, and additional confidence, at the five selected laboratories, in the results of our internal control testing overall.

To provide views on how the program might be improved, we relied on observations obtained throughout the course of our audit work.

We provided a draft of this report to DOE for review and comment. According to representatives of the Office of Science responsible for the LDRD program, DOE agreed with our findings, conclusions, and recommendation. DOE also provided a number of clarifying comments, which we incorporated, as appropriate, in this report.

Our review was performed from December 1999 through September 2001 in accordance with generally accepted government auditing standards.