National Laboratories: Are Their R&amp;D Activities Related to Commercial Product Development?
Released: 1994-12-22T00:00:00

Letter

Background

The Department of Energy’s multiprogram laboratories have had missions that are national in scope since their inception during World War II. The original laboratories—Lawrence Berkeley (Calif.), Los Alamos (N. Mex.), and Oak Ridge (Tenn.)—were established as government-owned, contractor-operated institutions to apply the productive capability of private industry to the development of atomic weapons. The weapons-development mission continued during the cold war, and six additional laboratories—Argonne (Ill.), Brookhaven (N.Y.), Sandia (N. Mex. and Calif.), Idaho Engineering (Idaho), Lawrence Livermore (Calif.) and Pacific Northwest (Wash.)—were created between 1946 and 1965 to foster civilian applications of nuclear technology. A 10th laboratory, the Solar Energy Research Institute, was designated a national laboratory in 1991 to expand federal energy research and development (R&D) capability in alternative energy sources, and it was renamed the National Renewable Energy Laboratory (Colo.). As a group, the 10 laboratories are known as the national laboratories.

As the laboratories’ experience and research capability evolved, mission emphases shifted among them. Sandia, Los Alamos, and Lawrence Livermore acquired primary responsibility for nuclear weapons research and development and the largest share of the laboratories’ funds. Responsibility for research in the environmental and biological, energy, and national security areas was distributed among all 10 laboratories to varying degrees. However, the Congress and DOE are reassessing this mission configuration.

Since 1980, the Congress has had an active interest, expressed in a series of laws, in seeing that more of the national laboratories’ outputs be put to commercial uses. Changing needs for defense technology resulting from the end of the cold war and concern with maintaining U.S industry’s competitiveness in global markets have led several members of Congress to open a public debate and propose new legislation that addresses the national laboratories’ missions, structure, and cooperation with industry.Among the alternatives being considered in the public debate are reducing all the laboratories’ budgets, consolidating or closing some of them, and redirecting their weapons development mission toward commercial product-related R&D in such areas as technology development for environmental restoration, energy, and high-performance computing.

Underlying these discussions are questions about the type of R&D activities the national laboratories are performing now, the nature and scope of their outputs, and their potential for assisting industry in bringing technology to the marketplace. This report is an effort to inform the debate by providing an empirical base for these questions, as a starting point for addressing the broader issues. It examines whether the balance of laboratories’ effort is in basic and applied research or research related to commercial product development, the distribution of the laboratories’ research outputs, and their potential for commercial application. Findings were based on a cross-section of the laboratories’ R&D activities for the period 1989-92. However, the objectives for most of the programs in the study population were initiated before the national laboratories’ legislative mandate for technology transfer in the National Competitiveness Technology Transfer Act took effect in late 1989. In most fields of R&D, more than 4 years are required for outputs to evolve after objectives have been established. Therefore, the commercial product-related effort we found is to be considered a baseline against which future activities and outputs can be measured.

Methodology

We began our work by developing a comprehensive description of current research activities in the 10 laboratories. We chose to survey the laboratories directly because we could find no sufficiently comprehensive existing documentation. We collected our data through a survey of the 10 laboratories’ research programs and the facilities and equipment that support them. The survey scope consisted of all major research programs and facilities with costs of at least $10 million, as well as special nominations by the laboratories themselves of other less costly programs and facilities.

These two criteria were designed to ensure that all large subprograms and smaller subprograms that were important to the laboratories’ missions would be included in our sample. This allowed us to describe the laboratories’ major research efforts. However, findings based on these criteria should not be considered representative of a laboratory’s entire research effort since the proportion of programs budgeted at less than $10 million can vary from one laboratory to another.

DOE’s Budget and Reporting System categories provided a common classification scheme for the laboratories’ 12 research programs, which permitted cross-laboratory comparisons of program characteristics. Research program and subprogram names are shown in table 1. The data we collected on these programs covered fiscal years 1989-92.

We conducted pilot tests of the survey methodology and data collection instruments at Brookhaven and the National Renewable Energy Laboratory. We then revised the instrument and administered one version to the remaining eight laboratories. After we processed the survey responses, we asked each laboratory to confirm by letter that our list of research programs and facilities was, in fact, complete.

The national laboratories engage in a wide range of defense and nondefense R&D-related activities. These range from generating hypotheses and testing fundamental science principles to assisting a potential user in adapting laboratory outputs to a production or service delivery system. To analyze the extent to which the laboratories are engaged in basic and applied research or research related to commercial product development, we divided their activities into five categories: basic research, applied research, development, technology transfer, and technical assistance.

Basic research is research undertaken primarily to gain fuller knowledge or understanding of a subject and to contribute to the knowledge base in the field of investigation. Applied research is research directed toward the practical use of knowledge or understanding of a subject to meet a recognized need. Development is research directed toward the production of useful materials, devices, systems, or methods, including the design and development of prototypes or processes. Development has some type of product as the output goal, but may conclude with a prototype rather than a usable good. Additional time, research, and testing are usually required to convert the prototype to a weapon or commercially viable product.

Because the national laboratories perform R&D only through the development stage, additional mechanisms and arrangements are required to achieve application of the laboratories’ outputs in the public or private sector. These activities are technology transfer and technical assistance. Technology transfer is the process that fosters the use of devices, processes, “know-how,” or scientific and technical information produced in a national laboratory by universities, private industry, or government agencies. It includes making potential users aware of the laboratories’ research outputs, assisting in their selection or use, and collaborating with representatives of private industry and public or nonprofit institutions to ensure that some of the laboratories’ outputs will have commercial or public applications. Technical assistance applies the laboratory’s expertise to practical problems but does not involve the use of a laboratory’s outputs. It is any form of assistance, other than financial, to a state or local government or a business, including publications, workshops, conferences, studies, or telephone consultation.

Development, technical assistance, and technology transfer are the three national laboratory research activities related to commercial product development. All five categories, already used by the laboratories but specially grouped for our analysis, constitute a natural framework that, together with DOE’s program classification scheme, allowed us to look at R&D-related activity across all 10 laboratories, using expenditures as a measure of activity.

Recognizing that the laboratories do not maintain records of their R&D expenditures in terms of our five categories, we asked managers of the subprograms in our study population to estimate, for each subprogram they managed, the proportion of funds expended in each of the five areas. Our analysis of R&D activity is therefore presented as percentages, not actual dollar values. To provide a context for considering our findings, we present in table 2 the fiscal year 1992 budgets for subprograms in the study population that were included in our analysis and the laboratories’ total budgets in fiscal year 1992.

We also examined the laboratories’ outputs. As output measures, we selected products of laboratory R&D that were clearly identifiable to our respondents and for which they were likely to maintain records. Since our study objective was to examine the balance among laboratory activities rather than their impact, we focused on outputs of R&D activity that occurred within the laboratories rather than their efforts at job creation or increased sales. Because of great variation in the size, scope, field of investigation and funding level of the subprograms in the study population, both within and among laboratories, we presented our findings as simple tabulations, rather than as standardized units. Use of a single measure for standardizing the outputs, such as dollar of funding per output, would have failed to account for variations among the subprograms on other dimensions. Moreover, because of the institutional complexity this variation represents, we interpreted our output findings very conservatively, treating them as measures of activity rather than indicators of performance. We looked at the outputs in two broad categories: (1) publications and reports and (2) outputs related to commercial product development. The outputs attributed to each category are described in the Principal Findings section.

Finally, we looked at three other indicators—the formation of cooperative R&D agreements, R&D effort devoted to critical technologies, and program managers’ assessment of their on-going research—to gauge the laboratories’ potential for commercial product development.

We conducted our review in accordance with generally accepted government auditing standards.

Principal Findings

Balance Among R&D-Related Activities

To examine the balance of the national laboratories’ current R&D-related activities, we analyzed the distribution of laboratory expenditures for R&D within and among laboratories and research programs. For the 10 laboratories overall, R&D-related activity was almost evenly divided between basic and applied research on the one hand, and research related to commercial product development on the other. Approximately 8 percent more of the effort was devoted to R&D activities related to commercial product development, as shown in figure 1. More applied research than basic research was conducted: 27.2 percent versus 17.4 percent. Among research activities related to commercial product development, most (30.9 percent) was development, but more activity was devoted to technical assistance (14.4 percent) than technology transfer (7 percent). Thus, R&D-related activity directly targeted on potential commercial applications of the laboratories’ outputs currently constitutes the smallest proportion of the laboratories’ R&D-related effort. Despite its small size, however, this level of effort exceeds the laboratories’ minimum statutory requirement for technology transfer activity.

These overall percentages, however, mask major differences among the laboratories with regard to R&D funding distribution. (See table IV.1 in appendix IV.) Four laboratories—Argonne, Lawrence Berkeley, Oak Ridge, and Brookhaven—spent 25 percent or more of their research funds on basic research. These laboratories account for over half (59.3 percent) of the total national laboratory research budget that is spent on basic research. (See table IV.2.) Los Alamos spent 19.4 percent of its R&D funds to support its mission to perform “basic research in selected disciplines that help maintain an outstanding science and technology base.” Only about 10 percent or less of the laboratory research budget was spent on basic research at the other laboratories. The energy research program accounted for the greatest proportion of funds spent on basic research, both within and among research program areas. (See tables IV.3 and IV.4.)

As table IV.1 shows, four laboratories—Oak Ridge, Pacific Northwest, Lawrence Livermore, and Los Alamos—spent 29 percent or more of their research funds on applied research. Among the 10 laboratories, Lawrence Livermore and Los Alamos accounted for almost half (47.9 percent) of applied research expenditures. (See table IV.2.) Most applied research was supported by programs in the areas of defense, energy research, and work for others. (See table IV.4.)

As noted earlier, most of the laboratories’ development work, the most product-oriented of R&D activities, was devoted to defense, rather than nondefense, research. Almost three-quarters (71.5 percent) of all the laboratories’ development research was conducted at Lawrence Livermore, Los Alamos, and Sandia. (See table IV.2.) In turn, the largest share of development research was performed in the defense and nuclear energy programs. (See tables IV.3 and IV.4.) Therefore, while it is true that across the 10 laboratories, a greater proportion of research funding was devoted to activities more closely related to commercial product development than to basic and applied research, most of these funds currently support defense research. To determine whether this research will have commercial opportunities for use, we examined the national laboratories’ outputs.

National Laboratories’ R&D Outputs

A second measure of the type of effort in which the national laboratories are engaged—as between basic and applied research or research related to commercial product development—is output. The laboratories produce two major types of outputs: (1) publications and reports, and (2) outputs related to commercial product development. Table 3 shows that, across a 4-year period, most of the laboratories’ outputs were publications and reports. This finding was expected because reports and publications are the primary mechanisms for diffusion of R&D findings, and they are prepared at all stages of the R&D process. Reports, conference papers, and published articles, which can be produced more quickly than books and book chapters, substantially outnumber the latter.

As we discussed above, a slightly higher percentage of the laboratories’ expenditures was devoted to R&D activities related to commercial product development than to basic and applied research; nevertheless, few of their outputs were commercial product-related. Prototype devices and materials, algorithms, and software are the largest number of outputs in this group. These outputs tend to arise from the development stage of the R&D process, which often occurs several years before production of a marketable or usable good. Not all outputs of the development stage will, of course, achieve commercial application.

Most of the prototype devices and materials, algorithms, and software, as indicated in tables V.1 and V.2 in appendix V, were produced at the weapons laboratories, and most were funded by DOE’s defense program. Other outputs laboratory managers identified as commercial products or commercial processes also tend to arise from the development stage. Although they will require a substantial additional investment before they are ready to market, these products or processes will more likely result in actual commercial applications because a potential commercial use has already been identified. Most of these outputs were produced by Los Alamos, Sandia, and Pacific Northwest, and the defense program supports most of the research that has led to these outputs. The point here is that although defense-funded R&D has produced more outputs that could lead to commercial products, whether these outputs will achieve commercial application is still unknown.

Patent applications may be submitted for inventions throughout the entire R&D process, but a license is usually acquired only when a decision to market a technology has been made. The number of licenses awarded, therefore, is a stronger measure of output activity related to commercial product development than the number of patents. A trend in the data indicative of the laboratories’ production of outputs related to commercial product development is the increase in the number of licenses awarded during fiscal years 1989 through 1992. (See table 3.) In fiscal year 1992, Sandia and Pacific Northwest awarded the most licenses, and most licensed outputs were supported by defense program research. (See tables V.1 and V.2.)

We expected to find that most commercial product-related outputs were supported by research programs that spent most of their R&D funds for development. However, the R&D expenditures of those programs that supported the most outputs related to commercial product development covered the range of R&D activities. We found that in fiscal year 1992, four research programs—energy research, conservation and renewable energy, defense, and work for others—supported most of the commercial product-related outputs of all types and that, over 4 years, commercial product-related output production had been increasing each year in three of the programs, as shown in figure V.1. We also found that in fiscal year 1992, the largest proportion of expenditures in the defense and conservation and renewable energy programs was for development.

As expected, the defense and conservation and renewable energy programs supported more of the outputs specifically designated as commercial products and processes than any of the 10 other research programs. However, in looking more closely at these four programs, we found some interesting differences. Work for others, which supports more commercial product and process-type outputs than eight other programs, devoted a slightly higher proportion of R&D expenditures to applied research than to development. But in energy research, which supports more commercial product- and process-related outputs than nine other programs, the largest proportion of expenditures was for basic research. (See table V.3.)

Laboratories’ Potential for Commercial Product Development

We looked at three indicators of the national laboratories’ potential for commercial product development: (1) formation of cooperative research and development agreements; (2) proportion of R&D expenditures in critical technology areas; and (3) research program managers’ judgments about their programs’ outputs. Of these three, the most frequently used indicator of the national laboratories’ potential for commercial product development is the formation of CRADAs. Here we found a major increase in activity. The national laboratories reported that from fiscal year 1989 through 1992, they entered into 196 CRADAs. Among programs in the study population in operation all 4 years, the number of new CRADAs formed increased from 17 in fiscal 1989 to 130 in fiscal 1992.

Sandia and Oak Ridge laboratories were most active in entering into CRADAs. (See table VI.1. in appendix VI.) Most were formed for research sponsored by programs in the defense and conservation and renewable energy areas. (See table VI.2.) The greatest increase in CRADA formation occurred at Sandia, where 74 CRADAs were in effect in fiscal year 1992. Fifty-three of the CRADAs effective in fiscal year 1992 were sponsored by the defense program technology transfer initiative at Sandia. This subprogram was initiated in June 1990 to identify opportunities for commercializing technologies produced by DOE-funded defense research activities in such areas as advanced manufacturing and precision engineering, materials and processes, advanced microelectronics and photonics, and computer architecture and applications.

Although the national laboratories do not yet have a legislative mandate or mission for research in the critical technologies, their research program managers reported that 74.1 percent of R&D expenditures are devoted to work in critical technology areas. This research was distributed over the 22 areas identified by the National Critical Technologies Panel, with the greatest concentration in energy technologies (13.6 percent); pollution minimization, remediation, and waste management (8.8 percent); computer simulation and modeling (6.7 percent); and materials synthesis and processing (6.2 percent). (See table VI.3.)

Work in these critical technology areas was distributed broadly among the laboratories and research programs. Five laboratories—Argonne, Lawrence Berkeley, Oak Ridge, Idaho, and Lawrence Livermore—devoted approximately 20-30 percent of their research funds to energy technologies. (See table VI.3.) Pacific Northwest expended the greatest proportion of R&D funds (41.3 percent) on pollution minimization technologies. Idaho and Lawrence Livermore were most active in computer simulation and modeling. Oak Ridge and Los Alamos devoted the greatest percentage of effort to materials synthesis and processing. As a group, the laboratories devoted approximately three-fourths of their R&D expenditures to research in critical technology areas, but Sandia and Los Alamos expended only about half of their resources on critical technologies research. All of the research programs sponsored research in critical technologies to some degree, with the least effort expended by environment, safety, and health. (See table VI.4.)

Finally, laboratory research program managers’ judgments about their research programs’ potential for commercial product development were optimistic. Among the subset of all national laboratory programs with a potential for commercial product development, almost 58 percent of the program managers expected that development to occur within 5 years of fiscal 1992. (See figure VI.1.) An additional 27.6 percent reported that their program has the potential for commercial product development within 5-10 years.

Conclusions

As of 1992, the national laboratories spent slightly more than half of their R&D funds on research related to commercial product development. However, most of this R&D was performed at the weapons laboratories and was supported by the defense and nuclear energy programs. Analysis of the outputs produced by the national laboratories indicated that defense-funded research produced more outputs—prototype devices and materials, algorithms, software, and other products and processes that have an identified commercial application—that are precursors to marketable goods, but at this point, whether they will achieve commercial application is not known.

Moreover, three indicators of the laboratories’ potential for commercial product development—CRADA formation, critical technology research, and program managers’ expectations for commercial potential—showed that some activity was occurring. CRADA formation was increasing, but these arrangements ensure only that collaboration between the laboratories and industry will occur, not that a commercial product will be generated. Almost three-fourths of the laboratories’ effort was devoted to research in critical technology areas, but achievement of commercial application will not be known for several years. Over half of the managers of research subprograms that have commercial product potential expected innovations to arise within 5 years, but these expectations must be considered “best educated guesses.” While we can conclude, therefore, that the national laboratories’ were engaged in slightly more research related to commercial product development than basic and applied research, it is too early to determine whether this activity will produce technologies with commercial uses.

Agency Comments

We requested comments on a draft report and received a response from DOE and the 10 national laboratories. DOE questioned the definitions and categories we defined to analyze the laboratories’ R&D-related activities and our finding that the laboratories perform slightly more research related to commercial product development than basic and applied research. DOE also thought that this study should have examined additional institutional factors, including the R&D activities of other agencies, and should have used data maintained by DOE headquarters rather than surveyed the laboratories for data.

We note that the definitions for R&D-related activities we employed are derived from a Congressional Budget Office study of the federal R&D enterprise, our study of the Technology Transfer Act of 1986, and expert opinion. We also disagree with DOE’s proposed broader scope for this study because it exceeds our study objective and would have required additional data collection and analyses that are beyond the study scope. Furthermore, our exploration of data available at DOE headquarters found that it was not adequate to satisfy our information needs.

Eight laboratories agreed with the report’s objective, analyses, and conclusions. However, one of this group, Lawrence Berkeley, thought that the relationship of commercial product development to the broader needs of industry and the nation should have been addressed in the study. Two of the laboratories raised issues about study methodology. Idaho believed that a greater proportion of the budget for its subprograms should have been included in the study sample. Oak Ridge questioned the effect of the study’s sampling methodology on output findings for the laboratory and the definition of the category called outputs related to commercial product development.

Lawrence Berkeley said that we had overlooked an important issue. The laboratory thought that the study should have included an examination of the relationship of the national laboratories’ role in commercial product development to the broader needs of industry and the nation. We agree that this issue is important to address as part of the public debate about the laboratories’ missions and structure. However, we disagree that it should have been examined in this report, which focuses on establishing an empirical baseline of national laboratories’ activities.

DOE’s Idaho Operations Office responded for Idaho National Engineering Laboratory. The Idaho Operations Office said that the budget figure reported for Idaho subprograms included in the study sample should have been higher. We did not agree to revise Idaho’s budget figure, because to do so would have violated the study methodology used to sample programs at other laboratories.

Oak Ridge took the position that most of its commercial product-related outputs were produced by subprograms that were not selected in the study sample because they were funded at less than $10 million. The laboratory expressed concern that the subprograms we sampled produced only 7 percent of its commercial product-related outputs while representing 73 percent of its overall budget. Oak Ridge based this position on summary output data for the entire laboratory and sampled subprograms that laboratory representatives had tabulated. Again, we could not include the output data for Oak Ridge’s unsampled programs in our analyses without violating the sampling methodology. We also had some questions about the large number of outputs the Oak Ridge analysis ascribed to unsampled programs.

Oak Ridge also thought that our definitions for these outputs equated the laboratories’ development work with commercial product development. We disagree. The definitions we used make it clear that the laboratories were not expected to produce commercial products. Our conclusion reiterates that the laboratories’ outputs related to commercial product development are “precursors to marketable goods” and that “whether they will achieve commercial application is not known.”

We provide a more detailed discussion of all these comments and our response in appendixes VII through XVII.

As agreed with your offices, we plan no further distribution of this report until 30 days from its date of issue, unless you publicly announce its contents earlier. We will then send copies to interested parties, and we will also make copies available to others upon request.

If you have any questions or would like additional information, please call me at (202) 512-3092. Other major contributors to this report are listed in appendix XVIII.

Description of the National Laboratories

The descriptions of the national laboratories are adapted from the 5-year institutional plans that the laboratories update and issue annually and from U.S. Department of Energy, Multiprogram Laboratories, 1979 to 1988, A Decade of Change (Washington, D.C.: Apr. 1990).

Argonne National Laboratory

Argonne was established in 1946. The University of Chicago operates the laboratory, which develops and operates national facilities for use by university, industry, and national laboratory groups; performs basic research, technology-directed research and technology evaluations; and conducts technology transfer through cooperative research, and development agreements, sponsored research, staff exchanges, and licensing of intellectual property or through the formation of new firms by the laboratory’s Arch Development Corporation.

The laboratory’s basic research effort includes experimental and theoretical research on fundamental problems in the physical, life, and environmental sciences to advance scientific understanding and support energy technology development. Argonne’s technology-directed research includes conceptualization, design, and testing of advanced fission reactors and other technologies for power applications in both the civilian and defense sectors and investigations of strategies for overcoming materials, chemical, and electrochemical barriers to the development of these technologies. Argonne also supports DOE and, where appropriate, other federal agencies in characterizing and evaluating nationally important projects and technology options in terms of their environmental cost or other implications.

Lawrence Berkeley Laboratory

Lawrence Berkeley, founded in 1931 as the Radiation Laboratory by Ernest Orlando Lawrence of the University of California at Berkeley, was one of the original national laboratories. It was funded under government contract in 1942. The University of California, which operates the laboratory, renamed it the Lawrence Radiation Laboratory after his death in 1958, and later called it Lawrence Berkeley. The laboratory conducts a wide range of interdisciplinary research with core competencies in biosciences and biotechnology; particle and photon beams; advanced detector systems; characterization and synthesis of materials; chemical dynamics, catalysis, and surface sciences; advanced techniques for energy supply and energy efficiency; and environmental assessment and remediation. It performs research in the energy, physical, and life sciences; develops and operates national experimental facilities; fosters industry’s interactions with the laboratory’s research programs; and offers scientific and engineering education programs.

The laboratory’s work in the energy sciences includes applied science, such as the energy efficiency of buildings; chemical sciences, such as the structure and reactivity of transient species; earth sciences, including geophysical imaging methods, isotopic geochemistry and physicochemical process investigation; and materials sciences, such as advanced ceramic, metallic, and polymeric materials for electronic, magnetic, catalytic, and structural applications. Accelerator and fusion research, nuclear science, and physics are pursued in the general science area. Lawrence Berkeley’s work in the life sciences includes cellular and molecular biology, chemical biodynamics, and research medicine and radiation biophysics. This work is supported by the laboratory’s scientific and technical resources in the areas of engineering, information and computing sciences, and occupational health.

Oak Ridge National Laboratory

Oak Ridge was one of the original national laboratories. Now operated by Martin Marietta Energy Systems, Oak Ridge was established in 1943. The laboratory’s R&D activities are focused on basic and applied research, technology development, and other technological challenges in areas that include energy production and conservation technologies; experimental and theoretical research in physical, chemical, materials, computational, biomedical, earth, environmental, and social sciences; the design, building, and operation of unique research facilities for the benefit of university, industrial, and other federal agency and national laboratory researchers; and the development of environmental protection and waste management technologies. Oak Ridge also performs technology transfer and offers educational services from the preschool through the postdoctoral level.

Pacific Northwest Laboratory

Pacific Northwest was established in 1965. Battelle Memorial Institute now operates the laboratory, which performs scientific research and rapid technology development and deployment to meet national needs. Laboratory efforts include molecular science, hazardous waste characterization, global environmental studies, subsurface science, biological systems, technical support for environmental policies and procedures, federal infrastructure modernization, national security technology, energy-efficient methods, advanced analytical methods, materials research, magnetic fusion research, civilian nuclear waste management, technical support for nuclear power plant operation, space exploration technology, fossil fuel technology, renewable energy sources, energy policy analysis, and surveillance and oversight of operations at its Hanford site.

Idaho National Engineering Laboratory

This laboratory was established in 1949. Three contractors operated the laboratory during the time period of our study: Westinghouse Idaho Nuclear Co., Rockwell-INEL, and EG&G Idaho. The laboratory’s areas of primary emphasis are nuclear reactor technology R&D, defense production-related support, waste management and environmental restoration analysis, advanced energy production technology development, and research and development on energy and environmental issues, including performance testing of industry-developed electric vehicles, small hydropower and geothermal power production, and fossil energy research. Idaho also offers educational activities and performs technology transfer.

Lawrence Livermore National Laboratory

Lawrence Livermore was established in 1952. The University of California operates the laboratory, which serves as a national resource in science and engineering, focused on national security, energy, environment, biomedicine, economic competitiveness, and science and mathematics education, with a special responsibility for nuclear weapons. National security has traditionally been a special focus of the laboratory’s research and development effort. Lawrence Livermore’s major areas of activity have included research, development, and testing for all phases of the nuclear weapons life cycle; strategic defense research; arms control and treaty verification technology; inertial confinement fusion; atomic vapor laser isotope separation; magnetic fusion; other energy research; research in biological, ecological, atmospheric, and geophysical sciences; charged-particle beam and free-electron laser research; advanced laser and optical technology applications; technology transfer; and science education. The laboratory also participates in human genome research as part of a nationally directed initiative.

Los Alamos National Laboratory

Los Alamos, one of the original national laboratories, was established in 1943 and is operated by the University of California. Ensuring the nation’s deterrence capability through nuclear weapons technology is the laboratory’s primary focus. Los Alamos’ major R&D activities include research, design, development, engineering, and testing of nuclear warheads; maintenance and enhancement of the weapons technology base and warhead stockpile management; research, development, and testing support for advanced nuclear directed-energy concepts; nuclear materials R&D for the nuclear weapons program; nonnuclear strategic defense R&D activities; advanced conventional munitions development and simulation; verification and safeguards R&D; vulnerability, lethality, effects, and countermeasures research; advanced defense technologies; intelligence activities involving hardware analysis and technology security; weapons and energy technology systems studies; and R&D in nonnuclear energy and technology areas. The laboratory’s basic research activities in defense and energy areas include atomic and molecular physics, bioscience, chemistry, computational science and applied mathematics, geoscience, space science, astrophysics, materials science, nuclear and particle physics, plasma physics, fluids, and particle beams. Los Alamos also performs technology transfer and offers science and engineering education programs.

Sandia National Laboratories

Sandia was established in 1949 under an agreement with AT&T to operate the laboratory for the government as a public service on a nonprofit basis. AT&T stepped out of this role in 1993. A contract was recently awarded to Martin Marietta Corporation to operate the laboratories. Sandia’s major areas of effort are nuclear weapons, arms control and treaty verification, environmental restoration and waste management, energy supply and conservation, advanced conventional military technologies, and other programs in the national interest. The laboratories’ R&D activities in these areas include research, development, and engineering associated with advancing nuclear explosives to integrated, functional weapons for Department of Defense weapon delivery systems; other defense programs, including development of verification and control technologies to support arms reduction and concepts and systems for the safeguarding and security of nuclear materials; research, development, and engineering for hazardous waste removal, minimization, and remediation; and nonnuclear energy research in energy efficiency, recovery techniques, conversion technologies, alternative energy sources, characterization of environmental change phenomena, environmental restoration technologies, and basic energy sciences. Sandia also conducts technology transfer and offers mathematics and science education opportunities.

Brookhaven National Laboratory

Brookhaven was established in 1947 by a group of nine universities to facilitate their mutual access to large-scale research facilities, particularly in nuclear science. The laboratory is operated by Associated Universities, a corporation governed by a board of trustees representing the original nine universities as well as other universities, research institutions, and industrial organizations. Brookhaven’s primary role is to conceive, design, build, and operate large-scale, complex facilities for scientific research and to conduct basic and applied research in energy-related physical, life, and environmental sciences. When feasible, Brookhaven makes its laboratory facilities available to state and federal agencies, universities, and private industry. The laboratory’s major areas of R&D are high-energy and nuclear physics; basic energy sciences emphasizing research on biological, chemical, and physical phenomena underlying energy-related transfer, conversion, and storage systems; life sciences, nuclear medicine, and medical applications of nuclear techniques; and a broad span of applied programs that draw on the laboratory’s unique capabilities. Brookhaven makes all useful results and knowledge obtained from its research activities available to private industry. Brookhaven also performs technology transfer and offers science and engineering education programs.

National Renewable Energy Laboratory

The former Solar Energy Research Institute was designated a DOE national laboratory in 1991 and renamed the National Renewable Energy Laboratory. The focus of the laboratory’s effort is on developing competitive renewable energy and related technologies and facilitating their commercialization. The laboratory’s R&D activities include basic and applied research, exploratory and advanced development and other activities in renewable energy and related technologies; analytic studies and technology evaluations; and collaborative R&D with universities and industry. The laboratory also manages subcontracted R&D on behalf of DOE and serves as a source of scientific and technical information on renewable energy.

National Laboratory Inventory Research Program Data Collection Instrument

Please return the completed questionnaire in the enclosed envelope within 10 working days of receipt. the event that the enclosed envelope is misplaced, please mail the questionnaire to: Nancy Briggs, Ph.D Project Manager U.S. General Accounting Office Program Evaluation and Methodology Division Room 5853 441 G Street St., N.W. Washington, DCThank you in advance for your cooperation and assistance in addressing an issue of such critical importance to the nation.We will send a report on the analysis of the information to the Congress and you.

I-2 The definitions listed below are included to provide a common frame of reference for responding to the survey.

Applied Research: meet a recognized need.

Basic Research: contribute to the knowledge base in the field of investigation.

Capital Equipment Budget: and general purpose equipment.

Cooperative Research and Development Agreement (CRDA): of fostering technology transfer from the federal domain to the private sector.

Development: including the design and development of prototypes and processes.

Research directed toward the production of useful materials, devices, systems, or methods, Facility: building or defined structure, some area within a structure, or a defined area not confined to a structure (for example, a testing area).

An entity that comprises the equipment used for research programs.

Laboratory: perform research and development. testing areas), buildings, human resources, research programs, and equipment.

A group of facilities owned, leased, or otherwise used by the U.S. Department of Energy to A laboratory consists of land (including, but not limited to, remote The primary scientific and technical research programs that a laboratory pursues.

Operating Budget: including salaries and wages, expendables and overhead.

For a national laboratory this comprises research and development program costs, Research and Development: understanding of the subject under investigation; practical use of knowledge; or the production of materials, devices, systems, or methods.Research and development includes basic research, applied research, and development.

Intensive, systematic study directed toward fuller scientific knowledge or One of several broad areas of research activity within a laboratory’s mission.

Technology: research and development process.

Devices, processes, "know how," or scientific and technical information produced through the Technology Transfer: technical information produced in a federal laboratory by universities, private industry, or government agencies, whether national (federal, state, or local) or foreign.

Total Budget: costs.

For a national laboratory, this includes operating, capital equipment, and construction User Facility: reimbursed basis by investigators from private industry, academic institutions, or state and local government agencies.

A federal laboratory facility available for use either free of charge or on a cost- I-3 1. operating budget during fiscal year 1992 and which is on-going for fiscal year 1993.

Please identify the defense subprogram for which you are reporting.

Report only for a subprogram funded by the laboratory’s (Check only one.)

GB-02 Inertial Confinement Fusion (Guidance)

GB-02 Inertial Confinement Fusion (Required)

GC Verification and Control Total Other (Please specify)

U.S. Department of Energy Budget and Reporting and System code.

I-4 Please describe the subprogram’s research objectives and the major scientific and technical areas the subprogram addresses.

What is the time period of performance for this subprogram? (Write in the subprogram start date and end-date below). (month) (Day) (Year) (month) (Day) (Year)

I-6 4. objectives?

How important, if at all, is each of the research and development-related activities listed below to your subprogram (Check the relevant space.) (1) (2) (3) (4) (5) (6) f. private firms or industrial organizations g. laboratory to U.S. government organizations h. laboratory to foreign government organizations i. Transfer technology from this laboratory to U.S.firms or industrial organizations j. laboratory to foreignfirms or industrial organizations Other (Please specify)

Basic Research: knowledge base in the field of investigation.

A study undertaken primarily to gain fuller knowledge or understanding of a subject and to contribute to the A study directed toward the practical use of knowledge or understanding of a subject to meet a recognized need.

Development: development of prototypes and processes.

Research directed toward the production of useful materials, devices, systems, or methods, including the design and I-7 5. 1989-92 for each of the research and development-related activities listed below? "N/A" if the program was not in operation in given year.)

In your estimation, what percent, if any, of your total research subprogram budget was expended each year during fiscal years Write(Annual total should equal 100 percent. d. assistance to government agencies f. assistance to private firms or industrial organizations g. this laboratory to U.S. government organizations h. this laboratory to foreign government organizations i. this laboratory to U.S.firms or industrial organizations j. this laboratory to foreignfirms or industrial organizations Other (Please specify)

Basic Research: knowledge base in the field of investigation.

A study undertaken primarily to gain fuller knowledge or understanding of a subject and to contribute to the A study directed toward the practical use of knowledge or understanding of a subject to meet a recognized need.

Development: development of prototypes and processes.

Research directed toward the production of useful materials, devices, systems, or methods, including the design and I-8 6. 1989-92, which of these technologies did the research of this subprogram support? research subprograms budget was spent each year for research about these technologies? estimated percentage for activities checked.)

We identified the technologies listed below using the Report of the National Critical Technologies Panel. During fiscal years In your estimation what percentage of the Write in the(Check all that apply.

Pollution minimization, remediation, and waste management Other (Please specify)

I-10 7. 1989 to 1992?

How many of the research and development-related outputs listed below did your subprogram produce each year from fiscal year Write "N/A" if the program was not in operation in a given year.)

Technical and scientific reports/monographs for internal use only Technical and scientific reports/monographs for release to others outside the laboratory Papers for presentation at professional conferences Other (Please specify)

I-11 8. dollars in millions.

How much royalty income was earned each year during fiscal years 1989-92 by technologies supported by this subprogram? (Write "N/A" if the program was not in operation in a given year.)

I-12 10. product development (Check only one).

During what time period, if at all, does this research subprogram have the potential for industrial application or commercial Immediate future (Less than 5 years)

Short-term future (Next 5-10 years)

Long-term future (Over 10-20 years)

Very long-term future (Over 20 years)

I-13 Is any university or private industry working in cooperation with your research subprogram?

If yes, please provide the following information: Research topic of cooperative effort: ____________________________________________________________ Please also indicate if this is a U.S. or foreign government, university or firm. (Check the relevant space.) (1) (2) (3)

If there is more than one organization working in cooperation with your subprogram, please photocopy this page and provide this information for all organizations.

I-14 12. that in your view are working on scientific and technical problems similar to those addressed by this research subprogram.

Contact person’s name and telephone number a. a. b. b. c. c. a. a. b. b. c. c. a. a. b. b. c. c.

I-15 13. research institutions who are familiar with the scientific and technical aspects of your research subprogram as well as the facilities and equipment that support it. Please do not list anyone who is affiliated with programs in the U.S. Department of Energy.

I-16 (Dollars in millions)

14. budget during fiscal years 1989-92?Show dollars in millions. (Write the dollar amount in each column. Write "N/A" if the subprogram was not in operation in a given year.)

What was your research subprogram’s total annual a. b.

Other (Please specify) 15. fiscal years 1989-92? (Write "N/A" if the subprogram was not in operation in a given year. percent.)

I-17 15. following sources during fiscal years 1989-92 (Annual total should equal 100 percent.) (Continued)

What percentage of the research subprogram’s total annual budget was contributed by each of the I-18 16. How many workers are employed annually by your research subprogram in each of the following job categories and what is the total employment?Please provide both the number of full-time personnel and the number of full-time equivalent (FTE) staff years for each year during fiscal years 1989-92. a. involved in research) b. Scientists, engineers, and other researchers (including research administrators directly involved in research) c. Technicians supporting research (through testing, inspection, maintenance, or construction of research equipment, computer programming) d. Clerical maintenance and other support personnel e. Other (Please specify)

Survey Methodology

Survey Response

The research program survey population was enumerated by applying selection criteria to each laboratory’s research programs. After processing the surveys, we sent the laboratories a letter requesting confirmation that our list of research programs and subprograms was complete. In response to our letter, the laboratories confirmed a total of 252 research subprograms. The laboratories returned a total of 247 data collection instruments, for a survey response rate of 98 percent.

Data Quality Issues

The data contained in this report are results of analyses of national laboratory program managers’ responses to questions 5, 6, 7, 9, and 10 in part I of the national laboratory inventory. These responses represent program managers’ judgments or self-reports about question elements, as follows. We made no attempt to validate these responses through independent sources.

Question 5

Responses are research program managers’ best estimates of the proportion of the total program budget expended for each R&D-related activity. Although they had our definitions for key R&D-related activities listed in the data collection instrument, their responses also may reflect their own understanding of terms such as basic research, applied research, or technical assistance.

Question 6

Responses are research program managers’ best estimates of the proportion of the total program budget expended for research in critical technology areas. The response categories in the question are the critical technologies identified by the National Critical Technologies Panel. Some overlap may exist among these categories because they were not identified for research measurement purposes. The Panel’s critical technology categories were used in this question to determine the congruence between research already being conducted at the national laboratories and the research needs articulated by a congressionally mandated body.

A few responses submitted for this question summed to more than 100 percent. These responses were prorated to include them in the calculation of mean percent expenditures for R&D in critical technologies.

Question 7

Responses are research program managers’ reports of research program outputs. The responses concerning commercial products and commercial processes are judgments made about research outputs that have reached only the precompetitive stage of the R&D process.

Question 9

Responses are research program managers’ reports about CRADAs in effect through the end of fiscal year 1992.

Question 10

Responses are research program managers’ judgments about potential industrial application or commercial product development for outputs of their research program over a 20-year planning horizon. The size of research subprograms in the study population varied; thus, managers were considering outputs of one or more research activities in making their assessments.

Balance Among the National Laboratories’ R&D Activities

Data concerning the distribution of the national laboratories’ expenditures among R&D-related activities, by laboratory and by program, are presented below.

Table IV.1: Mean Percent Expenditures for R&D-Related Activities Within LaboratoriesANL = Argonne National Laboratory LBL = Lawrence Berkeley Laboratory ORNL = Oak Ridge National Laboratory PNL = Pacific Northwest Laboratory INEL = Idaho National Engineering Laboratory LLNL = Lawrence Livermore National Laboratory LANL = Los Alamos National Laboratory SNL = Sandia National Laboratories BNL = Brookhaven National Laboratory NREL = National Renewable Energy Laboratory Subprogram expenditures for activities other than those listed above, such as training graduate students and postdoctoral fellows or safety procedures.

Table IV.2: Mean Percent Expenditures for R&D-Related Activities Among Laboratories

ANL = Argonne National Laboratory LBL = Lawrence Berkeley Laboratory ORNL = Oak Ridge National Laboratory PNL = Pacific Northwest Laboratory INEL = Idaho National Engineering Laboratory LLNL = Lawrence Livermore National Laboratory LANL = Los Alamos National Laboratory SNL = Sandia National Laboratories BNL = Brookhaven National Laboratory NREL = National Renewable Energy Laboratory Subprogram expenditures for activities other than those listed above, such as training graduate students and postdoctoral fellows or safety procedures.

Table IV.3: Mean Percent Expenditures for R&D-Related Activities Within Programs

ER = Energy Research CE = Conservation and Renewable Energy ES&H = Environment, Safety and Health NE = Nuclear Energy DP = Defense Programs NPR = New Production Reactors ERWM = Environmental Restoration and Waste Management FE = Fossil Energy CRWM = Civilian Radioactive Waste Management PPA = Policy Planning and Analysis INT = Intelligence WFO = Work for Others Subprogram expenditures for activities other than those listed above, such as training graduate students and postdoctoral fellows or safety procedures.

Table IV.4: Mean Percent Expenditures for R&D-Related Activities Among Programs

ER = Energy Research CE = Conservation and Renewable Energy ES&H = Environment, Safety and Health NE = Nuclear Energy DP = Defense Programs NPR = New Production Reactors ERWM = Environmental Restoration and Waste Management FE = Fossil Energy CRWM = Civilian Radioactive Waste Management PPA = Policy Planning and Analysis INT = Intelligence WFO = Work for Others Subprogram expenditures for activities other than those listed above, such as training graduate students and postdoctoral fellows or safety procedures.

National Laboratories’ R&D Outputs

Data concerning outputs of the national laboratories’ R&D-related activities, by laboratory and by program, are presented below.

ANL = Argonne National Laboratory LBL = Lawrence Berkeley Laboratory ORNL = Oak Ridge National Laboratory PNL = Pacific Northwest Laboratory INEL = Idaho National Engineering Laboratory LLNL = Lawrence Livermore National Laboratory LANL = Los Alamos National Laboratory SNL = Sandia National Laboratories BNL = Brookhaven National Laboratory NREL = National Renewable Energy Laboratory Responses were not collected from Brookhaven National Laboratory and National Renewable Energy Laboratory.

Research subprogram outputs other than those listed above, such as technical abstracts, workshops for laboratory users, and an electronic bulletin board service.

Table V.2: National Laboratories’ Research Program Outputs by Program

(Table notes on next page)

ER = Energy Research CE = Conservation and Renewable Energy ES&H = Environment, Safety and Health NE = Nuclear Energy DP = Defense Programs NPR = New Production Reactors ERWM = Environmental Restoration and Waste Management FE = Fossil Energy CRWM = Civilian Radioactive Waste Management PPA = Policy Planning and Analysis INT = Intelligence WFO = Work for Others Pacific Northwest Laboratory provided information about all outputs for the laboratory as a whole that are not included in the data presented here.

Responses were not collected from Brookhaven National Laboratory and National Renewable Energy Laboratory.

Research subprogram outputs other than those listed above, such as technical abstracts, workshops for laboratory users, and an electronic bulletin board service.

Private firms or industrial organizations Private firms or industrial organizations ER = Energy Research CE = Conservation and Renewable Energy DP = Defense Programs WFO = Work for Others Fiscal year 1992.

The number of outputs is shown only for research subprograms in the study population that were in operation all 4 years (fiscal years 1989-92).

National Laboratories’ Potential for Commercial Product Development

Data concerning the formation of cooperative research and development agreements, expenditures for R&D in critical technologies, and the views of national laboratory program managers on their programs’ potential for commercial product development are presented below.

Responses on CRADA formation were not collected from National Renewable Energy Laboratory.

Environmental Restoration and Waste Management

Responses were not collected from National Renewable Energy Laboratory.

Table VI.3: Mean Percent Expenditures for Critical Technologies Within Laboratories

(Table notes on next page)

ANL = Argonne National Laboratory LBL = Lawrence Berkeley Laboratory ORNL = Oak Ridge National Laboratory PNL = Pacific Northwest Laboratory INEL = Idaho National Engineering Laboratory LLNL = Lawrence Livermore National Laboratory LANL = Los Alamos National Laboratory SNL = Sandia National Laboratories Responses were not collected from Brookhaven National and National Renewable Energy Laboratory.

Subprogram expenditures for activities other than those listed above, such as robotics, special nuclear materials, environmental R&D, and detector technology.

Table VI.4: Mean Percent Expenditures for Critical Technologies Within Programs

(continued)

ER = Energy Research CE = Conservation and Renewable Energy ES&H = Environment, Safety and Health NE = Nuclear Energy DP = Defense Programs NPR = New Production Reactors ERWM = Environmental Restoration and Waste Management FE = Fossil Energy CRWM = Civilian Radioactive Waste Management PPA = Policy Planning and Analysis INT = Intelligence WFO = Work for Others Responses were not collected from Brookhaven National and National Renewable Energy Laboratory.

Subprogram expenditures for activities other than those listed above, such as robotics, special nuclear materials, environmental R&D, and detector technology.

Total exceeds 100 owing to rounding.

Comments From the Department of Energy

The following are GAO’s comments on the September 14, 1994, letter from DOE.

GAO Comments

1. The definitions for basic research, applied research, and development that our study employs are derived from a Congressional Budget Office study of the federal R&D enterprise. The definition of technology transfer is the one used in our study of the Technology Transfer Act of 1986 and the definition of technical assistance is based on expert opinion. Our analysis examined the laboratories’ effort in each type of activity separately, and grouped, in two major categories, in order to address the study objective: to provide an empirical base for examining the extent to which the laboratories are engaged in basic and applied research or research related to commercial product development. Figure 1 and tables IV.1-IV.4 allow the reader to view our findings in both the two major categories and as separate R&D-related activities. The finding for each major category presented in figure 1 is the sum of the findings for the corresponding separate R&D-related activities presented in the last column of table IV.1.

DOE disagrees with the category we established for “research related to commercial product development”—that is, that development, technical assistance, and technology transfer are all laboratory activities related to commercial product development—but does not question our definitions or findings for each separate activity. We agree that DOE may decline to accept our definition for research related to commercial product development, but we do not agree that our finding for the sum of the three separate activities is erroneous. This finding is based on laboratory research managers’ estimates of the distribution of their subprograms’ expenditures that were collected, verified, and analyzed according to generally accepted government auditing standards. We consider these estimates, made by research managers who are closely involved with the R&D, more accurate than estimates that may be obtained by other methods. 2. The analyses we produced were intended to establish baseline data for addressing empirical questions underlying the public debate, rather than to serve as a comprehensive analysis of the laboratories’ roles. To address the study objective, we focused on the 10 laboratories as a set of institutions, on comparing the distribution of expenditures for five types of R&D-related activities both within and among the 10 laboratories, on the nature and scope of their outputs, and on their potential for working with industry to bring commercial products to market. Given this approach, with the exception of expenditures for critical technologies and collaboration with industry, which we do examine, the other factors DOE suggests for analysis are beyond the scope of this study. However, we anticipate that our study might stimulate another party to undertake the type of institutional, comparative analysis that DOE suggests. 3. We agree with DOE that the laboratories collaborate in R&D with industry partners who then perform the additional testing and research activities required for commercial application. We also agree that “it typically should take years from the conclusion of a CRADA and the transfer of a technology to a partner, to the commercialization of a product.” The explicit definitions of terms and the discussion of CRADAs in the report make this clear. (See pp. 6, 12, and 15.) However, we disagree that the report attributes commercial product development work to the national laboratories. 4. We state in the section on Methodology that we began our work with a survey of the laboratories’ R&D activities because we could find no sufficiently comprehensive (emphasis added) existing documentation. To confirm that we had not overlooked an important information source when we designed and implemented our data collection strategy, we made inquiries about DOE’s institutional plan and research and development databases. We found that DOE headquarters maintains only the institutional plan database and that it includes only one of the data items, research program budget, that we used in our report. This budget information was available for fiscal years 1989-91 when we implemented our survey but would not have been useful for our analyses because it is not compiled at the same level of detail as our data.

We also found that the research and development database is not one of DOE’s databases. It is being developed by the Critical Technologies Institute for the Office of Science and Technology Policy (OSTP) in the Executive Office of the President. When it is complete, it will have five data items analogous to our data. However, this database was not available when we developed our national laboratory inventory and is not now available to users other than OSTP. Forty-one of the items in our report are not included in either the institutional plan or research and development databases.

The Laboratory Management Division in DOE’s Office of Energy Research maintains the institutional plan database. It has research program budget data for fiscal years 1979 to the present at the program level for 9 of the 10 national laboratories, and it has subprogram budget data for selected programs, such as energy research, defense programs, civilian radioactive waste management, and work for others. Because they are incomplete at the subprogram level, these data would not have been useful for our R&D-related activities and critical technologies analyses, which required budget data for all subprograms in our sample. Further, none of the budget data for the National Renewable Energy Laboratory are included in the institutional plan database. These data must be obtained from NREL’s hardcopy institutional plan, which is available from the Office of Energy Efficiency and Renewable Energy at headquarters.

The Critical Technologies Institute’s research and development database will have information on laboratory expenditures for basic research, applied research, development, and technology transfer for research subprogram categories analogous, but not identical, to those we used, and on CRADAs—for the national laboratories as well as for the laboratories of several other federal agencies—when it is available to organizations other than OSTP. The Critical Technologies Institute representative to whom we spoke could not specify when the database will be available. However, the research and development database will not have information comparable to the 16 research subprogram outputs we collected from the laboratories nor on the proportion of subprogram expenditures for the 22 critical technologies and the proportion of expenditures for technical assistance.

We are also aware that abstracts of CRADA agreements can be obtained through DOE headquarters from the Office of Scientific and Technical Information, which is based in Oak Ridge, Tennessee. However, we also found these data to be incomplete. In August 1992, we requested these data through DOE’s Office of Technology Utilization at headquarters and received 147 abstracts for the nine laboratories from which we collected CRADA information—49 fewer than the total the laboratories reported to us. Since the fiscal year was not then complete, we assumed that all CRADA information had not yet been reported to DOE or entered into the database.

Our experience developing the survey frame, moreover, suggested that the laboratories’ institutional plan data needed modification to address our study requirements and that the information available from DOE was not consistent with information available from the laboratories. We used the list of research programs included in the institutional plans as a preliminary frame for part I of the survey. Recognizing that the laboratories are dynamic institutions, we asked each laboratory to confirm the list before survey implementation. Most of the laboratories made both deletions and additions to the list to meet our survey selection criteria. (See p. 4.) We used the lists of facilities reported in the DOE report, Capsule Review of DOE Research and Development Laboratories and Field Facilities, as a preliminary frame for part II of the survey. The laboratories made deletions and additions to these lists as well and in two cases almost completely replaced them. Changes of this magnitude confirmed the strategy of collecting data directly from the laboratories to address our study’s information requirements. 5. During the agency review of our draft report, two laboratories provided us with additional CRADA information, bringing the total number of CRADAs in effect among all programs in operation in any year from fiscal year 1989 to 1992 to 196. (See table VI.1.) This total is the number of CRADAs in effect in fiscal year 1992, rather than “now,” to which DOE refers and which we assume is fiscal year 1994. Moreover, we found a substantial increase in CRADA formation in fiscal year 1992, sponsored by DOE’s defense program technology transfer initiative at Sandia. (See tables VI.1 and VI.2.) It is possible that the increase we found persisted and included more laboratories, bringing the total to 1,000 in fiscal year 1994. However, such a change would not render our finding for fiscal year 1992 inaccurate. 6. Brookhaven brought it to our attention that the number of CRADAs formed is limited by the amount of money allocated to a laboratory and that this amount varies widely from laboratory to laboratory. We agree with Brookhaven that characterizing CRADA formation as the “strongest” indicator of a laboratory’s commercial product potential is misleading for this reason, and we have modified our discussion of CRADA findings.

Scientific user facilities and personnel exchanges will be examined in a separate study. Licensing is described in the section on Principal Findings of this report. (See pp. 13-14.) CRADAs are cost-shared cooperative agreements targeted to a commercial innovation. 7. We treat laboratory outputs as measures of activity, not as measures of impact or productivity. (See pp. 7-8.) 8. We found that the 10 laboratories produced many more publications and reports (21,593) than they did outputs related to commercial product development (2,510) in fiscal year 1992. This is a statement of fact, tabulated from reports to us by the laboratories’ research managers. It describes the laboratories’ activity. It is not intended as a criticism of the research enterprise. 9. The purpose of this report was to examine the balance of R&D-related activity across the laboratories, rather than to examine the magnitude of the R&D investment. We used the proportion of funds expended for each type of R&D-related activity as a measure of activity, not as a measure of investment. (See pp. 6-7.) An examination of human resources and a comparison of DOE’s national laboratories to those of other agencies was beyond the scope of this study, given its focus on laboratory R&D-related activity.

Comments From Argonne National Laboratory

A representative of Argonne, Internal Audit, called us on July 5, 1994, to report that the laboratory had no substantive comments on the report draft.

Comments From Brookhaven National Laboratory

The following are GAO’s comments on the June 27, 1994, letter from Brookhaven National Laboratory.

GAO Comments

1. We have added a statement to the report clarifying this difference. 2. We have evaluated the data Brookhaven submitted and, after making the appropriate changes, added it to the database. These data have been incorporated into the tables included in the report letter and appendixes. 3. We agree with Brookhaven’s evaluation of this response and have made the change they requested to the database and report tables. 4. We agree with Brookhaven and have modified the discussion of CRADA findings. 5. The information on CRADA formation Brookhaven submitted in the pilot version of the data collection instrument has been added to the database and the tables in appendix VI. We also have added the sentence Brookhaven suggests to appendix I.

Comments From DOE’s Idaho Operations Office

The following are GAO’s comments on the June 29, 1994, memorandum from DOE’s Idaho Operations Office.

GAO Comments

General Comments

DOE’s Idaho Operations Office representative, who responded for Idaho, observed that the value in the “R&D Budget” column of table 2 for Idaho should be $275 million, rather than $98.7 million and that the Idaho Operations Office made this determination by applying DOE headquarters’ definitions for research programs to Idaho’s research programs. The list of Idaho research programs to which the Idaho Operations Office applied DOE headquarters’ definitions is unspecified. We disagree with this determination, because it violated the study methodology.

Specific Comments

1. The R&D budgets of the 10 national laboratories in table 2 were not compared. 2. We coordinated data collection from the laboratories with DOE’s operations office representatives, but none of them participated in any of the technical activities involving survey implementation. Therefore, the Idaho Operations Office representative may not have been aware that GAO program selection criteria should have been employed to assess the “R&D Budget” column value for Idaho in table 2 to be consistent with the methodology employed for the other nine laboratories. The use of DOE headquarters’ definitions for research programs to make this determination would result in a list of subprograms that differs substantially from the one jointly developed by GAO and Idaho.

Subprograms included in the survey population were identified by laboratory representatives who applied the selection criteria we specified (see p. 4) to a preliminary subprogram list we compiled from the institutional plans and sent to the laboratories. This approach was followed by Idaho’s representatives, who identified 10 subprograms. We reduced the number of Idaho subprograms to nine during the editing and coding process. The $98.7-million value in the “R&D Budget” column is the total of nine research subprogram budgets reported by Idaho program managers on part I of the national laboratory inventory data collection instrument.

Comments From Lawrence Berkeley Laboratory

The following are GAO’s comments on the June 17, 1994, letter from Lawrence Berkeley Laboratory.

GAO Comments

Although Lawrence Berkeley agreed with the study’s analytic framework and with the need for studies of this type to inform congressional policymakers, the laboratory raised an issue about the relationship of the national laboratories’ role in commercial product development to the broader needs of industry or the nation, which was not addressed in the report. This omission warrants clarification.

General Comments

The relationship of the national laboratories’ role in commercial product development to the broader needs of industry is an issue being discussed in the public debate about the laboratories’ missions and structure, but one that falls outside of the study scope. The purpose of this study was to examine the extent to which the national laboratories are engaged in basic and applied research or research related to commercial product development. Scientific and technical infrastructure, which Lawrence Berkeley gives as an example of industry need, while important to the considerations of laboratory mission and structure that serve as the study’s policy context, was not addressed. It was our expectation that the findings of this study would serve as an empirical base for designing a study to address this and other institutional issues.

Specific Comments

marketing activity that accomplish commercial application. (See pp. 6, 12, and 15.) 2. This study was not designed as a broad assessment of the national laboratories’ roles, but to examine the balance of the laboratories’ R&D-related activities in two major areas: basic and applied research and research related to commercial product development. We looked at these activities with three types of measures, and our conclusions interpret our findings for each type. The conclusion focuses on research related to commercial product development because we found slightly more activity in this area. We amplified this conclusion with an interpretation of findings for the other two types of measures. A discussion of the noncommercial product output of nuclear weapons research was not relevant. 3. We have added a discussion of these limitations to the Methodology section. (See pp. 7-8.)

Comments From Lawrence Livermore National Laboratory

The following are GAO’s comments on the July 6, 1994, letter from Lawrence Livermore National Laboratory.

GAO Comments

We have added the revised text describing Lawrence Livermore Laboratory to appendix I.

Comments From Los Alamos National Laboratory

The following are GAO’s comments on the June 23, 1994, letter from Los Alamos National Laboratory.

GAO Comments

We have added a footnote to the Background section discussing the legal division of Department of Defense and civilian responsibility for nuclear weapons research and development. We also expanded the phrase on page 2 from “weapons development” to “nuclear weapons research and development.”

Comments From National Renewable Energy Laboratory

The following are GAO’s comments on the June 22, 1994, letter from National Renewable Energy Laboratory.

GAO Comments

1. One future report will provide a descriptive statistical analysis of the technical and operating characteristics of the national laboratories’ major research facilities. Other topics are yet to be determined. 2. We have made this correction to the text. 3. Graphs and tables are presented in the section on Principal Findings. 4. The aggregation in figure 1 is intentional. The graph is designed to illustrate the balance between the two major areas of R&D-related activity we examined. The last column of table IV.1, labeled “All Labs,” presents percentages for development, technical assistance, and technology transfer for the 10 laboratories. 5. See table VI.3 in appendix VI. Table VI.4 presents these percentages by program area. 6. We have made this correction to the text.

Comments From Oak Ridge National Laboratory

We did not receive Oak Ridge’s written comments from DOE. We did discuss Oak Ridge’s views with laboratory representatives by telephone on June 22 and July 13 and 19, 1994, and we spoke with a representative of DOE’s Oak Ridge Operations Office on July 7, 1994. We also received new output data for Oak Ridge’s subprograms by facsimile from representatives of both organizations. A summary of their comments and our response follows.

GAO Comments

Oak Ridge raised two general issues. One was the effect of the study sampling methodology on findings for the laboratory’s outputs related to commercial product development. Oak Ridge took the position that most of the laboratory’s outputs related to commercial product development were produced by subprograms not selected in the study sample and, consequently, expressed the concern that GAO’s findings for outputs related to commercial product development based on the sampled subprograms may not be representative because of this distribution of outputs among all laboratory subprograms. Most of these outputs, they explained, are produced by programs that fall below the $10-million threshold for inclusion in the survey. In fact, according to tabulations they had performed, the sampled programs, while representing 73 percent of the overall budget, produce only 7 percent of the outputs in question.

Secondly, Oak Ridge thought that the report’s definitions and analyses equate development work with commercial product development and that the conclusion based on this definition is not supported by the data. We address these issues separately.

First, Oak Ridge actually had identified two sources of potential underreporting: (1) data for outputs of sampled subprograms that were not available at the time of the survey and (2) data for outputs of unsampled programs. We agreed that additional data for sampled subprograms should be added to findings for Oak Ridge. We requested and received from Oak Ridge the new data for the sampled subprograms, and we added them to our database and report tables. matter of some concern to us, particularly in light of the large number of outputs Oak Ridge ascribed to the unsampled programs.

Second, we disagreed that the report definitions and analyses equate development work with commercial product development. Our definitions, analyses, and conclusions make it clear that the laboratories were not expected to produce commercial products. We defined development as having “some type of product as the output goal (emphasis added),” but concluding “with a prototype rather than a usable good.” Further, we point out that “Additional time, research, and testing are required to convert the prototype to a weapon or commercially viable product.” The definitions of outputs related to commercial product development, including those for precompetitive commercial products and processes, state that these outputs tend (emphasis added) to arise from development work, but that “they will require a substantial additional investment before they are ready to market.” The conclusion, moreover, reiterates that these outputs are “precursors to marketable goods,” and that, for this reason, “it is too early to determine whether this activity will produce technologies with commercial uses.”

We also examined the assumption that R&D is a linear process, with all commercial product-related outputs arising from development, and found that our data did not support it. We included this segment of the analysis to emphasize the uncertainty associated with current understanding of the operation of the R&D process, and the origin of technologies with commercial potential. The conclusion we reached concerning the uncertain prospects of the laboratories’ commercial product-related outputs is an interpretation of this finding as well as our definitions for outputs related to commercial product development.

Comments From Pacific Northwest Laboratory

A representative of Pacific Northwest, called us on June 22, 1994, to comment on the draft report by telephone. A summary of the laboratory’s comments is included in our response, which follows.

GAO Comments

Pacific Northwest offered one general comment and several comments and questions about specific items in the text. We address the general comment first and then the specific comments.

General Comment

Pacific Northwest suggested that a section be added to the report describing the major commercial product-related initiatives the national laboratories have undertaken since the end of fiscal year 1992. Partnership for a New Generation of Vehicles (PNGV) and American Textile Partnership (AMTEX), two consortia for R&D targeted on commercial applications in which several laboratories are participating, were mentioned as examples.

We are aware that the laboratories have been active in technology transfer activities of many types since the end of fiscal year 1992. This activity will be captured in any follow-up study that is performed in the next few years to determine if progress has been made since fiscal years 1989-92, the time period measured in this report.

Specific Comments

1. Pacific Northwest thought that the word “primarily” in the sentence beginning on draft line 10, page 4 (now line 12, p. 3), should be deleted because it implies that the laboratories have only one primary mission. We have modified this sentence. 2. Pacific Northwest said that the output data in table 5 (now table V.1) not reported for the laboratory are available and will be submitted to us. We received and reviewed the data, and we added it to table V.1. 3. Pacific Northwest said that information on CRADA formation for the laboratory as a whole was submitted to us during survey implementation. We confirmed that this information had been received and added it to table VI.1.

Comments From Sandia National Laboratories

The following are GAO’s comments on the July 1, 1994, letter from Sandia National Laboratories.

GAO Comments

Sandia agreed with the report’s objective, methodology, and conclusion, but made two general comments. First, Sandia suggested that the report include a description of the national laboratories’ expanded efforts in technology transfer during fiscal years 1993-94. Second, Sandia suggested that we review the substantial variation in the percentage of laboratory funds not expended for critical technologies reported for Lawrence Livermore, Los Alamos, and Sandia in table VI.3. Sandia expected this percentage to be very similar for all three laboratories.

We are aware that the national laboratories have been active in technology transfer activities of many types during fiscal years 1993-94, including participation in large-scale R&D consortia such as PNGV and AMTEX. These activities will be captured in any follow-up study that is performed during the next few years to determine if progress has been made since fiscal years 1989-92, the time period measured in this report.

We reviewed all responses by Lawrence Livermore, Los Alamos, and Sandia concerning percent of expenditures for critical technologies and funds not expended for R&D in these areas. We found that Lawrence Livermore program managers allocated a percentage of funds expended to the “other” category to a much greater extent than did program managers at Sandia or Los Alamos. We also found considerable variation among all laboratories in the proportion of expenditures allocated to this category. R&D activities specified in the “other” category included items such as robotics, special nuclear materials, environmental R&D, and detector technology. Allocations to this category, and to the energy technologies category, accounted for most of the difference in proportion of funds not expended for critical technologies by Lawrence Livermore, Los Alamos, and Sandia.

Major Contributors to This Report

Program Evaluation and Methodology Division

Denver Regional Office

Miguel A. Lujan, Project Adviser The first copy of each GAO report and testimony is free. Additional copies are $2 each. Orders should be sent to the following address, accompanied by a check or money order made out to the Superintendent of Documents, when necessary. Orders for 100 or more copies to be mailed to a single address are discounted 25 percent.

U.S. General Accounting Office P.O. Box 6015 Gaithersburg, MD 20884-6015 Room 1100 700 4th St. NW (corner of 4th and G Sts. NW) U.S. General Accounting Office Washington, DC Orders may also be placed by calling (202) 512-6000 or by using fax number (301) 258-4066.

Each day, GAO issues a list of newly available reports and testimony. To receive facsimile copies of the daily list or any list from the past 30 days, please call (301) 258-4097 using a touchtone phone. A recorded menu will provide information on how to obtain these lists.

Address Correction Requested