John W. Lyons and Richard Chait 

Center for Technology and National Security Policy 

National Defense University 

March 2009 

Strengthening Technical Peer 

Review at the Army S&T 

Laboratories 
ii

The views expressed in this article are those of the authors and do not reflect the official 

policy or position of the National Defense University, the Department of Defense, or the 

U.S. Government. All information and sources for this paper were drawn from 

unclassified materials. 

John W. Lyons is a Distinguished Research Fellow at the Center for Technology and 

National Security Policy (CTNSP), National Defense University. He was previously 

director of the Army Research Laboratory and director of the National Institute of 

Standards and Technology. Dr. Lyons received his PhD from Washington University. He 

holds a BA from Harvard. 

Richard Chait is a Distinguished Research Fellow at CTNSP. He was previously Chief 

Scientist, Army Material Command, and Director, Army Research and Laboratory 

Management. Dr. Chait received his PhD in Solid State Science from Syracuse 

University and a BS degree from Rensselaer Polytechnic Institute. 

Acknowledgements 

In addition to the subject matter experts listed in appendix B, the authors express 

gratitude for cooperation and support of the Army Science and Technology Executive, 

Dr. Thomas Killion, and the Director of the Center for Technology and National Security 

Policy, Dr. Hans Binnendijk. 

Defense & Technology Papers are published by the National Defense University Center for Technology 

and National Security Policy, Fort Lesley J. McNair, Washington, DC. CTNSP publications are available 

at http://www.ndu.edu/ctnsp/publications.html. 
iii

Contents

Chapter 1: Introduction to Peer Review...........................................................................................1 

 Background.............................................................................................................................1 

 Some Factors in Laboratory Quality.......................................................................................3 

 Managing the Peer Review Process........................................................................................4 

 

Chapter 2: Peer Review at Army Laboratories................................................................................6 

 Deputy Assistant Secretary of the Army (Research and Technology) ...................................6 

 The Army Materiel Command’s Research Development & Engineering Command ............6 

Chapter 3: Peer Review at Some Other Government Laboratories ...............................................12 

 Assessment of the Laboratories at the NIST.........................................................................13 

 Other Approaches .................................................................................................................13 

Chapter 4: A Summary of Peer Review Practices .........................................................................19 

 

Chapter 5: Closing Comments and Recommendations .................................................................21 

 Recommendations.................................................................................................................22 

Appendix A: Interviewees .............................................................................................................24 
iv
1

Chapter 1: Introduction to Peer Review 

The assumption underlying the value of peer review is that the quality of work is 

substantiated or improved through critiques by individuals who are independent, 

objective, and have specialized knowledge in the subject matter. One of the authors (Dr. 

Lyons) has had firsthand experience with such reviews. Some time ago, while in the 

private sector, he was asked to serve on a National Research Council (NRC) ad hoc 

committee of external, independent experts to review the fire research program at the 

National Bureau of Standards (NBS). The committee found the programs to be 

fragmented and deficient in basic research, and made a number of recommendations on 

how to improve them. (For more on how such a committee operates, see the discussion of 

the National Institute of Standards and Technology in chapter 3.) The result was the 

creation of a newly focused organization with a strengthened scientific knowledge base. 

These reforms helped NBS become a world leader in the field of fire research. 

A second example of the contributions made through the peer review process is provided 

by the establishment in 1995 at the Army Research Laboratory (ARL) of an NRC board, 

the Technical Assessment Board, to review the quality of ARL programs. The first report 

made a number of recommendations, including reduction of what the panel called 

parochialism at ARL. Actions taken as a result of this review improved the laboratory’s 

interactions with academia and industry. Such outside review often provides a different 

perspective that can be useful in shaping future work. The use of peer review, using 

experts in possession of varied and extensive experience, can much improve the quality 

of work and efficacy of results. 

The Army Science and Technology Executive, Dr. Thomas Killion, requested a study of 

peer review methods in use at Army laboratories. This paper discusses Army laboratories 

in terms of generally accepted best practices and compares them with techniques at other 

DOD laboratories, and at a few other government agencies. 

Background 

Peer review of technical work is a well-established tradition in research and development. 

Review can be on individual projects or, more broadly, of a laboratory as a whole. There 

are three forms of project-focused review: one is used to evaluate proposals for new 

work, such as research proposals to funding agencies; a second is used to review work in 

progress, which is the primary focus of this study; and the third is used to review 

completed work in the form of reports or publications. In peer-reviewed journals, 

sometimes called archival journals, each paper is sent to technical experts for critique; 

changes may be suggested, or the paper may be rejected, based on lack of technical merit 

or originality. 

While peer review has been used for many years in the Federal laboratories, there are no 

uniform standards or approaches. In a 1983 report to the White House Science Council, 

its Federal Laboratory Review Panel recommended that individual projects be subject to 

peer review at the outset. The panel also recommended the creation of, in its own words, 
2

“an oversight function responsible for assuring the continuing excellence of the 

laboratory. This function could be performed by an external committee [and] would focus 

on productivity and on the excellence, relevance, and appropriateness of research.”1 

The report provided no details as to how the reviews should be performed. The President 

instructed the Office of Science and Technology Policy and the Office of Management 

and Budget to lead an interagency effort to respond to the Panel’s recommendations. The 

results of this effort are documented in a 1984 report.2 Various actions were undertaken; 

in particular, the DOD asked the Services to form external advisory groups. 

 

1

 Office of Science and Technology Policy, Report of the White House Science Council Federal Laboratory 

Review Panel, Recommendation 4-1 (Washington, DC: Executive Office of the President, 1983). 2

 Office of Science and Technology Policy, Progress Report on Implementing the Recommendations of the 

White House Science Council’s Federal Laboratory Review Panel (Washington, D.C: Executive Office of 

the President, 1984). 

Peer review was invented at the Royal Society in 1665 by the founding editor of 

Philosophical Transactions of the Royal Society, Henry Oldenburg. The present-day 

peer review system evolved from this 17th-century process.” 

A practice similar to a peer review process is found in Ethics of the Physician, written 

by Ishaq bin Ali al-Rahwi (854–931) of al-Raha, in what is now Syria. His work, as 

well as later Arabic medical manuals, state that a visiting physician must always make 

duplicate notes of a patient's condition on every visit. When the patient was cured or 

had died, the notes of the physician were examined by a local medical council of other 

physicians, who would review the practicing physician's notes to decide whether his/her 

performance have met the required standards of medical care. If their reviews were 

negative, the practicing physician could face a lawsuit from a maltreated patient. “ 

Peer review has been considered essential to the modern scientific method only since 

the middle of the 20th century. Before then, its application was frequently lax. For 

example, Albert Einstein's revolutionary "Annus Mirabilis" papers in the 1905 issue of 

Annalen der Physik were not reviewed by anyone other than the journal's editor in chief, 

Max Planck (the father of quantum theory), and its co-editor, Wilhelm Wien. Although 

clearly they were peers (both won Nobel prizes in physics), a formal panel of reviewers 

was not sought, as is done for many scientific journals today. Established authors and 

editors were given more latitude in their journalistic discretion at that time. As a recent 

editorial in Nature stated, "in journals in those days, the burden of proof was generally 

on the opponents rather than the proponents of new ideas.” 

Source: derived from Wikipedia entry on peer review, available at <http://en.wikipedia.org/ 

wiki/peer_review>. 
3

Peer review tests the effectiveness of a laboratory’s research staff by comparing the 

conduct of the laboratory’s research with that at the best laboratories in the field. Outside 

experts compare the material under review with what they know of comparable work 

elsewhere—perhaps including their own work. External reviews reduce the tendency of a 

laboratory to become parochial, a special problem for laboratories that perform classified 

research. The expectation of scheduled peer reviews challenges the laboratory’s technical 

staff to keep up with technical advances worldwide and to stay in touch with colleagues 

elsewhere. Peer review also brings varying perspectives on the work from laboratories 

across the Nation, or even around the world. Therefore, peer review is a way to establish 

a reputation for high-quality performance at a laboratory. For some portion of the 

assessment, review may be enhanced by the provision of data on performance against a 

number of metrics.3

 

Some Factors in Laboratory Quality 

There are several factors in evaluating the quality of individual programs or groups of 

programs in a laboratory: 

Status of the work. Planning, originality and creativity, publications and patents, 

comparison to work being done elsewhere, coordination or collaboration with others, and 

pacing or productivity are considered. The panel may also assess the program as a whole, 

e.g., the balance among basic, applied, and development research. Within these 

categories, reviewers may also consider the potential, or lack thereof, for beginning new 

areas of basic research.4 

Qualifications of the research team. Reviews look at the record of accomplishments, 

awards and honors, and ability to present the work. First-rate equipment and modern 

facilities are essential for success—are they well supported? Is the overall budget 

adequate for the task? Finally, the working environment is important to an evaluation. 

Stimulating coworkers with technical prowess in their fields, and a supportive 

management team are important elements of productive laboratory work. 

Relevance. This paper is not focused on relevance, though many of the reviews discussed 

herein address a mixture of quality and relevance. Factors in assessing relevance include: 

probability of success, priority of the need being addressed, validation of the need by the 

user, and collaborations with other laboratories. Relevance is also influenced by the 

timeliness of the delivery of results.

Credibility. For a review to have credibility across communities of interest, reviewers 

must be independent and impartial, able to comprehend technical details, and familiar 

with similar work being done elsewhere. Reviewers can be drawn from within the 

institution, if they are appropriately remote from the work, or from outside sources such 

 

3

 Edward A. Brown, Reinventing Government Research and Development: A Status Report on Management 

Initiatives and Reinvention Efforts at the Army Research Laboratory, ARL-SR-57 (Adelphi, MD: Army 

Research Laboratory, August 1998). 

4

 Hans Mark and Arnold Levine, The Management of Research Institutions: A look at government 

laboratories, NASA SP-481 (Washington, DC: NASA, 1984), 226. 
4

as academia, industry, or other institutions. Any potential for bias or the perception of 

bias must be resolved. 

Reporting. Review panels should make explicit reports, preferably in writing, to the head 

of the laboratory, and perhaps to higher levels of external management. The personnel 

reviewed should be given the results of the review. It is unusual to find a laboratory that 

cannot benefit from a thoughtful, professional critique. Such outside reviews should be 

sought by practitioners in any field. For laboratories, the review may take many forms. 

Some of these forms are investigated in this paper in terms of current practices at various 

Army, DOD, and other government laboratories. 

Managing the Peer Review Process

The peer review process is generally strengthened by attention to the following concerns: 

Review process. Two archetypal methods of establishing a reviewing system should be 

discussed. In the first, the laboratory under review may contract for the review with an 

outside party and only make recommendations to the contractor. In this model, the 

contractor manages the selection of panel members and provides logistical and 

operational support to the review panel. After finalizing the contract, the laboratory plays 

only a supporting role—offering suggestions, but otherwise maintaining a hands-off 

approach. In the contrary model, the laboratory selects panel members, defines review 

topics, and exercises close management control—a hands-on approach. The advantage of 

the hands-off approach is the enhanced credibility of the product. An advantage of the 

hands-on approach is that management is able to select panel members known to be 

particularly expert in the details of the programs under review. 

Relationship of reviewers to reviewed. Another important factor is the relationship of 

the members of the review panel to the laboratory under review. To establish credibility, 

panel members should be seen as objective and independent of the laboratory. To this 

end, experts should be selected from outside the laboratory, and preferably outside the 

organizational parent of the laboratory. The use of panels selected from internal staff 

gives the appearance of favoring the laboratory’s efforts out of loyalty and parochialism. 

Review panel members should include academics qualified to look at the basic research 

component. For applied research, members should be added from industry and other 

relevant laboratories. Possible conflicts of interest should be made known, and the panel 

constituted with appropriate balance to offset conflicts. Another benefit of external 

review is the injection of different experiences and points of view not available to inhouse staff. 

Quality and relevance. Quality is best judged by expert scientists and engineers active in 

the appropriate fields. Relevance is best judged by the users of the technical results, who 

may or may not know the technology in detail but have a technical appreciation of the 

need, and know whether the work adequately addresses that need. If the review is to 

address both technical quality and relevance, the membership of the panel will be mixed. 

One aspect of quality review is assessment of the laboratory environment—the suitability 

of the laboratory equipment and buildings, the characteristics of the staff, support groups, 
5

such as the machine and electronic shops, the support and encouragement given by the 

senior managers of the laboratory, and the adequacy of the budget. To judge these 

aspects, the panel will necessarily need members who are researchers with experience at 

similar laboratories. 

Following this general discussion of peer review, Chapter 2 presents the many ways the 

Army’s S&T laboratories conduct reviews. Chapter 3 then discusses several approaches 

now being used at other Federal laboratories. Chapter 4 discusses the findings of this 

investigation. Chapter 5 provides closing comments and recommendations. 

The paper recommends that the Army require peer review of the technical quality of its 

laboratories and proposes a set of norms that must be met. The principal recommendation 

is that reviews be performed by independent experts who visit the laboratory for two or 

more days, looking at the technical projects and the strength of the technical staff, 

equipment, and facilities. The recommendations include a caveat about potential conflicts 

of interest in these panels. 
6

Chapter 2: Peer Review at Army Laboratories 

To discover how the various Army laboratories carry out technical peer review of their 

work, interviews were conducted with leaders of each of the four segments of the Army 

that run research programs; namely, the Army Materiel Command’s Research, 

Development, and Engineering Command (RDECOM) and its laboratories, the Medical 

Research and Materiel Command (MRMC), the Corps of Engineers (COE), and the 

Army Research Institute for the Behavioral and Social Sciences (ARI). For RDECOM, 

discussions were held with seven of the ten technical components—those that conduct 

bench-level research and development. Our investigation showed that the Army uses a 

variety of approaches to evaluate the performance of its research and development work. 

The following sections summarize the findings. 

The Office of the Deputy Assistant Secretary of the Army (Research and 

Technology) 

This office conducts many levels and kinds of reviews, two of which, done by its 

Director of Research and Laboratory Management (DRLM), are pertinent here. The first 

is the Basic Research Review; the other is the U.S. Army Research and Development 

Laboratory of the Year review. Both reviews use external panels of experts. 

The DRLM appoints a set of independent review panels that review the Army’s basic 

research (referred to as “6.1”), including in-house laboratory innovative research (ILIR). 

Rather than reviewing individual laboratories, the review considers broad technical areas 

that may include the work of more than one laboratory. The DRLM also appoints an 

external Army Science Assessment Group that reviews the findings of the independent 

review panels and makes comments and recommendations to the DRLM. This activity is 

performed every 3 years. 

In the Army Laboratory of the Year competition, each laboratory makes a submission 

covering many different aspects of their work. One of these is a description of what they 

consider to be the single most significant accomplishment in fundamental research; 

another is a description of what they consider to be the single most significant 

accomplishment in development work. The DRLM appoints an Army Laboratory 

Assessment Group (ALAG), comprising external experts in relevant fields, to review and 

grade these submissions. The ALAG then sends a member to each of twelve laboratory 

installations for a day-long on-site review. Technical peer quality review of the research 

programs is only a small part of this program. 

The Army Materiel Command’s Research Development and Engineering 

Command 

The Research, Development, and Engineering Command of the Army Materiel 

Command (AMC) manages the AMC laboratories, which consist of the Army Research 

Laboratory (ARL) and its Army Research Office (ARO), and six Research, Development 

and Engineering Centers (RDECs). 
7

The Army Research Laboratory—ARL contracts with the National Research 

Council (NRC) of the National Academies to assemble an ARL Technical 

Assessment Board (TAB). The TAB consists of six panels, each composed of eight 

to ten individuals having high repute within the technical community. Each panel 

reviews one of ARL's primary technical mission areas. The Board, with its panels, 

provides an appraisal of the scientific and technical efforts of ARL. It is specifically 

enjoined from making judgments on the programmatic structure of ARL’s work, 

partly because it is not equipped to perform this function, and partly because there 

are numerous other channels through which ARL receives such programmatic 

guidance. (Examples include: the normal chain of command above ARL, the Deputy 

Assistant Secretary of the Army for Research and Technology, the Director of 

Defense Research and Engineering, the Training and Doctrine Command, and the 

entire planning and budgeting process to include Congressional committees.) 

Because of the size and diversity of the ARL technical program, the panels review 

the total program in depth over a 3-year period. There is a formal report from the 

TAB every 2 years. The panels produce a written assessment that is published by the 

NRC. The TAB also meets with the ARL director to provide an informal report of 

its findings before the publication of the report, and to receive guidance on the 

director's desires for special areas of emphasis for the subsequent year's review. 

The AMC Research, Development and Engineering Centers—All of the RDECs work 

closely with acquisition program executive officers and program managers, and draw a 

considerable fraction of their support from these sponsors. RDECs have relatively little 

basic research funding; what little they have is from a special fund called Independent 

Laboratory Innovative Research (ILIR). Thus, peer review will emphasize the customer 

for the work, hence, relevance, timeliness, and quality. 

None of the six AMC centers that conduct extensive in-house laboratory research conduct 

peer review of the kind described by ARL, but some are considering a similar approach. 

One uses a contractor to put together a panel to conduct biannual technical reviews for its 

6.2 and 6.3 work (it has very little 6.1 money), and that work is reviewed in any case by 

the DRLM’s Basic Research Review. Another RDEC began an external review in 2007 

by contracting to obtain a panel of outside experts to review one major technical area of 

the RDEC for a given year; subsequently another panel will be set up to review another 

segment the following year. 

Another RDEC is considering the introduction of formal technical reviews by external 

experts, but has not yet selected an approach. The remaining three RDECs are not using 

external technical peer review. It should be noted that all the AMC RDECs conduct 

extensive internal program reviews for RDECOM and frequent reviews with their 

customers. 
8

Details of the interviews at the RDECs follow: 

Aviation and Missile Research, Development, and Engineering Center—AMRDEC 

relies on two processes to review its programs. The technical quality is briefly looked at 

in the Army's Laboratory of the Year program. The panel of judges is composed of senior 

technologists external to the Army. They review AMRDEC’s nomination package, and a 

panel member visits AMRDEC for a day. The AMRDEC program is also reviewed by a 

panel consisting primarily of retired general officers, with some representatives from 

senior S&T leadership. Panel members are chosen to have a diverse set of backgrounds to 

insure balance in their advice. This panel serves as a liaison for AMRDEC to the rest of 

the Army. 

Armaments Research, Development, and Engineering Center—ARDEC uses a 

contractor to put together biannual technical peer reviews of 6.2 and 6.3 work. (The only 

basic research they have is the in-house laboratory innovative research (ILIR) funding). 

The panels are made up of technical specialists from across DOD, including some retirees 

from the uniformed Army. The panel members are paid for expenses and time, although 

some members will not accept remuneration. The panel receives briefings for 2–2 1⁄2 days 

at the laboratory to assess quality and content, and spend additional time writing a report. 

The lab is reviewed this way every 2 years. The review uses a checklist designed to look 

at both individual program elements and the lab as a whole. The ARDEC presentations 

compare the work to other labs, such as Los Alamos or Livermore National Laboratories. 

Natick Soldier Research, Development, and Engineering Center—NSRDEC brings in 

university academics to look at 6.1. NSRDEC has relied on the Army Science and 

Technology Working Group (ASTWG) process for additional review of outcomes and 

relevance, but does not carry out technical reviews. These reviews do not evaluate the 

quality of people, equipment etc. The Army Laboratory of the Year competition review 

sends an outside expert to NSRDEC for one day. This person obtains an overview and a 

tour of the laboratories. A technical peer review is not conducted. 

The NSRDEC management is now ready to look at comprehensive peer review for 6.2 

(and beyond) work. They want a methodology for looking at both quality and relevance. 

They want to evaluate technical quality, relevance, quality of individual skills, and the 

skill mix. They are also interested in benchmarking with other laboratories. They have 

had a prior process (described in a 2006 RDECOM report5

), but discontinued that 

approach after judging it insufficient for peer review—it was more of an information 

sharing effort. 

Edgewood Chemical Biological Center—The ECBC technical director first discussed 

the Army-wide laboratory of the year program, since the ALAG had just visited. The 

ALAG visitors impressed them, and ECBC in particular liked the on-site feedback they 

received from the visitors. The visit was for one day only; a second day would have 

provided time for more detailed presentations and visits to staff in their laboratories. 

 

5

 RDECOM Committee, Chair Dr. J Mait, “RDECOM Initiative on Peer Review and Benchmarking”, Final 

Report, September 2006. Available from Dr. J. Mait, Army Research Laboratory, Adelphi, MD. 
9

ECBC believes the visit would have been more useful had it not been tied to a 

competition and ultimately an award to one laboratory. (The award process was felt to 

have limited the feedback to ECBC at the time of the visit, because some of the criticism 

would be used subsequently in the award decision.) 

ECBC has used an eight-member focal group and facilitator to look at some of their 

programs. Members include retired generals and some technical people. They have 

looked at organizational issues such as the state of the facilities, and the technical merit of 

the work. They have provided a written report. ECBC has used IDA, RAND and the 

National Academies for studies focused on narrow areas of technology; for example, 

toxicity work at ECBC. The studies include visits to the labs. These are not intended to be 

comprehensive and should not be construed as related to an overall peer review process. 

Communications and Electronics Research, Development, and Engineering Center—

CERDEC does not have a technical peer review process in the sense used in this paper. 

CERDEC relies solely on the laboratory of the year ALAG visits for external assessment. 

However, a former technical director had earlier invited groups of “graybeards” to the 

laboratory periodically to review the technical work. This process was unstructured, 

informal, and eventually discontinued. 

In regard to the process for the Army Laboratory of the Year, CERDEC offered some 

criticisms. CERDEC claims not to receive formal feedback for two months or more after 

the visit. The funds allocated by DRLM for the ALAG visits evidently will not allow for 

visits to both Ft Monmouth and Ft Belvoir, so the ALAG is able to visit only part of the 

RDEC each year. 

This year, the ALAG visitor had a security clearance and was able to go into the 

classified facility and receive detailed briefings not available to previous ALAG visitors. 

CERDEC believes the ALAG visits are too short and too limited as to the amount of 

information that can be provided. They also find that statements made by the visitors 

during the visits do not always agree with the formal feedback received much later. They 

believe, with ECBC and NSRDC, that the ALAG process would be better without 

limiting the competition for an award to just one laboratory. They believe that a longer 

visit, say two or more days, using a panel of experts would improve the process by 

allowing more in-depth discussion and more visits to laboratories. 

Tank and Automotive Research, Development, and Engineering Center—TARDEC 

uses senior technical personnel, including two STs (senior technical positions without 

administrative duties and equivalent to positions in the Senior Executive Service) for 

internal technical reviews. On a quarterly basis they do full program reviews. They also 

do an internal technical peer review before starting new programs. The internal reviews 

also include work done by Small Business Innovation Research (SBIR) program grants 

and all Congressional add-ons. 

TARDEC has begun outside peer review by contracting with the Institute for Defense 

Analysis (IDA) to select a panel of outside experts. The first review was in 2007 and was 
10

on survivability technology. They plan to continue these reviews; next up are power and 

energy technologies. The expert panels have five to ten members who meet in 3-day 

workshops. This process reviews just one topical area each year; TARDEC has found this 

to be a very effective way to evaluate technical quality. 

The technical director believes that the laboratory of the year program does not meet the 

criteria of a peer review process for technical work. The technical director expressed 

interest in the National Research Council review program. 

Medical Research and Materiel Command—All of the MRMC’s R&D is externally 

reviewed under a contract managed by the American Institute of Biological Sciences 

(AIBS), formerly a part of the National Academies and now independent. Review is done 

using panels consisting of distinguished external experts who possess the appropriate 

subject matter expertise. These reviews are for research proposals made to MRMC. 

Intramural researchers must also make annual proposals for their work; these too are 

reviewed by AIBS panels. The research program at MRMC is divided into four program 

elements, each of which is overseen by a program manager at headquarters. Every 3–5 

years, each program element brings in a review panel to look at both quality and program 

content. Most reviews are conducted by AIBS; on occasion reviews are obtained through 

the National Academies’ Institute of Medicine or the National Research Council. The 

quality of facilities and equipment are only sporadically reviewed; comments on facilities 

and equipment are sometimes offered by panel members during site visits. 

Engineer Research and Development Center—The Army Corps of Engineers’ ERDC 

consists of seven laboratories at four sites. Each laboratory is reviewed every 2 years by 

external panels of about ten experts. The ERDC director manages the selection of experts 

and the meeting agendas. The panels review quality of work, staff, equipment, and 

facilities. The panels also look at program relevance. The panels provide a qualitative 

ranking comparing the ERDEC programs to the best peer laboratories around the world. 

The reviews usually last 3 days. Written reports are submitted, usually within a month 

after the meetings. The panels review technical programs in detail and visit the laboratory 

staff at the bench level. Panel members are nominated by each of the seven laboratory 

directors and approved by the ERDC director. The members are paid for their time. There 

is some dual membership across the panels. Members are from industry and retired 

military and include several members of the National Academies. Some are eminent in 

their respective fields; some are generalists. There is a standard protocol for the labs to 

follow in conducting their biannual reviews. 

Army Research Institute for the Behavioral and Social Sciences—ARI has 124 

authorized positions and a budget of $32M. About half of the budget is spent outside the 

Institute on contracts or grants. Last year the 6.1 budget was doubled to $6.0 million, 

thereby allowing ARI to broaden and deepen its research. ARI operates its own office to 

solicit, evaluate, and allocate research grants. The ARI has no laboratories in the usual 

sense. Instead it has staff distributed across many different Army Training and Doctrine 

Command (TRADOC) centers and schools. 
11

ARI sometimes has joined with ARL’s Human Research and Engineering Directorate for 

research reviews of focused areas. ARI believes that quality can be measured by success 

in transition and adoption, by the Army, of their recommendations. This is not, however, 

a technical peer review. 

ARI has the NRC Board on Behavioral, Cognitive, and Sensory Sciences study possible 

new areas for work every 3–4 years. For external scientific reviews, ARI uses advisory 

groups that include both internal and external panel members. There is a formal panel on 

leader development, a small TRADOC panel on training, and a mixed military/science 

panel on personnel. The ARI staff who work in a given area of technology attend these 

reviews and receive instant feedback. The advice is usually verbal and not in written 

reports. 
12

Chapter 3: Peer Review at Some Other Government 

Laboratories 

At this point it is useful to review approaches to peer review used at other DOD 

laboratories and at laboratories at other Federal agencies, either owned and operated by 

the government or contracted by the government to private entities. It should be noted 

that, as was the case with the Army laboratories, peer review is done in several different 

ways at these other laboratories. One type of review is that done by the National 

Academies’ National Research Council (NRC). The NRC has study boards for all three 

Services. The Boards may perform in-depth technical reviews. The Defense Science 

Board (DSB) has reviewed the Defense laboratories, but not in terms of specific technical 

work. Rather, the DSB has looked at DOD’s policies for the laboratories—a higher level 

of review. One aspect of external review is that the Federal Advisory Committee Act 

(FACA) may govern the operation. This means that meetings must be public, members of 

the public may make presentations to the committee, and members are appointed only 

with concurrence of the President’s agents (typically the General Services Administration 

or agency heads). These requirements would have infringed on the traditional operations 

of the National Academies. After consideration, the FACA was amended to exempt the 

Academies from many of the requirements. 

The Services have traditionally convened their own advisory committees to look at 

technology issues, e.g., the Army Science Board, the Air Force Science Advisory Board, 

the Naval Studies Board, and the Naval Research Advisory Committee. These groups 

differ in their operations. Some are involved in detailed technical studies, others restrict 

themselves to policy studies. All of them come under the FACA. On occasion, ad hoc 

advisory committees may be set up, often by direction of the U.S. Congress. Two 

examples are the Federal advisory commission established to review the Services’ plans 

for realigning their laboratories during the base realignment and closure activity for 

1991,6 and a committee chartered by the Congress to study the future relevance of DOD 

laboratories.7

 A series of reports on visits by the latter committee to several DOD 

laboratories was issued in 2002.8

 These reports were the result of one-time reviews, after 

which the committee in question ceased to exist. Congressional committees, the 

Government Accountability Office, and even the Library of Congress have also done 

policy studies of the laboratories. 

 

6

 Federal Advisory Commission on Consolidation and Conversion of Defense Research and Development 

Laboratories, Report to the Secretary of Defense (Washington, DC: Department of Defense, September 

1991). 

7

 National Defense Authorization Act of FY 2000, S-1059, 106th Congress, Section 913. 8

 National Defense University, Section 913 Report #1, #2, #3, (Washington, DC: Center for Technology 

and National Security Policy, 2002). Copies of this report are available from the Office of the Director, 

Center for Technology and National Security Policy, National Defense University. 
13

Assessment of the Laboratories at the National Institute of Standards and 

Technology 

The programs at the National Institute of Standards and Technology (NIST), formerly the 

National Bureau of Standards (NBS), have been assessed by the National Academies’ 

National Research Council (NRC) since the 1950s. This was among the first regularly 

recurring quality reviews of a Federal laboratory by the NRC, and it continues to this day. 

(Recently the review has been emulated at the Army Research Laboratory. See discussion 

of ARL in chapter 2.) 

Peer review by the NRC began at NBS after a prolonged controversy over NBS’ 

evaluation of a commercial product. After these issues were settled in NBS’ favor, the 

Secretary of Commerce asked the National Academy of Science (now the National 

Academies) the parent of the NRC, to study various aspects of NBS activities. One result 

was the establishment of a separate NRC board to assess on a regular basis the quality of 

NBS work and the factors affecting that work. A set of evaluation panels was created 

under the NRC Board on Assessment for the National Bureau of Standards (now the 

NRC Committee on NIST Technical Programs). These panels met once a year (more 

recently every other year), generally for 2 days, to review technical work, visit the 

laboratories, view buildings and equipment, and talk with staff. They provided a written 

report vetted by the NRC and published to the general public. Typically there were (and 

are) separate panels for each major operating unit at NBS. 

 

The panels occasionally look at cross-cutting research that spans more than one division. 

On occasion, the NIST director requests attention to certain areas, e.g., quality of support 

services, state of buildings and working laboratory spaces, and status—age, morale, and 

so on—of technical staff. 

From the outset, the panels were made up of experts drawn mostly from outside the 

Federal government. Members may be nominated by anyone, but the selections are 

controlled by the NRC. Panel members are not paid, except for travel expenses. (A firm 

NRC policy holds that none of its committee members should receive remuneration.) The 

assessment work is scheduled and managed by NRC staff dedicated to the purpose. 

Reports are published by the National Academies Press and are not restricted. 

The individual panels are overseen by the NRC Committee on NIST Technical Programs. 

The chairman of the committee often has joined in discussions with the NIST statutory 

visiting committee about NIST, both in presentations to the visiting committee and in 

presenting findings to the Secretary of Commerce and the NIST authorization committees 

in Congress. 

Other Approaches 

Interviews with senior officials at a number of Federal laboratories have revealed several 

different models for peer review. Some are formal review panels that come under the 

FACA; others are formal but managed internally by the Service or the laboratory. An 

example of one managed by a Service is the review by the Air Force Scientific Advisory 

Board (SAB). 
14

The Air Force Research Laboratory—The SAB, a formal Federal Advisory 

Committee, was set up shortly after World War II by General Henry (“Hap”) Arnold and 

headed by Dr. Theodore von Karman. At first known as the Scientific Advisory Group, it 

prepared a landmark series of reports known collectively as “Toward New Horizons.”9

This work focused on the then state-of-the-art and outlined detailed plans for research in 

the sciences and technologies deemed critical to the future of air power. The group 

reported directly to Gen. Arnold. Today, as the SAB, it continues to report to the Air 

Force Chief of Staff and the Secretary of the Air Force. Among other duties, the SAB 

reviews in depth the scientific work of the Air Force Research Laboratory (AFRL) on a 

2-year cycle. The review covers both the 6.1-6.3 work but also work sponsored by 

downstream users, such as program managers or executive officers. One panel has been 

created for each AFRL directorate. Panel members are selected by the SAB; all are 

external to the Air Force. Most are members of the SAB, but when necessary the SAB 

brings in consultants. At least one member of each panel must be a member of the 

National Academies. 

The SAB conducts reviews of each directorate, devoting a full week to each. The panels 

provide an exit briefing and a formal report. The reports are sometimes made available to 

the public, but some may be restricted (FOUO, for official use only). The panels look at 

four factors for the programs: technical, relevance for the near term (5 yrs), future 

impacts, and resources. When evaluating the 6.1 programs, the SAB does not look at 

near-term impacts. For the technical work (factor 1), SAB evaluates technical innovation, 

technical rigor, productivity, and collaboration. 

The Naval Research Laboratory—Two external groups have been established to look 

at various technical topics for the Navy: the NRC’s Naval Studies Board, and the Naval 

Research Advisory Committee, created by the Chief of Naval Operations. Both groups 

consider the impact of technical developments on the future of naval forces. For detailed 

technical review of its technical base research programs, the Naval Research Laboratory 

(NRL) establishes and manages its own peer review. NRL has seven focus areas: 

materials and chemistry, electronics, battlespace environment, underseas warfare, 

electromagnetic warfare, space research/space technology, and information technology. 

Technical review of the technical base covers about one-third of the total research 

program each year. NRL selects members for the external review panels for each focus 

area. A panel typically has four to six members drawn from academia and elsewhere, 

along with at least one member of the National Academies. The NRL asserts that these 

members are unbiased. The panels meet for from two to four days with time for 

immersion in the labs and staff. The panels give exit briefings to the NRL management. 

Subsequently, they submit a formal written report of about 10-15 pages, and NRL 

responds to this in writing. 

 

9

 Michael H. Gorn (editor), Prophecy Fulfilled, ‘Toward New Horizons’ and its Legacy (Washington, DC: 

Air Force Historical Studies Office, 1994). Available at <http://www.airforcehistory.hq.af.mil/ 

Publications/authorindex.htm/> 
15

The panels evaluate the programs for scientific merit: they examine the research 

approach, the credentials of the staff, the project’s relevance, equipment, and costs. 

Details of these differ for 6.1 and 6.2. For 6.1, the panels look for work that seeks to 

expand the frontiers of known science; for 6.2 the panels look for whether NRL is 

investigating and developing recent advances in science and technology. They have seven 

categories of evaluation containing metrics. For the customer-funded work, the criteria 

used are those of the customers. 

The National Institutes of Health—The NIH operates peer review by nineteen formal 

advisory committees, termed boards of scientific counselors, one for each of the 

institutes. Their duties are described as follows: “Boards of Scientific Counselors serve a 

dual function in providing expert scientific advice to Scientific Directors regarding 

particular employees and projects, and providing the NIH as a whole with an assessment 

of the overall quality of its intramural efforts.”10

There is a Committee Management Office at NIH to track these and many other NIH 

advisory committees. The office stays in contact with the FACA Office of the General 

Services Administration. The NIH process is the most formal type of quality review 

discovered in interviews for this study. 

Three National Laboratories of the Department of Energy—Los Alamos National 

Laboratory (LANL), Lawrence Livermore National Laboratory (LLNL), and Sandia 

National Laboratory (SNL) are government-owned, contractor-operated facilities under 

the Department of Energy (DOE). Interviews at these three laboratories revealed very 

similar processes for peer review. The Department of Energy National Nuclear Security 

Administration (NNSA) contracts for LANL and LLNL explicitly call for regular peer 

review; Sandia’s contract contains similar language. LANL and LLNL are operated by a 

consortium of the University of California and the Bechtel Corporation, with additional 

partners from the Babcock and Wilcox Company and the Washington International 

Group. Two limited liability corporations have been created to manage the contracts: the 

Los Alamos National Security LLC, and the Lawrence Livermore National Security 

LLC. There is a joint Board of Governors for the two LLCs. The Board has a 

Subcommittee on Science and Technology (the S&T Committee) that oversees and 

controls peer reviews of S&T at both laboratories. The S&T Committee makes the 

appointments to the peer review panels upon nomination by the laboratories. The panels 

are independent and balanced. Each does both review and critique. The panels have eight 

to ten members drawn from academia (about half), industry, and other labs; there are 

some University of California faculty, and some from other National Laboratories. The 

operation of the panels does not come under the FACA. 

Los Alamos National Laboratory conducts three kinds of review. The first review is 

strictly concerned with quality of scientific activity and is focused on capability areas 

rather than on scientific disciplines. Typically, these are crosscuts from the discipline 

 

10 Directory of NIH Federal Advisory Committees and HHS Advisory Committees Managed by NIH Staff, 

(Washington, D.C.: National Institutes of Health, January 2008), 6. Available at

<www1.od.nih.gov/cmo/committees/directoryofnihadvisorycommittees>. 
16

areas, such as weapons science and information science. They cover eight capabilities per 

year; each capability area is reviewed every 3 years. The review panels meet for 3–4 

days. (In earlier times these reviews were run strictly in-house and covered only scientific 

disciplines, not capabilities.) The panels also look at the adequacy of the lab 

infrastructure, the morale of the staff, and the research environment. The other reviews 

are on weapons design and customer programs. The design reviews are done internally by 

DOE weapons design teams; the customer reviews look at quality, relevance, and 

performance against the mission. Customer reviews are set up by the laboratory subject to 

approval by the Board of Governors’ S&T Committee. 

Lawrence Livermore National Laboratory follows a similar assessment process. At 

LLNL, there are four principal disciplines and three major program areas. All are 

reviewed by peer panels. In addition, they conduct cross-cutting reviews of portfolios (for 

example, the National Ignition Facility) involving more than one of the seven areas. The 

panels consist of 10–15 members drawn from academia, industry, and other laboratories. 

Members are selected by the LLNL directorates and vetted by the Board of Governors’ 

S&T Committee. Selection factors include diversity on the panels and turnover of 

membership. The panels usually have a member from LANL and someone from NNSA. 

Panel reports contain both critique and advice. Verbal exit briefings are given to the head 

of the unit under review, as well as to senior managers and the LLNL director. Reports of 

the panels are circulated within DOE but are restricted. 

Sandia National Laboratory is operated under a contract between NNSA and Sandia 

Corporation, a wholly owned subsidiary of the Lockheed Martin Corporation. There are 

six Lockheed employees on the Sandia Corporation Board of Directors. The 

Corporation’s S&T Committee is chaired by the Chief Technical Officer of Lockheed. 

About half of the $2 billion plus budget at Sandia National Laboratory is S&T. 

Peer review at Sandia is divided between S&T and nuclear weapons work. For S&T, the 

reviews are conducted under contract to the University of Texas (UT), which has two 

positions on the Board of Sandia Corporation. Under supervision by the Sandia 

Corporation Board of Directors Subcommittee on S&T, UT selects and convenes review 

panels. Both Sandia management and Sandia Corporation have input into the final 

selections. 

Each panel has 6–12 members from various disciplines, including physical sciences, 

computation, electronics, and materials. UT draws some panel members from its faculty; 

other members are drawn from wherever they can find qualified individuals. There is an 

external panel for each competence. Panels meet for multi-day sessions and cover 

individual projects/programs. The panels meet with staff members to assess morale and 

the research environment. They also may meet with groups of principal investigators at 

the program level, as well as with individual project leaders. The review results are 

reported to laboratory management and the Board of Directors’ S&T Committee. 

Principal Investigators receive the reports and must respond to panel critiques. 
17

Nuclear weapons-related peer review at Sandia includes three kinds of internal reviews: 

design, management, and internal peer review (members from entities other than the 

design team under review). Reviews are assisted by a full-time office of assessment that 

reports directly to the laboratory director. The assessment staff is internal but separate 

from the program areas. There is also a standing panel for external independent review of 

these Sandia assessments. These assessments are a critical part of the regular certification 

of America’s nuclear weapons stockpile. 

Peer review at grant-making organizations 

Entities that award grants—for example, the National Science Foundation (NSF), the 

Army Research Office (ARO), the Office of Naval Research (ONR), and the Air Force 

Office of Scientific Research (AFOSR)—do not operate laboratories and, therefore, 

conduct quality reviews in a different manner. NSF is overseen by the National Science 

Board that reports to the Congress. Each NSF directorate is monitored by a formal 

advisory committee that meets regularly to review performance. Periodically, members of 

the committee review the grant folders to ensure that procedures have been followed. 

Grant proposals are sent out to experts for evaluation; subsequently the folders are 

evaluated by NSF staff before the decision to award or not to award is made. For work in 

progress, grantees are visited on site by NSF program managers. Quality is judged by 

these reviews, regular reports, and copies of publications. The ultimate test of how well a 

grantee is doing is the renewal or termination of the grant. This is true for all the granting 

agencies. 

ARO conducts two kinds of peer review concerning single investigator (SI) proposals for 

new work. One review evaluates technical merit. Typically the proposal is sent to 

external reviewers, mostly university faculty. The other review focuses on military 

relevance and is done by Army and DOD scientists and engineers. The SI grants are 

typically for 3 years. Two or more site visits are usually conducted during this time. 

Program managers will audit grantees presentations at scientific meetings and gauge 

audience reaction. ARO receives formal annual reports and copies of all publications by 

its grantees. 

The ARO divisions are evaluated biennially by external boards of visitors, one for each 

division. The boards look at the overall portfolios, evaluating the strategic direction of the 

divisions and looking out for overlap with other programs in the DOD or elsewhere. 

AFOSR, a directorate within AFRL, manages the 6.1 funds for the Air Force. Funding 

executed by AFRL internal research directorates is evaluated by the SAB during the 

biennial reviews of those directorates. The AFOSR as a whole is reviewed by the SAB 

every 2 years. 

Office of the Secretary of Defense 

The Director of Defense Research and Engineering for a number of years conducted 

Technology Area Reviews and Assessments (TARA) reviews that covered DOD basic 

research programs. Representatives of the Service laboratories and operating commands 
18

participated in these reviews, along with outside experts. TARA reviews are no longer 

conducted. 

Clearly there is no single, accepted best way to conduct peer review of Federal S&T 

laboratories. Chapter 4 summarizes the results of the preceding investigation into the peer 

review process. Recommendations are presented in chapter 5. 
19

Chapter 4: A Summary of Peer Review Practices 

There are almost as many approaches to peer review as there are laboratories contacted 

for this study. Laboratory practices range from practically no review to formal reviews 

conducted under the Federal Advisory Committee Act. The following discussion presents 

the range of practices uncovered in interviews and literature study. This is followed by 

consideration of the pros and cons of the several approaches. Chapter 5 offers a set of 

performance guidelines for the Army. 

Most of the laboratories studied have some kind of peer review. At one end of the 

spectrum stands NIH, which has an external advisory committee for each of its institutes. 

These committees operate in the public eye under the FACA. NIH review processes 

operate under a very formal arrangement. AFRL has a formal arrangement with the Air 

Force Science Advisory Board (SAB), operated from the offices of the Chief of Staff and 

the Secretary of the Air Force. Thus, the SAB is once removed from the laboratory. The 

SAB also takes its authority from the FACA. Another formal process is the use of 

dedicated boards and committees of the NRC, under contract with the laboratories. 

Several laboratories manage their own review processes, some of which are subject to 

oversight by a higher level of management. The DOE weapons laboratories are reviewed 

by expert panels, whose nominations from laboratory management are subject to 

approval by the contractor that operates the laboratory. Some of the Army laboratories 

contract out at least a portion of the process. Laboratory directors generally value outside 

review, but emphasize that the reviewers must be experts in the specific sciences and 

technologies being studied at their laboratories. In cases where the technologies are 

highly specialized, there may be only a small external community from which to draw 

reviewers. Furthermore, these may be somehow involved in the laboratory’s research, 

making conflict of interest a problem. (NSF and the NRC address this issue not by 

disqualifying an otherwise desirable expert, but rather by making the potential conflict 

known and balancing the various interests that may exist on their panels.) 

Some experts are drawn from universities or industry, while some come from within the 

Army or other government laboratories. Several Army laboratories use exclusively 

external review panels. Most of the others have some external and some internal 

members. No evidence was found of unresolved conflicts of interest. 

The scope of the peer reviews varies. Some reviews are strictly limited to the quality of 

the research, and to assessment of the quality of the staff and facilities. Other reviews 

address a mixture of quality and relevance, while still others appear limited to questions 

of relevance. Typically, the quality-oriented reviews employ many academics who 

specialize in the subjects of interest. The relevance assessments are carried out by 

customers, such as program executive officers and program managers; or representatives 

of soldiers, such as senior Army officers from the Training and Doctrine Command. 
20

Most of the Army’s technical peer reviews are aimed at evaluating individual project 

areas. A few look more broadly at the laboratory’s overall efforts. ARL’s TAB looks at 

all the technical work, both S&T-funded and customer work; the Air Force’s SAB 

similarly looks at all the AFRL work. An important question is whether or not the review 

extends beyond the technical work itself to broader questions, such as the status of staff, 

equipment, facilities, and the work environment. These judgments would require that the 

panels make extensive visits to the laboratories, talk to staff, and generally familiarize 

themselves with the environment. . 

Most of the panels under consideration here prepare formal written reports in addition to 

the informal feedback provided during the reviews. The reports are sent to managers and 

are available to the research staff. In some cases they are available to the public. The 

reports prepared for the DOE weapons laboratories are marked FOUO. In some 

laboratories, local managers are required to report back the actions taken in response to 

recommendations made by the panels. 
21

Chapter 5: Closing Comments & Recommendations 

It is unlikely that any one model of peer review will suffice for the Army laboratories. 

The authors believe, however, that some form of independent external review of technical 

quality is necessary for every Army laboratory. The Army should require that these 

reviews be undertaken, even if differing in some details from laboratory to laboratory. 

A model that focuses on quality review of the technical work will apply well to 

laboratories that perform a large amount of basic and applied research (6.1 and 6.2). 

However, if a laboratory has a large component of advanced development work and 

beyond, this work too should be included in the review. In such cases, the panel experts 

will include technically trained individuals representing both customers and users. Some 

of the development and engineering centers will likely want to devote relatively more 

effort to issues of project relevance to the armed forces. A center that has little 6.1 

research will probably count on the DRLM basic research reviews. However, all the 

technical work, whether in the S&T program or in customer-sponsored work, should be 

reviewed for quality. This can be combined with other types of reviews, or evaluated 

separately. 

Table 1 contains a list of questions relevant to the design of the peer review process; 

answers to these questions should be useful for comparing and contrasting practices at 

various laboratories. 

Table 1: Important Questions to Consider in Peer Review

Is there any peer review at all? 

Who manages the review? 

Are external experts used in the review? 

Are the experts independent and free of conflicts of interest? 

Are the experts external to the project staff, or external to the laboratory? 

Are the experts external to the Army? 

Does the review focus on issues of quality, relevance, or both? 

Does the review focus on individual projects, or on the laboratory as a 

whole? 

Does the review consider the quality of staff, equipment, facilities, or the 

working environment? 

Are explicit, formal reports produced? 
22

A uniform Army definition of the term external review should be established. A 

distinction should also be made in Army terminology between quality peer review and 

relevance review. Whether these reviews should be done separately or in combination is 

an open question that may be answered differently in different laboratories. Similarly, the 

expertise required of reviewers will vary. External review panels should be balanced 

between the requirements of expertise and “distance,” from the reviewed parties (to avoid 

damaging conflicts of interest). Contracting with an outside entity, such as the NRC or 

some other independent contractor, to convene review panels and manage the process 

will enhance the credibility of the results. Alternatively, reviews could be managed by an 

entirely separate internal organization, as is done in the DRLM Basic Research Reviews. 

(The BRR is a hybrid approach wherein the panels are external experts but the naming of 

members and convening of the panels are directed by the DRLM. The important point is 

that credibility is enhanced by independent reviews. 

The scope of reviews should cover individual projects in enough detail that quality of the 

work is exposed. If a single panel is used for an entire laboratory, the sampling of 

projects and presenters will be less likely to give a reliable picture than if multiple panels 

are used, each focusing on a particular set of disciplines or programs. A trade-off must be 

made between cost and the depth of the reviews. 

To evaluate not only the specific projects that are presented but also to assess the research 

environment—staff, management, equipment, and facilities—a panel will need to spend 

2–3 days at the laboratory to allow walking around, talking to staff, and getting a feel for 

the ambience of the laboratory. Simply sitting in a conference room will only give an 

impression of the projects selected for review, and little else. 

Review panels should provide informal feedback to the management and staff at the time 

of the review, but should also submit a formal written report. This report should be 

provided to interested parties within the laboratory and to managers up the chain of 

command. Internal and external organizations charged with oversight of the laboratory 

should receive the reports. Part of the process should be to require the laboratory to 

respond to panel recommendations at the next review. 

Recommendations 

The following recommendations are offered as the basis for a directive from the Deputy 

Assistant Secretary of the Army for Research and Technology (DAS(RT)) to laboratory 

managers. Since there is unlikely to be a single mode of review, the recommended 

approach is to establish guidelines that will set a minimum performance bar that is to be 

met or exceeded by all Army S&T laboratories. 

The DAS(RT) should, in consultation with laboratory management, issue a written policy 

on peer review covering all Army S&T laboratories. The policy should be accompanied 

by detailed guidance along the lines of the following recommendations: 

Recommendation 1. The policy should require the laboratories to empower outside 

groups to convene peer review panels and manage the review process. The policy should 
23

point out the enhanced credibility of panels that are so convened. These arrangements 

will normally be created through the use of contracts. The reputation of the contractor 

should be an important factor in the selection process. Specific steps should be taken to 

ensure that conflict of interest on the part of the proposed experts is dealt with either by 

disqualifying the candidate or, in the case of very small pools of available experts, 

making the potential conflict known in advance and achieving balance by selection of the 

other members. Laboratory managers may suggest panel members but should not control 

the appointments. Membership will vary, depending on whether the laboratory elects to 

have separate reviews for quality and relevance or consolidated reviews. 

Recommendation 2. The policy should require that the reviews themselves be done by 

panels of experts external to, and independent of, the laboratory. For review of technical 

quality, the panels should consist of researchers in the same fields drawn from academia, 

industry, research institutes, and other government laboratories. For review of relevance, 

the panels should have both technical experts and representatives of the user, such as 

program executive officers and program managers and the TRADOC. Retired Army 

officers can be of help in both quality and relevance reviews. 

Recommendation 3. Reviews should be done every 2 or 3 years. To spread out the 

burden of handling the reviews, the annual review process should be staggered such that 

any one area is only reviewed every 2 or 3 years. 

Recommendation 4. The reviews should cover technical details at the project level. 

Enough time should be allowed for the panel to hear many different project presentations, 

selected to give a true picture of the laboratory’s work. One way to provide good 

coverage of the laboratory programs is to have separate panels for each major segment of 

the laboratory program. The panels should be allowed to spend 2 or more days in review. 

Recommendation 5. The panels should also assess the quality of the staff, the 

management environment, the equipment, and the facilities. This can be done by 

encouraging panel members to walk through the laboratory and to talk to staff. 

Recommendation 6. The panels should provide feedback to the laboratory staff and 

prepare formal written reports. Reports should be submitted to the laboratory 

management, and copies provided to relevant members of the chain of command. Copies 

of the reports should be filed in the DAS(RT) office. The laboratories should be required 

to respond to panel recommendations or critiques at the next meeting. 

If this set of recommendations is adopted and executed, the result should be improved 

quality of the Army’s technical work, enhanced reputation of the laboratories, and 

strengthened support of the laboratories among the Army’s senior leaders, as well as 

within the Administration and in Congress. 
24

Appendix A: Interviewees 

The following individuals were interviewed for this report: 

• Air Force 

o Air Force Research Laboratory—Michael Kuliasha, Chief Technologist 

• Navy 

o Naval Research Laboratory—John Montgomery, Technical Director 

• Army 

o Army Materiel Command 

 Research, Development, and Engineering Command—Gary Martin, 

Deputy to the Commanding General 

• Aviation and Missile RDEC—William McCorkle, Technical 

Director 

• Armaments RDEC—Joe Lannon, Technical Director 

• Army Research Laboratory—Joe Mait, ST 

• Army Research Office—Dave Skatrud, Director 

• Communications and Electronics RDEC—Art Ballato, ST 

(with Geo. Oliva) 

• Edgewood Chemical and Biological Center—Rick Decker, 

Technical Director 

• Natick Soldier RDEC—Marilyn Freeman, Technical Director 

• Tank and Automotive RDEC—Grace Bocheneck, Technical 

Director 

o Army Research Institute of the Behavioral and Social Sciences—Michelle 

Sams, Director 

o Engineering Research and Development Center—James Houston, Director 

o Medical Research and Materiel Command—Frasier Glenn, Principal Assistant 

to the Commanding General for Research and Development 

• Department of Energy 

o Lawrence Livermore National Laboratory—Lee Younker, Director, 

Operations and Assessment 

o Los Alamos National Laboratory—Terry Wallace, Principal Deputy Director 

for Science 

o Sandia National Laboratory—Joan Woodard, Deputy Laboratory Director for 

Nuclear Weapons (with Gary Sanders)