Suggestions for Evaluating the Quality 

of the Army’s Science and Technology 

Program: 

The Portfolio and Its Execution 

John W. Lyons, Richard Chait, and James Ratches 

Center for Technology and National Security Policy 

National Defense University 

January 2013 
The views expressed in this article are those of the authors and do not reflect the official policy 

or position of the National Defense University, the Department of Defense, or the U.S. 

Government. All information and sources for this paper were drawn from unclassified materials. 

John W. Lyons is a Distinguished Research Fellow in the Center for Technology and National 

Security Policy (CTNSP) at the National Defense University. He was previously Director of the 

Army Research Laboratory and Director of the National Institute of Standards and Technology. 

Dr. Lyons received his Ph.D. in physical chemistry from Washington University at St. Louis and 

his B.A. from Harvard University. 

Richard Chait recently retired as a Distinguished Research Fellow in CTNSP. He was 

previously Chief Scientist at the Army Materiel Command; and Director, Army Research and 

Laboratory Management. Dr. Chait received his Ph.D. in solid state science from Syracuse 

University and a B.S. from Resselaer Polytechnic Institute. 

James Ratches is a Senior Research Fellow in CTNSP, and on the staff of the Army Research 

Laboratory’s Sensors and Electron Devices Directorate. He was previously Chief Scientist at the 

Army’s Night Vision and Electronic Devices Directorate. Dr. Ratches received his Ph.D. in 

physics from Worcester Polytechnic Institute and a B.S. degree from Trinity College. 

Acknowledgments. The authors wish to acknowledge the support of Dr. Marilyn Freeman, 

formerly the Deputy Assistant Secretary of the Army for Research and Technology; and Dr. 

Linton Wells, Director of CTNSP. 

ii 
iii 

CONTENTS 

I. Introduction ................................................................................................................................. 1

II. The Army S&T Portfolio ........................................................................................................... 2

III. The Basics of Evaluating the Portfolio ..................................................................................... 5

IV. Evaluating the Quality of the Program and Its Execution ...................................................... 12

V. Summary of Report and Recommendations ............................................................................ 19

VI. Appendix............................................................................................................................... A-1
I. INTRODUCTION 

The U.S. Department of Defense (DOD) Deputy Assistant Secretary of the Army for Research 

and Technology requested that the Center for Technology and National Security Policy (CTNSP) 

suggest approaches to evaluating the quality of the current U.S. Army Science and Technology 

(S&T) portfolio and the quality of its execution. This paper contains the results of both studies. 

There are two aspects of the U.S. Army S&T program that are evaluated in this paper: the 

portfolio and the quality of its execution. This paper poses the questions that should be asked. 

Emphasis for the analysis of the portfolio is on program content, the distribution of resources 

over the component parts of the portfolio, and the priorities among the parts. Normally in an 

assessment of this type it is necessary to identify the principal stakeholders and customers for the 

research and development program. In the case of Army S&T, the stakeholders are the senior 

leaders of the Army, DOD, Congress, and, ultimately, the American taxpayer. The customers are 

the U.S. Army warfighters. The study begins by outlining a clear statement of the vision and 

mission of the parent organization—the U.S. Army. We continue by describing in detail the 

needs and requirements of the U.S. Army by the Training and Doctrine Command (TRADOC), 

the Army senior leadership, and from various intermediary customers such as the U.S. Army 

Research, Development and Engineering Command and its components, and the Program 

Executive Offices and the Program Managers of the acquisition community. The discussion then 

turns to the mechanisms by which the portfolio is assembled and approved, as well as the 

limitations posed by the realities of the budget details of resource allocation. 

To be in tune with the Army’s needs now and into the future means that the portfolio should 

have a balance of programs between long-term basic research, medium-term applied research, 

and short-term work that can be transitioned into advanced development and beyond. The 

amount of work in each category will vary according to the nature of the individual laboratory’s 

assignment. Also there is a balance between the work done in-house and that done extramurally 

on contracts and grants to the private sector or by other government laboratories. 

To assess the quality of the execution of the program requires evaluating the strength of the 

Army laboratory system in terms of its managers, its technical staff, and available facilities and 

equipment. This evaluation can best be done by some form of peer review by subject matter 

experts. Additional evaluations should be made by the Army S&T Office, bringing in long-range 

technology forecasting to assure that the laboratory system will be prepared for future demands. 

Some form of benchmarking is needed to assess where the best work can be done. Collaborations 

can then be established, typically by contracts and grants to outside laboratories. 

The objective is to have the strongest possible technology program appropriately aligned to the 

needs of its customers and the expectations of its stakeholders. In this paper we present 

methodology that should accomplish this goal. In chapter II, we first present the essential 

elements of the Army S&T portfolio. The chapters that follow describe the Army S&T portfolio 

and the evaluation of it, the evaluation of the quality of the program of execution, and a summary 

of the report. 

1 
II. THE CURRENT ARMY S&T PORTFOLIO 

Analysis of the portfolio should begin with a study of how the current portfolio came into being. 

It is based on a long series of budget decisions made by the Army and Congress. These decisions 

were based on the threats facing the Army at the time and with forecasts of likely threats in the 

future. When we are at war, the laboratories are necessarily more focused on the short term. In 

the 2000s the intense focus on the wars in Afghanistan and Iraq and development of the Future 

Combat System shortened the time frame of the S&T program. Recent studies have expressed 

concern that the current time frame is too short.1 The Army is now undergoing a period of force 

rebalancing and downsizing. The mission is shifting away from counterinsurgency in 

Afghanistan and Iraq to the Asia-Pacific region. 

With the end of the Cold War, the Army underwent a dramatic downsizing. The S&T program 

was markedly reduced, sometimes by as much as one-half in budgetary terms. The reduction in 

the laboratory budgets removed many programs of lower priority; overhead costs were reduced 

to a minimum. This belt tightening left the most important research programs intact. The past 

decade of counterinsurgency led to some of the budget reductions being restored to mount 

programs directed at the wars in the Middle East. 

Also during the 1990s and beyond, the Army S&T management turned outward in the sense that 

it established some very large extramural collaborations. Centers of excellence included the 

Institute for Soldier Nanotechnology, the Institute for Creative Technologies, and the Institute for 

Collaborative Biotechnologies. The Army also has funded the Institute for Applied Technology 

at the University of Texas for studies of electric guns. The focus of this work has recently been 

transferred to the United States navy. These institutes are university-based, involve several 

different disciplines, and consist of formal or informal consortia from industry and academe. A 

similar approach was used in the creation of several extramural consortia comprising academic 

and industrial members at the Army Research Laboratory (ARL). These consortia were tied 

closely to in-house ARL programs in that normal grants and contracts are operated at arm’s 

length from the in-house staff, that is, once the contract or grant is awarded, interaction with 

Army personnel is not required until renewal discussions are held. These ARL collaborative 

centers of excellence—Collaborative Technology Alliances and Collaborative Research 

Alliances—differ from ordinary external centers of excellence in that they are designed to 

operate parallel with in-house research conducted by Army laboratory staff with the very best 

people in the private sector. This type of management involves rotation of staff into and out of 

the Army laboratory; such rotation both improves mutual understanding of the requirements for 

research and speeds transfer of new technical information. These collaborations are intended to 

ensure that the Army knows and can work with the best and the brightest people elsewhere. The 

result is that the work is first class, and the Army technical staffs are able to adapt and help adopt 

these new developments for use in other sectors of the Army. 

 

1

 Report of an Expert Panel on the Future of Army Laboratories, Improving Army Basic Research (Arlington, VA: 

RAND, 2012); Final Report of the Army 2010 Army Acquisition Review Chartered by the Secretary of the Army, 

Army Strong: Equipped, Trained and Ready (Washington, DC: Office of the Secretary of the Army, 2011). 

2 
The current Army S&T portfolio has seven segments: 

• Basic Research 

• The Soldier (nonmedical) 

• The Soldier (medical) 

• Ground 

• Command, Control, Communications, and Intelligence (C3I) 

• Air 

• Enduring Technologies. 

Each of these segments is subdivided into six or seven component programs. For this discussion, 

we classify the research in three categories: Army Unique, Army Shared, and Dominated by 

Others. Army Unique covers work that others have little or no motivation to work on and the 

subject matter is critical for the Service. In these circumstances, the Army maintains a robust 

research program in house benefiting occasionally from work done elsewhere. There will 

normally be a strong basic research component to ensure that the Army anticipates and is 

prepared for future developments in the technology and in threats involving that technology. 

Army Shared includes research topics where both the Army and others are motivated to push the 

state of the art. Such topics include sensors, hardened electronics, and situational awareness on 

the battlefield. For these types of technologies, the Army has active research programs 

addressing specific applications important to the Army and where the work of others is not 

focused on Army interests. The Army work is tailored to meet specific requirements provided by 

the representatives of the warfighters. Long-term basic research is limited to these specific needs 

and to solving unanticipated problems that arise in the Research and Development program. In 

shared areas, various forms of collaboration are employed, ranging from informal cooperation on 

specific points to formal collaborations under contract agreements. The Collaborative 

Technology Alliances and the Collaborative Research Alliances fall in the latter category. 

The third category, Dominated by Others, includes technology where the private sector or 

another part of the government is dominant and the Army relies on them for research. For 

example, in the development of new computers, the Army’s role in research is to help the 

acquisition community adapt the technology to special needs of the warfighter. It is unlikely that 

there will be a long-term basic research program in this category. 

Parsing the portfolio into these categories, however, is judgmental; people will differ about 

where to place a particular portfolio element. For example, the desire of the Soldier to see well at 

night led to the Army developing a dominant position in night vision technology. Army success 

produced market opportunities in the commercial sector so that, today, there is technical 

competence in industry as well. It is also true that, in fielding a technology such as night vision, 

industry has to learn how to manufacture the devices and components the Army needs. So there 

is expertise in various companies fostered by the Army. But the Army is maintaining its 

technical lead in night vision by developing and evaluating new concepts for next generations. 

Thus today both the Army and industry are competent in night vision technology, even though 

the Army will continue to press new developments. 

3 
Another example is in armor for the main battle tank. The armor for the Abrams tank grew out of 

some work done in the United Kingdom, the Chobham armor concept.2

 The Army was working 

on this in parallel with the British. Subsequently Army research added more modifications so 

that today Army armor leads the technology. Given the importance of survivability of the tank 

and its crew the Army is maintaining its research on armor and is pushing the limits wherever 

and however it can. 

The situation with armaments is similar. The Army has always had technical programs to 

improve its weapons and holds the lead for many. The work at Benet Laboratory and Watervliet 

Arsenal has given the Army a strong position in developing and manufacturing large bore guns 

both for itself and for others in the military. There is no motivation for the private sector to 

develop cannon technology. Current problems being addressed are in manufacturing materials, 

material coatings, and in manufacturing processes. An example of the importance of solving 

these problems is that the dimensional precision and accuracy of a gun tube, along with its 

stability under sustained firing, contributes to the accuracy of the weapon. Research on the 

factors affecting these errors have been a major focus over the years, and are a special niche for 

the Army weapons research program. One solution to this problem was the development of GPS 

fuzes for artillery rounds to reduce the number of rounds needed to hit a target. However, it took 

time to create a fully functioning weapons system. Early models only showed the path of the 

round and where it would hit, while later efforts added guidance to the fuze ensuring a very large 

reduction in the impact error calculations. The result is the Excalibur artillery round, which has 

been fielded in Afghanistan. The design of the fuze is such that it can be screwed on to rounds in 

the current stockpile. This is an example of an “Army Unique” capability. 

 

2

 Richard Chait, John W. Lyons, and Duncan Long, Critical Technology Events in the Development of the Abrams 

Tank: Project Hindsight Revisited, Defense & Technology Paper 22 (Washington, DC: Center for Technology and 

National Security Policy, December, 2005). 

4 
III. THE BASICS OF EVALUATING THE PORTFOLIO 

The process of evaluating the portfolio may be done by internal staff at the office of the Deputy 

Assistant Secretary of the Army for Research and Technology, by others in the Army Secretariat, 

by the Office of the Secretary of Defense, or by external experts. In any case, the review should 

proceed somewhat as described in the following sections of this paper. A first step is to ask a 

series of questions such as the following: 

• How does the work support the Army mission? 

• What is the priority given to the area by TRADOC and Army leaders? 

• Is the proposed technology essential or just nice-to-have? 

• What is the projected impact? Compared to other Army S&T programs? 

• What is the relation to forecasted trends in technology and in threats? 

• Is the area within the scope of the Army’s current programs? Is it within the expertise of 

the S&T staff? 

• Can the Army develop collaborations with others already competent in the field? 

• Can the private sector or other government entities provide the technology needed? If so, 

what and how much research would be needed to adapt their work to Army needs? 

• What would the absence of Army in-house expertise mean to the warfighter? 

• What is the Army’s expertise in this area? 

• What expertise exists elsewhere in the military or the private sector? 

• Is the Army dominant in the field? 

• What would happen if the Army were to stop research in this field? Are other sources 

ready and able to do the work? At what cost in time and money? 

• Is the size of the Army program at a critical mass? 

• Is the budget for personnel, facilities, and capital equipment adequate for the job? 

• Are the necessary resources already on board (for a new project launch)? 

• Is there provision for basic research to support the programs and ensure it is at the 

frontiers? 

• How competent are Army personnel in this field as measured by publications, patents, 

recognition, etc. compared to other similar facilities? 

• How will progress be assessed? Are there criteria for continuing or termination? 

To elaborate, consider the following approach. The review package and briefings should have 

the Component Technical Areas and Technical Descriptors. The Descriptors would include 

technical products that fill desired capabilities, list the Army contributors and funding, and the 

stakeholders. The final section should address four major categories: the quality of the technical 

staff and the technical managers, the relevance of the program, alignment of the program with 

the Army mission and priorities, and consistency with Army forecasts of future developments in 

technology. 

As an example, consider the area of C3I (Command, Control, Communications, and 

Intelligence), a technical area shared with industry. 

• Technical areas: intelligence and electronic sensors, other sensors, communications, and 

mission command 

• Products: sensors for: active/passive UV, visible, and Electro-Optical/Infra-Red; 

terahertz, mmw, radar, acoustic, magnetic, seismic, olfactory, and tactile sensors; sensor 

components, multisensors, sensor processing, and data fusion 

5 
• Capabilities: reconnaissance, intelligence, surveillance, and target acquisition (RISTA), 

navigation, pilotage, persistence surveillance, forensics, human intent and identification, 

health monitoring, early warning, missile seekers, and munitions guidance, detection of 

improvised explosive devices, countermine, chem./bio hazards, and robot perception 

under all weather conditions and terrains, and through walls. 

Contributors 

1. Communications and Electronics Research, Development, and Engineering Center: Night 

Vision and Electronic Sensors Directorate, Intelligence and Electronic Warfare 

Directorate 

2. Army Research Laboratory: Sensors and Electron Devices Directorate, Communications 

and Information Sciences Directorate, Human Research and Engineering Directorate 

3. Tank Automotive Research, Development, and Engineering Center 

4. Armament Research, Development, and Engineering Center 

5. Aviation and Missile Research, Development, and Engineering Center 

6. Edgewood Chemical and Biological Center 

7. Natick Soldier Research, Development, and Engineering Center 

8. Corps of Engineering Research and Development Center 

Customers 

1. Program Executive Office (PEO) Aviation 

2. PEO Ground Combat Systems 

3. PEO Missiles and Space 

4. PEO Soldier 

5. PEO Chemical/Biological 

6. PEO Command Control Communications-Tactical 

7. Medical Research and Materiel Command 

8. Training and Doctrine Command 

9. Logistics Innovation Agency 

10. Defense Threat Reduction Agency 

11. Edgewood Chemical and Biological Center 

12. Special Operations Command 

13. Joint Improvised Explosive Devices Defeat Organization 

14. Rapid Equipping Force 

15. Department of Homeland Security 

16. Department of Energy. 

6 
The data needed for evaluating the effort would be provided by the contributors and customers 

listed above.3

 The questions to be answered are, for example, those at the beginning of this 

chapter. 

Some Preliminary Assessments 

Ground and Air. The contents of the Ground portfolio suggest that most of those items are Army 

unique. Weapons, survivability, dealing with countermine/ improvised explosive devices and 

technical improvements to platforms are necessary for the warfighter. More effective sources of 

power, unmanned systems, and logistics are areas where the private sector has much to offer. 

Some, such as alternate fuels, are being addressed in very large programs by the U.S. 

Department of Energy. Subsequently, the Army should not invest heavily in this area except to 

adapt the findings to its needs. For the Air portfolio, the design and structure of helicopters is a 

shared area with industry. This is recognized in the Army’s National Rotorcraft Technology 

Center where cooperation among the players is the norm. Most of the topics are of special 

interest to the Army; some are also of interest to the private sector. Examples of the latter are 

better fuel efficiency for the helicopter engines, lightweight drive trains, improved reliability and 

durability, and reduced weight and vibration. Unmanned aerial vehicles are a topic of the highest 

priority for the Air Force at the current time, and the Army will learn from their work and adapt 

it to its fleet. 

C3I. Sensors, communications/computers, and command and control are essential items for the 

battlefield. Some will be specific to the Army, while others are of interest to the other services 

and to our military partners around the world. Everyone in and out of the military is interested in 

sensors. New developments in solid state physics continue to be made. The technology is moving 

to nano-scale devices. All of these influences will have an effect on future technology. Seen 

together, the impact on night vision, detectors for sensors on platforms of all types, and force 

protection warning devices will help improve the detection of movements, the launch of mortar 

shells, and sniper bullets on the battlefield. Communications and computer technology derive 

from the work of that industry. 

Developments in computers are the province of industry, but software is another matter. Special 

programs are needed as real-time decision aids for battlefield commanders. Many technologies 

are needed for the gathering and analysis of intelligence information. The amount of data 

collected by today’s satellites and surveillance aircraft is creating a need for better methods of 

mining and assessing this information in real- or near-real time. This is essential for today’s 

Army and will be even more so in the near future. However, data mining is of great interest in 

other agencies and the private sector. Researchers in astronomy and in atmospheric science deal 

with very large amounts of data coming in everyday, sometimes every hour. Thus, research in 

this area should be left largely to those sectors. Techniques for the fusion of data from different 

sources are also needed. A need highlighted in Iraq is for devices that can reveal the contents of 

buildings either by seeing through the walls or by entering surreptitiously via micro-autonomous 

robots. Research in both approaches is active in the Army. 

The Soldier. This portfolio is separated into two parts: one for nonmedical, the other for medical. 

The nonmedical section is divided into two subsections: the integration of humans with machines 

and measures of effectiveness regarding that integration, and how to improve the design of 

 

3

 For possible metrics, see Report of an Expert Panel on the Future of Army Laboratories, Improving Army Basic 

Research (Arlington, VA: RAND, 2012), 50–51. 

7 
equipment. Both these topics are of interest to many in and out of government because it is of 

pressing importance to soldiers on the battlefield and therefore there is justification for some 

work in the Army. Another area that has arisen recently is that of small unit operations. The 

Army is moving to a focus on independence in units as small as the squad. Studies are currently 

underway on communications within small units, as well as upward to high echelons. This has 

been and is especially important in operations in dense urban areas. The selection of Army 

warfighters and the training necessary to make them effective is always a priority. This includes 

finding ways to determine the toughness and resilience of new personnel to help improve their 

ability to surmount post-traumatic stress disorder and to maintain a high level of effectiveness on 

the battlefield. As this is far more prevalent in ground soldiers than members of other services, it 

may be of more importance to the Army than to other services. 

Medically, survivability of the troops and care of combat casualties are perhaps the top priorities 

of Army commanders. Topics such as improved body armor for protection of the head to 

minimize traumatic brain injuries are important priorities for the Army Medical Command. Part 

of this is improving the resistance of soldiers to infectious diseases. On the battlefield, ways are 

needed to monitor the vital signs of soldiers as they work and after they are injured. Research is 

ongoing on fabrics for uniforms that perform monitoring while serving the usual functions of 

cloth. Better means of stemming hemorrhage have been developed during the Middle East 

conflicts. This is a top priority in treating wounded on the battlefield. Finding markers in blood 

that can signal internal damage in the brain or elsewhere is a dream of medical researchers. 

Currently sufferers from traumatic brain injuries may be undiagnosed until much later. The 

Army seeks a biological marker such that a simple blood test would reveal brain damage long 

before there would be visible symptoms; this would enable early diagnosis and treatment. It 

should be noted that there is, in the medical research program, a large amount of research that is 

not directly connected to the needs of the warfighter. A notable example is the DOD research on 

breast cancer, a program managed by the Army Medical Command’s Medical Research and 

Materiel Command (MRMC). Finally, the Army is funding academic consortia to develop organ 

regeneration science and technology. The ultimate aim is to help soldiers suffering amputations. 

A related technology is the control of prostheses by electronic connections to the nervous 

systems at appropriate levels. Some success has already been achieved with the Defense 

Advanced Research Projects Agency leading the way. The Army medical research program is 

closely linked to non-military medical research, including pharmaceutical, and depends on the 

community for various efforts required to achieve Food and Drug Administration approvals. 

Collaboration and cooperation is the rule. 

Enduring Technologies Portfolio. The work on environmental quality at installations, including 

clean-up on ranges and at former dumps, is important for compliance with environmental rules 

and regulation. However Army research is relatively minor. Pollution is an issue that society in 

general shares. Disposal of chemical weapons has been and still is of major concern. Much of the 

research done in this area is engineering work, with analytical methods the subject of some 

study. These analytical methods are needed to ensure that not even the least amount of toxins is 

released during disposal. 

Much of the Army’s physics-based modeling and simulation work is done on high performance 

computers. A principle source of such computing is provided by the High Performance 

Computing Modernization Program, now managed by the Corps of Engineers at Vicksburg. 

They manage centers of excellence across DOD. This does not involve research on developing 

8 
better supercomputers; that is left to industry and academe. The centers provide the means to 

work on problems that are intractable on less capable computers. There is also very good work 

on network science for the battlefield that relates to work in the private sector and in academe. 

Basic Research. We consider here three categories of basic research: in-house research, and two 

types of extramural research. The in-house research, conducted by Army technologists, may be 

long-term exploratory work pushing the frontiers, or relatively short-term study to elucidate 

phenomena raising barriers to progress on current projects. In neither case does the DOD 

definition of 6.1 research apply. Namely, the work is curiosity work with no specific application 

in mind. The two extramural categories include single-investigator grants, typically to 

universities, and sponsorship of large centers of excellence. These latter include, under the rubric 

of University Affiliated Research Consortia, two sets of programs called Collaborative 

Technology Alliances and Collaborative Research Alliances established by ARL and designed to 

bring together the external consortia members with in-house Army technical staff. (External 

centers of excellence are not so closely related to Army laboratories.) 

ARL is the leader in Army basic research both in in-house work and, since the Army Research 

Office (ARO) is part of ARL, in extramural research. In recent years the amount of in-house 

basic research at ARL has been reduced by transferring 6.1 funds to the Collaborative 

Technology Alliances. 

A lengthy study of basic research in the Army by the Office of the Deputy Assistant Secretary of 

the Army for Research and Technology is currently underway looking at the portfolio. In 

particular, the study is looking at the selection of basic research areas and the scenarios and 

technology gaps derived from the analyses of TRADOC. This study is focused on the grants 

made by ARO and will lead to a more formal articulation of the Office’s philosophy and 

management approach. This study and a companion study of the formulation of the Army’s 

applied research program will produce some revamping of the procedures and programs to make 

them more understandable to the warfighters and the Army’s senior leadership.4

The Basic Research Portfolio. The basic research portfolio in the current analysis is in four 

component parts: platform centric; material centric; information centric; and human centric. 

ARO sponsors research that is intended to push the frontiers of already state-of-the-art science 

and technologies, and to identify and pursue new areas of research that may be useful to the 

Army in the future. Projects in the sciences include study of new concepts in physics, chemical 

sciences, and materials, each covering a number of subareas that are at the frontiers of 

knowledge. The current summary of the portfolio supplied by the Office of the Deputy Assistant 

Secretary of the Army for Research and Technology is arranged by major topics, not by 

organization. Thus there is no mention of ARO’s basic research in biotechnology, neuroscience, 

and other areas in life science now being pressed in the scientific community at large. A more 

detailed study would analyze these areas as well. 

 

4

 Dr. Marilyn Freeman, The United States Army’s Science and Technology (S&T) Program for Fiscal Year 2013,

Statement to the U.S. House of Representatives, Committee on Armed Services, Subcommittee on Emerging 

Threats and Capabilities, Second Session, 112th Congress, February 29, 2012, available at 

<http://defenseinnovationmarketplace.mil/resources/Freeman_Testimony_2013.pdf >. 

9 
ARO specifically excludes certain sub-branches of areas, although it does ensure some room 

remains available for unusual opportunities that could be useful for the Army: 

The Physics Division has little direct interest in relativity and gravitation, cosmology, 

elementary particles, nuclear physics, astronomy, or astrophysics, since they generally have 

little impact on the areas of Army needs. Nevertheless, the possible relevance of topics 

within these other physics disciplines is not absolutely discounted, and discussions of 

potential exceptions are welcome.5

 

An inspection of the science topics ARO is interested in reveals a broad range of research areas, 

each of which is related to some aspect of the Army’s mission. ARO’s Physics Division states: 

Condensed Matter Physics (CMP) is a foundational science enabling fundamental Army 

technologies in areas such as information processing, communications, sensors, optical 

components, electronics, optoelectronics, night vision, seekers, countermeasures, and many 

others. Technologies such as these would not exist today, at least not as we know them, 

without visionary research in the field of CMP. The ARO CMP program strives to continue 

this level of impact by looking beyond the current understanding of natural and designed 

condensed matter, to lay a foundation for revolutionary technology development for next 

generation and future generations of warfighters.6

The MRMC performs or sponsors basic research in many different areas. Recently it has 

established, with funding support from a number of other federal agencies, two research 

consortia in a number of universities on tissue and organ regeneration. Each of these civilian 

consortia is itself a multi-institutional network.7

 MRMC conducts programs on traumatic brain 

injury, blood loss control, pain management, and control of and immunization against a variety 

of infectious diseases that are hazards to soldiers. These last named programs are generally 

carried out cooperatively with the pharmaceutical industry and the Food and Drug 

Administration. MRMC has a long and notable history in developing and demonstrating 

techniques for combat casualty care on the battlefield and for subsequent long-term medical care 

and rehabilitation for wounded and traumatized soldiers. 

The Corps of Engineers Engineering Research and Development Center has a number of basic 

research programs—many in environmental areas such as remote sensing of terrain—and it 

conducts modeling and simulation studies on its High Performance Computing Shared Resource 

Facility. The Engineering Research and Development Center also manages DOD’s High 

Performance Computing Modernization Office. A group within this Office manages a program 

called CREATE that is “developing and deploying scalable, multiphysics-based computational 

 

5

 Army Research Laboratory, Army Research Laboratory Broad Agency Announcement for Basic and Applied 

Scientific Research, W911NF-12-R-0011, 15 May 2012 – 31 March 2017, 93, available at 

<www.arl.army.mil/www/pages/8/research/Final_Post_ARL_BAA_W911NF-12-R-0011.pdf>. 6

 Ibid., 93. 

7 Armed Forces Institute of Regenerative Medicine, homepage, available at 

<https://mrmc.amedd.army.mil/index.cfm?pageid=medical_r_and_d.afirm.overview>. One consortia is led by the 

Wake Forest University Baptist Medical Center along with the McGowan Institute for Regenerative Medicine at 

Pittsburgh; the other is led by Rutgers University, the State University of New Jersey, along with the Cleveland 

Clinic. 

10 
engineering tools to design and analyze DOD weapons system performance.”8

 This is 

exploratory work pushing the frontier in a new area for DOD computing, namely, becoming 

involved directly in the engineering and manufacturing end of the acquisition timeline. The use 

of high performance computing in this way in the private sector is shown to shorten the time to 

complete new product development by as much as a factor of three.9 This result is made possible 

by taking advantage of the power of these machines to model complex systems that have been 

beyond the reach of less powerful computers. 

Portfolio Summary. The S&T portfolio is the result of many studies and decisions over many 

years. It has been revised and shaped by the changing needs of the warfighter, as well as 

continuing discussions between the S&T community and the warfighters as represented by 

TRADOC at policy levels. It also maintains contacts between members of the Army laboratory 

system and soldiers at home and abroad. TRADOC regularly issues the top Army challenges. 

The S&T program planners update their priorities accordingly and add new programs (and 

terminate older, lower priority work) to maintain alignment of the S&T programs with doctrinal 

priorities. For short-term work, the alignment must be very close. Programs in advanced 

development are tailored to respond to a list of 24 Army S&T Challenges.10 Somewhat longerrange work in applied research anticipates the evolution of the problems. Long-term, basic 

research (6.1) seeks to uncover new opportunities at the frontiers of scientific and engineering 

knowledge. The Army has established centers of excellence in basic research and early applied 

work based on present needs, but with the flexibility to pursue new leads. 

The following is a list of areas which, in our judgment, occupy a special niche in Army 

technology and therefore should receive priority support: 

• Soldier and vehicle armor and armaments 

• Night vision equipment 

• Unattended ground sensors and sensor networks 

• Air acoustic, seismic, and magnetic sensors 

• Terrain and topographical analysis and navigation 

• Soldier power and thermal management of platforms 

• Helicopter protection and lethality 

• Individual soldier health sensing and monitoring 

• Individual soldier robotic associates 

• Mobile ground communications networks 

• Mine/IED detection and clearing at operational tempo 

• Tactical chemical/biological/nuclear detection and protection 

• Ground combat survivability/vulnerability analysis. 

 

8

 John W. Lyons, Richard Chait, Charles J. Nietubicz, The Use of High Performance Computing (HPC) to 

Strengthen the Development of Army Systems, Defense & Technology Paper 87 (Washington, DC: Center for 

Technology and National Security Policy, November 2011). 

9

 Loren Miller, quoted in Council on Competitiveness, Goodyear Puts the Rubber on the Road with High 

Performance Computing, Case study, 2009, 4, Available at 

<www.compete.org/images/uploads/File/PDF%20Files/HPC_Goodyear_032009.pdf>. 10 See Army Science & Technology problems and Challenges, PowerPoint, undated, available at 

<http://usarmy.vo.llnwd.net/e2/c/downloads/239849.pdf>. 

11 
IV. EVALUATING THE QUALITY OF THE PROGRAM 

AND ITS EXECUTION 

The second part of the Army S&T program review by the Deputy Assistant Secretary of the 

Army for Research and Technology is seeking answers to the following questions: 

• How does one assess the execution of the program with existing research staff and 

infrastructure? 

• Do we have the best personnel and infrastructure compared to leading research and 

development programs? 

In this section of the paper we develop a methodology for making these evaluations. We begin 

with a discussion of the factors that are necessary to achieve excellence in a research laboratory. 

Next is a review of the practice of expert review of the laboratory and its work by unbiased 

subject matter experts. We then consider the relevance of the work and the value of continued 

contact with customer and ultimate users of the results from the laboratories. We consider the 

need for and value of formal and informal collaborations with other laboratories. All of this 

discussion leads to the role of excellence in a laboratory to its standing in the community and to 

the support the laboratory has in the senior leadership in the Army and the authorizing and 

appropriating committees in the Congress. 

The quality of a program of research and development depends on many factors, but the most 

significant are the quality of the research staff and of the management. Close behind are the 

quality of the facilities and capital equipment. According to a 1991 report to the Office of the 

Secretary of Defense, an effective laboratory will exhibit the following characteristics: 

• A clear and substantive mission 

• A critical mass of assigned work 

• A highly competent and motivated work force 

• An inspired, empowered, highly qualified leadership 

• State-of-the-art facilities and equipment 

• An effective two-way relationship with customers 

• A strong foundation in research 

• Management authority and flexibility 

• A strong linkage to universities, industry, and other government laboratories.11

These statements however do not represent the whole picture. The laboratory program must be 

aligned with the vision and mission of the parent organization; in this case the U.S. Army. In the 

previous section we discussed the evaluation of the portfolio of the S&T program and pointed 

out that it is aligned with the needs of the warfighter through continuing discussions with the 

soldiers and their representatives. The quality of the S&T product depends on the relevance of 

the work to the needs not only of the soldiers but also of the initial customers of the work. For 

example the Army Materiel Command’s corporate laboratory—ARL—works on problems 

defined by Army TRADOC, but delivers its results to intermediaries such as the Research, 

Development and Engineering Command and the Program Executive Officers and Program 

 

11 Charles E. Adolph, chairman, Federal Advisory Commission on Consolidation and Conversion of Defense 

Research and Development Laboratories, Report to the Secretary of Defense (Washington, DC: Department of 

Defense, September 1991). 

12 
Managers of the acquisition community. They can judge the immediate impact on their work. 

The ultimate impact, also a measure of quality, is harder to evaluate because there are many 

hurdles before the final product or process is fielded. Even when fielded, there may be many 

years of experience needed before conclusive appraisals can be made. Since the managers of 

S&T can’t wait for years to discover how well they have done, they need some way of evaluating 

the work in the short term. 

The evaluation of the execution of the program covers three phases: the planning stage, current 

laboratory work, and completed work. Planning involves evaluating proposals for new work and 

assessing them for relevance; the availability of qualified research staff including the project 

leader; the match to facilities and equipment on hand; the funding necessary to acquire additional 

resources if needed; and whether or not the challenge of the problem posed is such that it can be 

solved in a time frame that is practical for the warfighter. One measure of quality of the program 

is the extent to which the laboratory or laboratories are prepared for future demands; 

preparedness is a quality that requires foresight (and technology forecasting) and flexibility. The 

planning phase of the Army S&T program is conducted jointly with TRADOC and the S&T’s 

customers for the 6.2 and 6.3 parts of the program. Recently TRADOC has become involved in 

the 6.1 program in terms of the coverage of the areas in the list of 24 Army priorities for S&T 

prepared by the Deputy Assistant Secretary of the Army for Research and Development in 

concert with TRADOC. Much of the planning exercise has been discussed in Section II on the 

portfolio. 

Peer Review. Evaluation of the quality of current research and development work is somewhat 

more straightforward than evaluating the portfolio in that peer review is the prime evaluating 

tool. Currently, there is an established technique for peer review of laboratories which can aid in 

ensuring quality in research laboratories.12 There are many variations—different laboratories 

may require different approaches. If the peers are drawn from within the laboratory or its parent, 

there will be good knowledge of the technical area and the challenges being addressed. However, 

internal judgments may be biased based on the stakes held by the evaluators. On the other hand, 

external evaluators will have less intimate knowledge of the specifics of the work but their 

evaluation will have more credibility. For most assessments, credibility is the leading 

characteristic as the results will help local management improve their work or even change 

course. For government laboratories, peer review processes will help in justifying budget 

documents. Conclusions from various studies of peer review of laboratories indicate that the best 

way to do this is as follows: 

• Contract the evaluation to a recognized and respected entity such as the National 

Research Council of the National Academies. 

• Have the contractor manage the entire process from picking evaluators, planning 

agendas, handling meeting logistics, and supervising preparation and distribution of 

reports. 

• Appoint panels of evaluators that consist of independent subject matter experts with 

knowledge of the areas of interest. 

 

12 National Research Council, Panel for Review of Best Practices in Assessment of Research and Development 

Organizations, Best Practices in Assessment of Research and Development Organizations (Washington, DC: 

National Academies Press, 2012). 

13 
• Have panels evaluate not only the technical work presented to them, but also the quality 

of the technical staff, the quality of the facilities and equipment, and the adequacy of the 

resources provided. 

• Seek to evaluate the culture of the laboratory and the role of upper management in hiring, 

promoting, and motivating and retaining the staff. 

• Hold out-briefing sessions with the staff, prepare written reports, and indicate the panel’s 

impressions of the follow-up actions by the laboratory on recommendations from the 

previous report. 

Most Army laboratories carry out some form of peer review, but not all. The evaluations run the 

gamut from using contractors as described above to conducting strictly in-house assessments. It 

would be useful for all Army laboratories to consider some kind of independent peer review of 

current work. One model is the review of ARL. This is done by the National Research Council 

and follows the above recommended course of action. ARL reviews began in the 1990s and have 

been done regularly since. The evaluation reports are published by the National Research 

Council and are available on the Web site of the National Academies Press. 

Completed Work: Metrics and Benchmarking. Some aspects of a laboratory can be assessed 

quantitatively and the results can be compared to those at other laboratories, preferably 

laboratories that are widely regarded as outstanding. Metrics may include a variety of topics: 

staff educational level, staff age distribution, number of archival publications per technical staff 

member, number of patents, number of external awards and other recognitions, elected 

memberships to honorary societies such as the National Academies, and so on.13 Such metrics 

are useful as management tools, but caution must be taken in using them. These measures do not, 

of themselves, indicate the quality of the entries. As an example, a laboratory can have a number 

of unused patents and a string of uninspiring publications. In this situation, this laboratory would 

have met its criteria for metrics, but be performing poorly. In comparison, a laboratory could 

have written brief papers that have won Nobel prizes.14 The quality of completed work as 

evidenced in publications, patents, and various prototypes for products can be assessed by 

sampling done by outside peers. This will ensure a mix of metrics and qualitative assessments, 

and provide a far greater and far more accurate representation of effectiveness. 

Simultaneously, the Army can compare itself with other military laboratories to give a more 

meaningful assessment. The Naval Research Laboratory would be a prime example for such a 

comparison, provided the assessment used similar metrics. Other possibilities for comparative 

analysis include the nature of the portfolio, such as percentages of basic and applied research, the 

amount of work in which the laboratory is a world leader in the technology, and the focus on 

short-term vs. long-term work. 

We recently have provided an informal report to the Deputy Assistant Secretary of the Army for 

Research and Technology in which we listed eight factors important for evaluation of quality of 

the S&T program. The factors include quality of staff; quality managers; peer review; relevance 

(inputs from customers and stakeholders); technology forecasting; leveraging; scientific or 

professional positions and the Independent Research & Development (IR&D) conducted by 

 

13 Report of an Expert Panel on the Future of Army Laboratories, Improving Army Basic Research (Arlington, VA: 

RAND 2012), 50–51. This report lists 39 metrics for evaluating basic research. 14 James D. Watson and Francis H. Crick, “A Structure for Deoxyribose Nucleic Acid,” Nature, vol. 171 (1953), 37-

38. 

14 
DOD contractors; and visibility of Army S&T. We compared our list against the list of metrics 

taken from a recent report from the RAND Arroyo Center.15 The result is shown in Appendix A. 

The published list of metrics amplifies our seven fundamentals. There is no contradiction in the 

two lists. 

Impact and Relevance—Relations with Customers and Stakeholders. A quality laboratory will 

know its customers and be close to its stakeholders to ensure it fulfills their expectations for 

relevance and timeliness. The laboratory should address the needs in the priority in which they 

are presented by the customers. As noted earlier, the Army laboratories deliver their research 

findings to a well-defined set of customers: namely, the acquisition community and the 

warfighter’s representatives. An assessment of a laboratory should include discussions held with 

samples of these communities. One way to determine how well the laboratory is serving its 

customers is simply to invite representatives to research reviews and listen to their comments. 

Laboratories, which detail a portion of their staff to acquisition offices under matrix 

management, hold regular meetings with those Program Executive Officers or Program 

Managers to review progress. One possible technique is to poll customers via a ratings sheet 

followed by serious discussion of any area not given a fully successful grade. 

As noted in the introduction, the stakeholders are the senior leaders of the Army, DOD, and the 

taxpayers as represented by the U.S. Congress. The opinions of senior Army and DOD leaders 

are usually made known by them to the Office of the Assistant Secretary of the Army for 

Acquisition, Logistics, and Technology (ASAALT). To enhance relations with stakeholders, 

meetings and laboratory tours can be effective, but stakeholder schedules make visits very hard 

to arrange. The ASAALT and sometimes senior assistants to the principals can be persuaded to 

make visits and discuss issues. Congressional hearings are too formal and too brief for effective 

communication. However, congressional staffers are often interested in the details and enjoy 

making laboratory visits. 

Taking Advantage of External Expertise. No laboratory can be at the frontiers of knowledge in 

every subject for which it is responsible. Therefore, it behooves the laboratory to seek out 

experts to learn what they know, to cooperate where feasible, and to enter into formal 

collaborations where justified. There is no place for insularity in today’s technological world, 

save only for classified work. Assessors of a laboratory’s work should ask the staff what other 

experts are working in this field. Who are the leaders? How does this laboratory compare? If the 

laboratory is not among the world’s elite in a particular field—not working at the state of the 

art—then it should be in active contact with those who are. There should be cooperative 

programs and perhaps formal collaborations. If not, then the laboratory should exit the field. It is 

a measure of the quality of the laboratory management as to whether it is addressing these 

questions. Use of comparative metrics, noted above, can be helpful in benchmarking the work 

against others. 

The quality of the preparedness of the laboratory for future demands is helped by a regular 

program of S&T forecasting. The laboratory should be looking ahead and comparing what it 

forecasts against what the demands on it are likely to be. Forecasting can be done in different 

ways. At the individual or small group level, one expects the local experts to have a good sense 

of where their disciplines are heading. At higher levels, such as the laboratory or its parent, 

 

15 See Report of an Expert Panel on the Future of Army Laboratories, Improving Army Basic Research (Arlington, 

VA: RAND, 2012). 

15 
forecasting is done in an organized fashion, perhaps on contract. Forecasts of individual sciences 

or technologies should be combined with cross-disciplinary and multi-disciplinary predictions. 

One concept is convergence forecasting in which mixed panels of experts in several potentially 

related disciplines work together to make predictions and suggest specific research programs.16

Forecasting should be a recurring activity in a rolling manner such that any given area is looked 

at every 5 to 10 years. The Deputy Assistant Secretary of the Army for Research and Technology 

sponsored a very detailed forecast of all areas of interest 20 years ago.17 Resuming forecasting is 

long overdue. Lack of a forecasting program is a factor in judging the overall quality of the S&T 

effort. 

Quality of S&T and Its Reputation. The reputation of an S&T program governs, to a large 

extent, its ability to secure resources for existing and new programs. The most important 

audience is therefore the parent of the program and other stakeholders with a financial interest. 

Their impression of the laboratory is formed in part by the technical results, but also by 

comments from outside the enterprise from customers and other interested people. For the 

laboratories, in particular, external opinion has been a problem. Reputation is formed by the 

impressions given by publications, presentations, external awards and other public recognition, 

and press coverage. Nothing is as impressive as the receipt of a Nobel Prize. The Naval Research 

Laboratory has one; the National Institute of Standards and Technology has four, and the 

National Institutes of Health have received five. The Army has none. However, over the years 

ARO has provided support for research that produced 18 Nobel prizes. The Army does not have 

an individual member elected to one of the National Academies. In the recent past the Army has 

had one. These omissions are at least, in part, because the effort has not been made to publicize 

the technical work and to nominate staff members for prizes and awards. The Army has not done 

the public relations work necessary to keep the laboratories’ work in the public eye. Compare 

this with the Naval Research Laboratory and its reputation, or Bell Laboratories, which 

continually educates the public on who invented the transistor. Some laboratories cooperate with 

authors writing books about their work and their history. Jon Gertner’s recent best-selling book 

on the history of the Bell Laboratories has received rave reviews and is an example of this 

cooperation.18 Could similar public relations efforts be carried out for Army S&T? In fact, it was 

attempted on the 50th anniversary of the introduction of the first general purpose electronic 

computer, ENIAC (Electronic Numerical Integrator and Computer). Creation of the ENIAC was 

funded by the Army’s ballistics program at Aberdeen Proving Ground at the end of World War 

II. 

Because of this lack of aggressive publicity, the reputation of Army S&T is not what it could be. 

This has led to lukewarm support for the Army’s S&T budget, lukewarm even on the part of 

some senior Army leaders. At best, this problem produces more support for Army contracts and 

 

16 John Lyons, Richard Chait, and Simone Erchov, ed., Improving the Army’s Next Effort in Technology 

Forecasting, Defense & Technology Paper 73 (Washington, DC: Center for Technology and National Security 

Policy, September 2010); Douglas Kiserow et. al., Report of an Army Workshop on Convergence Forecasting, 

Mechanochemical Transduction, Defense & Technology Paper 95 (Washington, DC: Center for Technology and 

National Security Policy, July 2012). 

17 National Research Council, STAR 21–Strategic Technologies for the Army of the Twenty-First Century

(Washington, DC: National Academies Press, 1992). 

18 Jon Gertner, The Idea Factory: Bell Labs and the Great Age of American Innovation (New York: Penguin Press, 

2012). 

16 
grants than for in-house work. Sometimes this is appropriate, sometimes it is not. The Army 

should not hide its S&T quality. Instead, it should be its biggest promoter and justifier. 

Quality Summary. This discussion is focused on the laboratory on the assumption that the 

quality of the S&T program as a whole would be, to a large extent, the sum of the individual 

parts. We have considered three aspects of laboratory quality: the planning phase, the current 

work in the execution of the plans, and the completed technical work. Planning includes the 

alignment of the plans with the mission of the laboratory and of the Army. A quality plan 

includes the availability of quality staff (and managers); availability of needed resources 

including budget, facilities, and equipment; the allocation of resources to a balance of short- and 

long-term research; and a regular program of technology forecasting to ensure preparedness of 

the laboratory for future developments in science and engineering and evolving trends in threats 

to the nation. 

Some form of peer review should be used to assess current laboratory work. We believe the best 

approach is to contract for panels of independent, unbiased subject matter experts. These panels 

should be asked not only to evaluate the current research and development as presented by the 

staff doing the work, but should also be asked to review the status of the facilities and 

equipment. In the course of talking with the staff and walking through the laboratories and 

meeting with management, the panels can get an idea of its culture and the nature of the 

management personnel. Such panel reviews should be done for each sector of the laboratory 

every 2 to 3 years. The degree to which the research staff interacts with and is cognizant of the 

work of others in the field is an important factor in rating the laboratory. Various kinds of 

cooperation and formal collaborations with external expertise are a way to make sure that the 

laboratory is at the state-of-the-art level and is pushing the frontiers. Working with others is a 

good way to be prepared for future developments. 

Completed work produces impacts on the Army. Results from the laboratory reach the warfighter 

through the efforts of many players including the technical work in the offices of Program 

Executive Officers and Program Managers in the acquisition community. Results may be 

transmitted directly to industrial firms that are preparing for manufacturing of the new products 

or using the new processes from Army research. During war time, some results obtained in the 

laboratory are transmitted though accelerated procedures directly to soldiers on the battlefield. 

Laboratory findings are summarized in publications and presentations and often result in awards 

and other forms of recognition. Satisfaction with the laboratories’ work in terms of relevance, 

quality, and timeliness should be tested by regular sessions with customers and stakeholders. 

Comparisons of the laboratory with other laboratories deemed to be outstanding can be made via 

benchmarking. In this approach various metrics can be obtained and comparisons made. These 

would include external recognition as well as items that can be counted, such as numbers of 

papers and patents, percentages of Ph.D.s, and so on. Reputation is important for a number of 

reasons, but the most fundamental is the impact a strong reputation will have on the ability to 

secure resources from year to year. 

Questions to be asked when assessing the quality of a research and development organization 

should include: 

• Is the work aligned with the mission and priorities? 

• What is the level of education of the staff? (percentage of Ph.D.s) 

• What is the average age of staff? (retention rate) 

17 
• Do the managers have experience at the laboratory bench? What is their reputation? How 

do they affect the culture? 

• What is the nature of the basic or exploratory work? How much? 

• How closely does the staff work with experts in other laboratories? 

• What is the nature of cooperative work with outside laboratories? Does the laboratory 

have formal collaborations? 

• What is the quality of the laboratory’s products? (number of transitions of results to 

customers) 

• For each technical staff member, what is the number of publications, internal reports, 

patents, and presentations? 

• How is the staff recognized? (awards, elected memberships) 

• What has been the impact of the program on warfighting? (retrospective studies) 

• What is the state of the facilities? 

• What is the state of capital equipment? 

• What are the results of benchmarking against peer laboratories? 

18 
19 

V. SUMMARY OF REPORT AND RECOMMENDATIONS 

In this paper we focused on the existing portfolio as provided by the ASAALT Office. Our 

object was to provide a methodology for evaluating only this existing program. However, to 

make a complete assessment one would have to consider what topics are either not being 

addressed or not being addressed adequately. The answers would assist in reorienting the 

portfolio. It would have been of great assistance if the Army had a current version of the Army 

Science & Technology Master Plan. 

 We classified the work of the Army S&T program in three aspects: work where the Army is 

dominant in the technology, work where the Army shares the technical lead with the private 

sector, and areas where the private sector leads. The priority and nature of the research program 

varies with the category. The paper discusses some details of evaluating the portfolio and 

presents a list of questions, the answers to which would help. The evaluation of the current 

research work in executing the portfolio covers three phases: planning, execution, and completed 

work. This paper provides another list of questions for assessing the quality of the current work. 

We discuss the quality of the staff, how to perform peer review, consider relations with outside 

experts, and ask about cooperation and collaborations with others. The paper provides an 

approach to reviewing and evaluating the quality of the portfolio itself and the technical 

execution of the programs and projects. 

We recommend: 

• The Army resume publication of the Army Science & Technology Master Plan 

• The Army evaluate the current S&T portfolio along the lines suggested herein. 

• The review should identify areas that are underfunded (or overfunded) and suggest 

corrections. 

• The review should attempt to identify areas of science and technology not included in the 

current portfolio but which should be included. 

• The evaluation should make sure that every program is directly in support of the Army’s 

mission and vision. 
VI. APPENDIX 

Appendix A includes eight fundamentals taken from a draft report by CTNSP19 in a matrix with 

a series of metrics. The X’s indicate the relevant items in the rows that match the elements in the 

column. 

S&T Quality Fundamentals 

Metrics 

Quality 

Staff 

Quality 

Mgrs 

Peer 

review Relevance 

Tech 

Fcstg. Leverage 

STs & 

IR&D Visibility 

Distribution by 

S&T degrees, 

age distribution, 

and field of 

research 

X X 

Fellowship in 

professional 

societies 

X X 

Membership in 

national 

academies or 

similar bodies 

X X 

Other 

professional 

recognition 

(prizes and 

awards) 

X X 

Breakdown by 

advanced 

degrees and 

scientific and 

engineering 

disciplines 

X 

Refresh and 

turnover rate of 

staff 

X X 

Invited and 

keynote 

speeches 

X X X 

Vision: 

commonality of 

goals 

X X 

Funding level X X X 

Consistency of 

funding levels 

X X 

Scientific 

reputation of 

research 

managers 

X X 

 

19 Private communication to the Deputy Assistant Secretary of the Army for Research and Technology, June 6,

2012. 

A-1 
S&T Quality Fundamentals 

Metrics 

Quality 

Staff 

Quality 

Mgrs 

Peer 

review Relevance 

Tech 

Fcstg. 

STs & 

Leverage IR&D Visibility 

Number of 

significant 

breakthroughs 

and inventions in 

the past 10 years 

X X 

Recruitment of 

high quality 

researchers 

X X X 

Relevance for 

the enterprise 

X X 

Rainmaker: 

ability to 

influence 

sponsors/funders 

X X X 

Selection 

process: 

subfields to 

invest and 

proposals. 

Support for 

external review 

of proposals and 

research 

X X X X 

Success in 

helping 

technology 

transfer 

X X X 

Portfolio view of 

research 

activities 

(balance 

between 

enhancement, 

extensions, new 

challenges in 

existing areas, 

new emerging 

areas) 

X X X 

Criteria and 

assessment 

process for 

researchers and 

their career 

advancement 

X X X 

Emphasis on 

peer-reviewed 

competition for 

basic research 

funding 

X X 

A-2 
A-3 

S&T Quality Fundamentals 

Metrics 

Quality 

Staff 

Quality 

Mgrs 

Peer 

review Relevance 

Tech 

Fcstg. Leverage 

STs & 

IR&D Visibility 

Specification of 

research 

process, score 

cards 

X X X X 

Administrative 

burden on 

researchers 

X 

Mentoring of new 

employees 

X X 

Operational 

management 

X X X X 

Intellectual 

stimulation 

X X X X X 

Unstructured 

activities 

X X X 

Interactions with 

customers 

X X X X 

Support for new 

ideas 

X X X 

Lab facilities; 

computing 

environment 

X X 

Publications X X X 

Citation by 

others 

X X X 

Peer recognition X X X 

Patents X X X X 

Citation of 

patents by others 

X X X X 

Tech transfer to 

field 

X X X X 

Long-term 

impact on users 

and funders 

X X