Risk-Informed Decisionmaking 

for Science and Technology 

Samuel Musa, William Berry, Richard Chait, John Lyons, and Vincent Russo

Center for Technology and National Security Policy 

National Defense University 

September 2010 
ii

The views expressed in this article are those of the authors and do not reflect the official policy or 

position of the National Defense University, the Department of Defense, or the U.S. Government. 

All information and sources for this paper were drawn from unclassified materials. 

________________________________________________________ 

Samuel Musa is the Homeland Security Science and Technology Chair and Senior Research 

Fellow at the Center for Technology and national Security policy (CTNSP). He was previously 

Associate Vice President for Strategic Initiatives and Professor of Electrical and computer 

Engineering at Northwestern University. Dr. Musa received his PhD in applied Physics from 

Harvard University and a BS in Electrical Engineering and a BA from Rutgers University. 

William Berry is Deputy Director for Science and Technology and Distinguished Research 

Fellow at CTNSP. He was previously Acting Deputy Under Secretary of Defense for 

Laboratories and Basic Sciences. Dr. Berry received his PhD in Zoology from University 

of Vermont and BS in Biology from Lock Haven University.

Richard Chait is Distinguished Research Fellow at CTNSP. He was previously Chief Scientist, 

Army material Command, and Director, Army Research and laboratory Management. Dr. Chait 

received his PhD in Solid State Science from Syracuse University and a BS degree from 

Rensselaer Polytechnic Institute. 

John Lyons is a Distinguished Research Fellow at CTNSP. He was previously Director of the 

Army Research Laboratory and Director of the National Institute of Standards and technology. 

Dr. Lyons received his PhD from Washington University and a BA degree from Harvard. 

Vincent Russo is former Executive Director of the U.S. Air Force Aeronautical Systems 

Center. Dr. Russo received his PhD in Metallurgical Engineering from Metallurgical 

Engineering from the Ohio State University and BS from University of Rochester. 

Acknowledgments 

The authors gratefully acknowledge the support of Starnes Walker, Director of Research, 

and Rolf Dietrich, Deputy Director of Research, and Capt. David Newton, Office of 

University Programs, Science and Technology Directorate, Department of Homeland 

Security. Samuel Bendett, Research Associate at CTNSP, assisted the team and 

conducted a literature search as outlined in the paper. 

Defense & Technology Papers are published by the National Defense University Center for Technology 

and National Security Policy, Fort Lesley J. McNair, Washington, DC. CTNSP publications are 

available at http://www.ndu.edu/ctnsp/publications.html. 
iii

Contents 

Introduction......................................................................................................................... 1 

Methodologies for Risk-Informed Decisionmaking ........................................................... 3 

Discussion–Risk Areas and Impact in Decisionmaking ..................................................... 6 

Conceptualizing Metrics ..................................................................................................... 8 

Decisionmaking Process for Program Evaluation .............................................................. 9 

Decisionmaking and Program Management..................................................................... 10 

Conclusions....................................................................................................................... 11 

Appendix. Performance Evaluation and Metrics for Program Selection, Army Research 

Laboratory......................................................................................................................... 12 
iv
1

Introduction 

To properly understand science and technology (S&T) leadership and decisionmaking 

in government laboratories, the Center for Technology and National Security Policy 

Homeland Security team conducted a study on practical approaches to S&T riskinformed decisionmaking and metrics for program selection. The study was conducted 

for the Director of Research of the Science and Technology Directorate at the 

Department of Homeland Security (DHS). The results are applicable to other S&T 

organizations. 

The S&T Directorate is the primary research and development arm of DHS. The 

directorate's mission is to improve homeland security by providing customers state-ofthe-art technology that helps them achieve their missions. S&T customers include the 

operating components of DHS, state, local, tribal, and territorial emergency responders 

and officials. The Directorate has three portfolios (research, innovation and transition) 

that address basic research through advanced technology development and transition - 

spanning six primary divisions (Borders and Maritime Security; Chemical and 

Biological; Command, Control and Interoperability; Explosives; Human Factors 

Behavioral Sciences; Infrastructure and Geophysical) that address critical homeland 

security needs. 

The Director of Research is one of three functional portfolio managers reporting to 

the Undersecretary for Science and Technology. The others are the Director of 

Innovation and the Director of Transition. These portfolio managers provide consistency 

in their respective areas of responsibility across the spectrum of the DHS S&T 

investment. Currently, four DHS laboratories report to the Director of Research, while 

two report to the Director of Test & Evaluation and Standards. There are six divisions 

that also report to the Office of the Undersecretary, where the majority of the 

programmatic resourcing and execution originates. Most of the DHS laboratory research 

budget funnels through one of the six divisions within S&T. DHS internal management 

and regulation continue to evolve as the department meets new national security 

challenges that demand solutions backed by scientific and technological approaches. 

The goals of this study are: first, to apply leadership lessons learned from the 

Department of Defense laboratory management experience to the needs of DHS; second, 

to recommend best practices for leadership in the S&T Directorate; and third, to respond 

to DHS-requested areas of evaluation.1

 This paper synthesizes the results of this study. 

Initially, the CTNSP team concentrated on the need for practical approaches that the DHS 

leadership could use to make critical risk-informed decisions and developed metrics for 

the selection of the most beneficial programs and projects. 

The subject of risk-informed decisionmaking has received considerable attention in 

the literature due to its broad application to a number of disciplines. It is basically “a 

process that organizes information about the possibility for one or more unwanted 

 

1 April 7, 2009 meeting between CTNSP Homeland Security team and Department of Homeland Security 

officials.
2

outcomes into a broad, orderly structure that helps decisionmakers make more informed 

management choices”.2 The major components of the process are3: 

1. Establish the decision structure 

2. Perform the risk assessment 

3. Apply the results to risk management decisionmaking 

4. Monitor effectiveness through impact assessment 

5. Facilitate risk communication 

The co-authors of this paper formed the team which conducted this study. Brief 

biographies of each member are at the front of this paper. The team brought to the table 

decades of combined experience in managing civilian and military laboratories and 

research centers. 

This paper discusses risk and impact areas in relation to decisionmaking and the 

development of metrics or a figure of merit for decisionmaking. The metrics are then 

applied to three examples of interest to the Army, Air Force, and DHS. The decision 

process utilizing the figure of merit is described for S&T program selection and funding. 

Various styles of decisionmaking used for the management of DOD laboratories is briefly 

described and the differences between management and leadership are highlighted. 

 

2

 Principles of Risk-Based Decision Making, ABS Consulting, Government Institutes, Rockville, 

Maryland, 2001. 

3

 Ibid 
3

Methodologies for Risk-Informed Decisionmaking 

Several methodologies for risk-informed decisionmaking were examined for the 

purpose of identifying metrics for program evaluation and selection. A summary of these 

methodologies is highlighted here and in Appendix A. The team performed research and 

analysis of relevant material dealing with risk-informed decisions in scientific institutions 

and organizations working with issues related to DHS. This material included books and 

periodicals about managing scientific institutions; newspaper and magazine articles about 

best practices in managing scientific organizations and making decisions in governmentfunding institutions; and analyses of best practices in managing government-funded 

scientific laboratories over the past several decades. Additionally, the team reviewed 

available information on privately run companies and institutions working on scientific 

and technological research. Appendix A outlines the performance evaluation and metrics 

for program selection by the Army Research Laboratory. This methodology was used to 

evaluate the health and performance of the Laboratory and was not used in this paper. 

Another technique was used by the Goldense Group in measuring overall research 

and development (R&D) program effectiveness4

. One way to measure overall R&D 

program effectiveness is as follows: 

ROII = ______(Cumulative N-year Proft from New Products)_________

 (Cumulative N-year Expenditure on New Product Development) 

wherein: ROII = return on innovation investment 

 N = a given number of years after the investment is made 

 

The numerator is sometimes referred to as “profit before tax.” The above equation has 

no time value term. By using the net present value (NPV) for both the numerator and 

denominator, one can take time factors into account. NPV can be affected by inflation 

and other changes in the cost of raw materials, salaries, and sale price of the new product. 

NPV allows comparison of projects with different time frames and different investment 

requirements. 

This proposal offered another key metric, the new product development (NPD) 

success rate. This is defined as the fraction of NPD projects entering the commercial 

development stage that become commercial successes, meeting or exceeding financial 

objectives. The average success rate for U.S. industry is 60.2%. This methodology was 

not used here since it is intended to identify program effectiveness versus decisionmaking 

in selecting programs. 

Another methodology responds to the question of whether risk management makes a 

difference; this information is based on a paper by the Association of Reported Project 

 

4

 Running Your Lab Like a Business, by John Borchardt, From "Leadership and Staffing" section of the 

LAB MANAGER magazine (P.O. Box 216, 478 Bay Street Midland, ON, Canada, L4R 1K9), Published 

7/21/2008 
4

Risk Management Practices and Project Success/Project Management Institute.5 The 

overall question answered by this paper was: does risk management make a difference? 

That is, do organizations that employ formal risk management practices outperform those 

that do not? Data collected from 175 web-survey respondents and 12 selective telephone 

interviewees from the Risk Management Specific Interest Group of the Project 

Management Institute answer this question in the affirmative, and the data analysis 

supports the positive influence of project risk management. Several detailed conclusions 

are presented and explained. 

In the course of this research we found that adequate resource allocation and staff 

training for project risk management are less pronounced than risk visibility in 

organizational policymaking, and that fewer than half of the respondents (45%) use 

quantitative tools on an “Almost Always” basis to develop contingency costs. Even fewer 

use them to estimate contingency times (38%) or to select projects (7%). In contrast, 

analysis of the data supports generally positive influence of project risk management. In 

particular, the following four general conclusions can be made: 

1. The more sensitive senior management is (perceived to be) to project risk 

management, the more frequent is the use of various project risk management practices. 

For example, 37% of respondents from organizations with a stated risk management 

policy report using qualitative risk tools “Almost Always” versus only 8% of the 

respondents from organizations without such a policy. 

2. The more that senior managers provide adequate resources for risk management 

processes, the more frequent is their implementation. For example, 58% of the 

respondents reporting receiving adequate resources for risk management likewise report 

using qualitative risk techniques during project selection “Almost Always,” whereas only 

10% of the respondents “Rarely” receiving such support also report “Almost Always” 

using risk tools during project selection. 

3. The more that formal risk planning practices are implemented, the more rigorous 

is risk monitoring. Also, fewer workarounds are reported. For example, 43% of the 

respondents reporting use of qualitative risk techniques during project selection also 

report conducting risk reviews “Almost Always,” whereas only 10% of those respondents 

reporting “Rarely” using such tools also report “Almost Always” conducting risk 

reviews. 

4. The more that project success is reported, the stronger senior management support 

for formal risk planning efforts, their subsequent actual practice, and regular risk 

monitoring are also reported. For example, 53% of the respondents who report their 

projects conduct risk reviews “Almost Always” also report completing projects on time 

“Almost Always,” whereas only 15% of the respondents who report “Rarely” conducting 

risk reviews also report completing projects on time “Almost Always.” 

In conclusion, there is a statistically significant relationship between senior management 

support for project risk management and the presence of a project risk management 

process and between each of these variables and reported project success. This 

methodology was not used here since it is more focused on the value of risk management 

versus its utility in program selection. 

 

5

 Excerpt from the paper by the Association of Reported Project Risk Management Practices and Project 

Success(published in 2005), from Project Management Association Finland, Elisantie 16 02970 Espoo 

Finland 
5

Another methodology provides a sample risk based decisionmaking graph. 6

 Prior to the 

attack on the USS Cole and the events of September 11, 2001 (9/11), the U.S. Navy 

(Navy) was developing antiterrorism (AT) programs to address potential terrorist attacks 

in a resource-constrained environment. Amidst unprecedented tragedy associated with 

these and other terrorist strikes against U.S interests, both domestic and abroad, 

government agencies, the armed forces, and private companies have rapidly executed a 

vast array of security improvements to ensure mission capability and protect personnel, 

critical facilities, transportation systems, and other infrastructure. The Navy has allocated 

resources to local commanders to mitigate these identified force protection challenges; 

however, development of Navy-wide investment strategies based on assessment of threat 

or vulnerabilities proved programmatically elusive, thus creating the need for a different 

investment protocol. Given the challenges of allocating limited resources among many 

AT capabilities, each with demonstrated abilities to reduce the vulnerability of 

individuals and property to terrorist acts, the Navy is incorporating risk-based 

decisionmaking (RBDM) processes (i.e., decision processes that are repeatable, 

consistent, and defendable) into its resource allocation model. This risk based 

decisionmaking process was not used here since it was more focused on operations 

versus technology program selection. 

Another approach is the Science & Technology Risk Model developed by the 

Homeland Security Institute.7

 It is a model that compares all S&T capability gaps and 

programs. It is based on three elements: attack/hazard scenarios, risk calculations, and 

mapping gaps and programs to scenarios. This model will not be discussed here due to 

the sensitivity of the dissemination of such information. Based on the review of these 

methodologies, the team believes that the current figure of merit discussed in this paper is 

more relevant to the decisionmaking process within the DHS S&T organization. It is 

based on decades of experience by the study team in laboratory and S&T program 

management. 

 

6

 Applying Risk-Based Decision-Making Methods and Tools to U.S. Navy Antiterrorism Capabilities, by 

Charles Mitchell and Chris Decker. Homeland Defense Journal, February 2004. 

7 S&T Risk Model for Science and Technology Planning and Resource Allocation, Volume I- Final Report, 

26 August 2008, Homeland Security institute, Arlington, VA.
6

Discussion–Risk Areas and Impact in Decisionmaking 

There are many methodologies for risk informed decisionmaking as outlined in the 

previous section. The panel decided to choose one particular methodology that was 

developed with the first author being part of a Defense Science Board Study8. The Board 

adopted a figure of merit technique for decisionmaking in the DOD technology base. This 

methodology, balancing opportunity with risk, was used to identify the top 17 

technologies. The same concept is used in this paper with different elements of 

opportunity and risk. A list of the major risk areas in S&T decisionmaking, including 

both technical and political factors was produced. Using this list, the team created logical 

sub-elements for each risk element in order to break down the needs in greater detail. 

Technical risk areas included skilled technical management; S&T workforce; odds of 

success; facilities and equipment; timeliness; and champion. Political risks included 

earmarks; Congressional direction; micromanagement by Congress or headquarters; and 

champion. 

Additional risk areas included transition risks; financial risks; and institutional risks. 

Transition risks included items such as customer pull, trust issues, champion, and 

timeliness. Financial risks included concerns about adequate funding for the effort, 

stability of funds, and timeliness. Institutional risks included relevance of mission, 

leveraging, social acceptability, balance of programs (near term vs. short term), legal 

issues, and organizational constraints. 

The team prioritized each of the sub-elements using a “voting” technique. It used the 

“1/2 +1” technique, wherein the total number of items to be prioritized is divided by two, 

and then one is added to that number. There were eleven items to consider, so that 1/2+ 1 

equaled 6.5 which was rounded off to 7. Each team member then assigned the number 7 

to what he considered the most important sub-element, the number 6 to the next most 

important sub-element, the number 5 to the third most important sub-element, and so 

forth. The total votes per sub-element were tallied and the process resulted in the risks 

being prioritized as listed below – with 1 being the element providing the most risk. 

1. Workforce: in-house and outside capability 

2. Customer: lack of customer support. 

3. Adequacy of funding. 

4. Mission relevance: part of core mission of organization. 

5. Odds of success: technical. 

6. Leveraging: utilizing resources of other organizations. 

7. Champion: chain of command/Congress willing to support effort. 

8. Balance of program: long term vs. short-term. 

9. Timeliness: may not meet the needs of the day. 

10. Facilities: existence and accessibility. 

11. Social acceptance/legality: lack of public acceptance. 

 

8

 Report of the Defense Science Board 1981 Summer Study Panel on the Technology Base, November 

1981, Office of the Undersecretary of Defense for Research and Engineering, Washington, DC 
7

 It should be noted that the prioritization of the risk areas is that of the team members and 

not necessarily DHS priorities. DHS priorities would be different, especially when it 

came to issues dealing with the public and domestic laws and regulations. Following risk 

prioritization, the team listed the key elements of impact, utilizing the same technique 

used to calculate and prioritize the risk areas. The list included, from most to least 

impact: mission; pervasiveness; knowledge base; need to avoid technical surprises; 

leadership; long-lasting effects; utility of work to others; commercial off-the –shelf 

(COTS); affordability; and convenience/acceptance. The risk to the American people has 

many dimensions. For example, there is the risk that a technical proposal will be 

unsuccessful, perhaps because it is a bad idea, or is premature, or the available funding is 

inadequate. The risk is that time and money will be wasted. There is a risk that not 

funding something denies the development of a capability that could have been critically 

important, hence, failure to develop costs lives. There is the risk of developing something 

that creates more serious problems than it solves and itself becomes a threat to the 

American people (examples here might be in the area of civil liberties, or a biology 

undertaking that backfires). There is the risk of spending money on things or areas that 

are actually not risks or serious threats. There is the perception of risk and the actual risk, 

e.g., we perceive that shark bites are common, but statistics say they are not. Thus, other 

than high, medium, or low it may be hard to come up with meaningful measures of risk. 

A brief description of each of the impact elements follows. Mission value involves a 

major contribution to mission advancement. Pervasiveness involves a contribution of 

some value to a wide variety of missions. Knowledge base means contribution to new 

understanding, as well as the development of a quality workforce. Avoiding technology 

surprises means knowing the technical advances others are making. Leadership implies 

international expertise in the area. Duration of impact implies long-lasting utilization. 

Utility of work implies for DHS utilization at state and local levels. Leveraging COTS 

means utilizing commercial off-the-shelf equipment whenever possible. Affordability 

means cost of development, implementation, and maintenance. Convenience/acceptance 

means minimal public disruption. 

The team next demonstrated how a figure of merit for a selected technology area can 

be determined, by contrasting risk versus impact using the risk elements and impact 

elements discussed above. The team selected three test cases: for each test case, the team 

assigned a 9, 4, or 1 to each risk element and each impact, tallied the results, and plotted 

the results on a risk vs. impact chart (see Figure 1). Nine was considered high, four was 

considered moderate and one was considered low. The figure of merit was calculated as 

the sum of impact divided by the sum of risk. Values were determined by quantitative 

data when available, as well as by qualitative judgments. For this activity, the team used 

the members’ consensus to arrive at the nine, four or one result. More sophistication 

would include gathering data for each element, thereby providing a more quantitative 

approach. 

The team used three sample programs to demonstrate the methodology. The programs 

included the Army Electric Gun; composites for the Air Force B-2 bomber; and a DHS 

High Priority Technology Need. In the DHS case, there was a need to develop the 

technology to enable the detection of person-borne concealed explosive threats in various 

high-throughput venues at standoff distances. 
8

Conceptualizing Metrics 

A chart to conceptualize metrics was developed that plots projects against a risk 

(vertical) and an impact (horizontal) axis. The best-case scenario would be a project that 

would have greatest impact with minimal risk. This approach made an unambiguous 

distinction between a successful program, the B-2 bomber; an extremely important 

program dealing with IED detection; and a questionable investment opportunity, the 

Army electric gun. The team proposed that a possible approach for DHS to use is to put 

all of its programs through such an evaluation. The funds supporting those programs that 

are high-risk but low impact could become available to reinvest in those that are high 

impact/low or moderate risk. The programs that have the same impact-to-risk ratio could 

be used to insure that a proper balance is achieved between near- and far-term 

investments; in other words, there should always be some high-impact/high-risk 

programs that push the S&T envelope and act as catalysts for paradigm shifts in strategic 

and technological thinking. Continuing the low-impact/low-risk programs should be 

questioned. 

Conceptual Metrics

Gun

65-21

B-2

28-56

IED

22-52

Risk

Impact

Figure 1. Conceptual Metrics 

The Risk Management and Decision Processes Center of the Wharton School of the 

University of Pennsylvania was consulted regarding the approach adopted in this study. 

The Center has over 25 years of experience in various aspects of risk management 

research for both the public and private sectors. The Center agreed with the approach 

taken by the study team but also recommended extending the figure of merit over time to 

include initial, midterm, and final term. In addition, they suggested examining the 

interdependency of the programs after selection and addressing any concerns raised by 

such examination. 
9

Decisionmaking Process for Program Evaluation 

The CTNSP team developed a flow chart for the decisionmaking process that could 

be used by evaluators to walk through the risk vs. impact evaluation (see Figure 2.) It 

starts with existing program needs/requirements or new ideas that have to be 

implemented. These go through the impact vs. risk assessment process detailed earlier 

resulting in a figure of merit (FOM). If the FOM is less than a desired threshold x, the 

leadership can still go forward with the program by exercising executive privilege. If it is 

“in lane,” then it should be accommodated. By “in-lane” is meant that the program 

evaluators determined the programs fit into the mainline activities of the DHS and have 

had some level of stable funding over the past several years, with that level projected to 

remain constant. If the program is not in lane, but the resulting FOM is greater than the 

threshold x, then it goes to the next step of institutional advocacy process, where it is 

vetted against funding and prioritizated needs. If the program/new idea falls within an 

existing lane, it is then absorbed. If a new lane has to be created for the program, the 

laboratory or the institution is either appropriated new funding for it or it absorbs the 

costs. If the laboratory/institution has to absorb the cost of the new idea/program, it 

would most likely readjust its existing programs and evaluate them against prioritized 

needs. Then, funding adjustments would be made on existing programs to support the 

new program. This decision process involving selection and funding can be applied to all 

new ideas and to existing programs. 

Existing Program

New Ideas

Impact vs Risk

Assessment

FOM

FOM <X- Exercise executive privilege

In Lane- accomodate

Not in Lane- FOM >X

Institutional Advocacy

Funding

Prioritization

Existing LaneAbsorb

Create New Lane

Absorb Cost New Appropriation

Readjust Existing Program

(Prioritization)

Tax Cut

Chain of Command Other Approaches

Decision Process

Figure 2.Decision Process 
10

Decisionmaking and Program Management 

During the course of the preceding evaluations, and in the day-to-day operations of a 

laboratory, the CTNSP team members suggested that there are at least five different 

styles of decisionmaking that can be used by senior leaders: command and control 

(follow orders as in military); consensus building (workable, requires knowledgeable 

leader, takes time and is actually harder, provides the best answer, but could lead to less 

risk taking); self policing (requires knowledgeable leader in technologies, uses peer group 

working for the leader to evaluate technologies); intuitive (using intuition after reviewing 

inputs from organizations); and distributive (leader not knowledgeable in technologies 

but good in leadership skills and management). The CTNSP team recommends that 

command/control and distributive styles not be used by the DHS. The team did favor the 

consensus building approach, but cautioned that it could be time consuming. Nonetheless, 

it usually leads to the “best” answer, the one that takes into account many diverse views. 

Self policing is effective if the organization is well populated with strong technical peers. 

The intuitive approach should be limited to whenever all other techniques fail, since it 

relies on the “gut feel” of a high-quality leader who is well-respected. 

While the team did not discuss in detail the differences between leaders and 

managers, the members agreed it was important to recognize they are different concepts. 

Warren Bennis,9 a noted authority on this subject, identified several differences: 

• Managers administer, leaders innovate 

• Managers ask how and when, leaders ask what and why 

• Managers focus on systems, leaders focus on people 

• Managers do things right, leaders do the right things 

• Managers maintain, leaders develop 

• Managers rely on control, leaders inspire trust 

• Managers have a short-term perspective, leaders have a longer-term perspective 

• Managers accept the status-quo, leaders challenge the status-quo 

• Managers have an eye on the bottom line, leaders have an eye on the horizon 

• Managers imitate, leaders originate 

• Managers emulate the classic good soldier, leaders are their own person 

• Managers copy, leaders show originality 

Most organizations are usually populated with good managers and fewer good 

leaders. However, balance between leadership and management is key to the scientific 

organization’s success. The team reiterated that fortunately, leadership can be an acquired 

quality and can be learned with enough dedication and hard work. However, too little of 

this key training is actually done. Based on the personal experience of team members, 

most directors of research and development organizations have both good leadership and 

managerial skills. 

 

9

 http://www.phd.antioch.edu/Pages/APhDWeb_Program/bennis. Dr. Warren Bennis (Antioch College, Class of 

1951) is one of the nation's foremost authorities on organizational development, leadership and change. He is currently 

a distinguished member of the Antioch College Ph.D. Program's Panel of Visitors, and was a member of Antioch 

University's Board of Trustees (1968-71).
11

Conclusions 

A risk versus impact analysis tool for decisionmaking was developed. This tool 

utilized technical, political, transition, financial, and institutional risk areas that were 

prioritized. The impact areas included mission, pervasiveness, knowledge base, avoiding 

surprise, leadership, longevity, utility to others, commercial off-the-shelf (COTS), 

affordability, and convenience/acceptance to the American public. The impact areas were 

also prioritized. A figure of merit, which is the sum of the impact areas divided by the 

sum of the risk areas, was generated. This figure of merit was applied to three examples: 

Army electric gun, composites for AF B-2, and the DHS high priority technology need of 

detecting person-borne IEDs from a standoff distance. The team then examined the flow 

process for potential program selection and funding. The styles of decisionmaking within 

DOD Laboratory management were then considered and the differences between 

leadership and management were highlighted. The team believes that the tools provided 

in this paper will be of value to the DHS S&T leadership in the process of program 

selection and ultimate funding. The next step is to apply these tools to other high-priority 

technology needs within the various divisions of the DHS S&T Directorate. Of course, 

these risk-informed decisionmaking tools can be used by any organization for analytical 

S&T program decisionmaking. 
12

Appendix. Performance Evaluation and Metrics for 

Program Selection, Army Research Laboratory 

Army Research Lab introduced its management innovation, which was a methodology to 

evaluate the health and performance of the laboratory. Research & Development 

performance evaluation had been a problem for the Army that has eluded a satisfactory 

solution for many decades. 

ARL developed a Performance Evaluation Construct. It was a semi-quantitative approach 

that required the Lab Director to take a broad view of many different factors, some 

numeric and some descriptive, and then to personally integrate them into a picture that he 

can present to a variety of audiences (including stakeholders) in whatever format is 

appropriate at the moment. Other evaluation areas were peer review, customer evaluation, 

relevance, productivity and quality. The interrelationship of all six constructs was then 

plotted on the following chart: 

Figure 3. Performance Evaluation Construct 

Peer Review 

To better utilize the concept of peer review, ARL contracted with the National Research 

Council of the National Academies of Science and Engineering to assemble an ARL 

Technical Assessment Board (TAB). The TAB consists of individuals of international 

reputation. Under this board are six panels of 8 to 10 people each, of high reputation in 

the international community. These panels review each of the ARL primary mission 

areas. The Board, with its panels, provides an appraisal of the scientific and technical 

efforts of ARL. It is specifically enjoined from making judgments on the programmatic 
13

structure of ARL work, partly because it is not equipped to perform this function, and 

partly because there are numerous other channels through which ARL receives such 

guidance. 

Because of the size and diversity of the ARL technical program, the panels review the 

total program in depth over a three-year period, reporting annually to the TAB on the one 

third of the program reviewed. The panels produce a descriptive assessment in the form 

of a written report published by the National Research Council. The TAB also meets 

annually with the Director to provide an informal report of its findings before the 

publication of the report, and to receive guidance on the Director’s desires for special 

areas of emphasis in the next year’s review. 

Metrics 

ARL assembled a set of about 60 metrics for consideration. They were broken into 5 

major categories: 

Preeminent in key areas of science 

Staff widely recognized as outstanding 

Miscellaneous 

Seen by Army users as essential to their mission 

Intellectual crossroads for the technical community 

Of the 60 metrics proposed, 17 were selected: 

• % of top tasks met; 

• % of Science and Technology Objectives met; 

• # of refereed papers/proceedings leaving tracks; 

• # of ARL technical reports leaving tracks; 

• % of Ph.D.s on the staff; 

• # of employees on long–term training; 

• # of employees on academic training; 

• total semester credit-hours completed by Ph.D. candidates; 

• indirect overhead in ($M); 

• ratings form customer surveys- technical planning annexes and reimbursable 

customers; 

• reimbursable customer orders (in $M); 

• total # of guest researchers out of ARL; 

• total work-year equivalents for guest researchers out of the ARL; 

• total # of guest researchers into ARL(# of post-docs); 

• total work-year equivalents of guest researchers into ARL. 
14

Lessons learned 

Lessons learned from the ARL performance evaluation were: 

The survey process for customer feedback is more applicable to the applied research 

that ARL does than to the more basic, opportunity-driven research. 

Peer review is more applicable to the ARL basic research work 

Metrics are applicable with varying emphasis, depending on the metric and when it is 

applied. 

This means that the three pillars of the performance evaluation construct are given 

different weights at different points in the R& D life cycle. The use of the metrics 

presents some difficulties. Good management practice suggests that only a few key 

metrics will be useful in evaluating performance. The metrics developed above are 

mostly measures of the research work while in progress, not of the ultimate use of the 

results. The laboratory has little control as to whether or not the parent organization is 

able to make good use of the research findings and, furthermore, the ultimate outcome of 

the laboratory work may be several years in the future. In the Army, it may be one or two 

decades between the laboratory demonstration of a new concept and its fielding in a war 

fighting system. ARL has concluded that there simply are no meaningful outcome 

metrics, in the sense that a manufacturing or service company would use them. ARL has 

concluded that there is no satisfactory approach to predicting the utility of laboratory 

results many years in the future. This has been confirmed in extensive discussions with 

the senior leaderships of leading American technology companies.