Timothy Coffey, Eli Zimet, Fred Saalfeld, and Timothy Lo 

Center for Technology and National Security Policy 

National Defense University 

September 2006 

Overcoming the S&T

Assessment Uncertainty

Principle

An Approach to Enterprise-Wide

Assessment of the DOD S&T Program
ii

The views expressed in this article are those of the authors and do not reflect the official policy 

or position of The National Defense University, the Department of Defense, or the U.S. 

Government. All information and sources for this paper were drawn from unclassified materials. _____________________________________________________________________________________________ 

Timothy Coffey holds the Edison Chair for Technology in the Center for Technology and 

National Security Policy. He is also a Senior Research Scientist at the University of Maryland. 

He was the Director of Research at the U.S. Naval Research Laboratory from 1982 to 2001. He 

has published extensively in the scientific literature and has served on numerous advisory panels. 

Dr. Coffey holds a BS (EE) from the Massachusetts Institute of Technology and a PhD (Physics) 

from the University of Michigan. 

Elihu Zimet is a Distinguished Research Fellow at CTNSP, The National Defense University. 

He is also a Senior Research Fellow at the Potomac Institute for Policy Studies. Prior to this he 

headed the Expeditionary Warfare Science and Technology Department at the Office of Naval 

Research. In this position he directed science and technology (S&T) programs in missiles, 

directed energy, aircraft, and stealth as well as S&T support to the Marine Corps. Dr. Zimet 

holds a BS (ME) from the Polytechnic Institute of Brooklyn and a PhD from Yale University. 

Fred Saalfeld headed the Naval Research Laboratory’s (NRL) Mass Spectrometry Section. Dr. 

Saalfeld also served as Superintendent of the NRL Chemistry Division, and Associate Director 

of Research for Material Sciences and Component Technology. He then served as the Chief 

Scientist and Scientific Director at the Office of Naval Research (ONR) Branch Office, London, 

the Director of ONR, and Technical Director and Deputy Chief of Naval Research, responsible 

for the Navy and Marine Corps S&T program. 

Timothy Lo is a Research Associate at CTNSP, NDU. He holds a MS in Public Policy and 

Management and a BA, both from Carnegie Mellon University. 

Acknowledgements: Special acknowledgements are due the late Dr. John C. Hopps for his 

insights regarding this study and Dr. Kevin Bowcutt of the Boeing Company for his 

contributions in developing the hypersonic technology examples discussed in this report. The 

authors benefited greatly from the helpful comments and insights provided by Dr. Ronald 

DeMarco and Dr. Ronald Kostoff. The authors also acknowledge the able assistance of Mr. 

Justine Perkins.

Defense & Technology Papers are published by the National Defense University Center for Technology and 

National Security Policy, Fort Lesley J. McNair, Washington, DC. CTNSP publications are available online at 

http://www.ndu.edu/ctnsp/publications.html.
iii

Contents 

Executive Summary.........................................................................................................................v 

1. Introduction..................................................................................................................................1 

2. Relationship of Current DOD S&T Documentation to Proposed Assessment............................5 

 

3. Established Review Process.........................................................................................................9 

 Background..........................................................................................................................9 

 Types of Reviews and Purpose..........................................................................................10 

 Conventional Review Process............................................................................................12 

 Service and DDR&E Reviews...........................................................................................13 

 Current Review Process Strengths and Shortcomings.......................................................14 

 Conclusion Regarding Conventional Reviews ..................................................................15 

4. An Approach to Enterprise-Wide Assessment ..........................................................................17 

 Background .......................................................................................................................17 

 Program Documentation ...................................................................................................17 

 Establishing Communities of Experts................................................................................30 

 Assessment Process ...........................................................................................................34 

 Displaying Assessment Results .........................................................................................37 

5. Conclusions and Recommendations ..........................................................................................47 

 

Appendix A: Work Unit Summary Hypothetical Example ..........................................................A1 

Appendix B: Potential Communities of Experts...........................................................................B1 

Appendix C: NASA Technology Inventory .................................................................................C1 
iv
v

Executive Summary 

The science and technology (S&T) program supported by the Department of Defense (DOD) is a 

large undertaking (~$11B in 2004) aimed at ensuring continued technological superiority of the 

U.S. military. Exercising stewardship over this program is of great importance. In this regard the 

program is subjected to numerous reviews dealing with its relevance, viability and productivity. 

At various levels of detail, every element of the program is reviewed. However, the sum of all 

the reviews does not constitute an assessment of the entire DOD S&T enterprise. This study 

examines why this is the case and examines the prospects for rectifying this situation. 

An effective enterprise-wide assessment (EWA) would contribute to the ability of the Director of 

Defense Research and Engineering (DDR&E) to provide oversight to enhance the content of the 

S&T portfolio through metrics of program quality, performance, and relevance. It would increase 

collaboration between projects with aligned objectives and provide a knowledge base of the S&T 

program sufficient to justify the investment to the services and Congress. An effective EWA 

would also support the services in that it would focus their internal topical reviews to problem 

areas identified in the EWA. It would also provide them a broader insight into the entire S&T 

program and their place within it. This would result in greater program coordination and reliance 

on other service programs. Also, an EWA could form the basis of an interrelated community of 

experts across the entire S&T spectrum. 

Principal Findings 

• Many different and valuable reviews of the DOD S&T program are conducted. However, 

there is no obvious way (short of drastically increasing the resources available to the 

review process) to modify the current review structure so as to produce a coherent EWA. 

The underlying problem is the sheer magnitude and breadth of the program. This results 

in an inability to assess it in scientific and technical detail and to simultaneously examine 

the entire program. We refer to this conundrum as the S&T assessment uncertainty 

principle. A new methodology must be developed to overcome this uncertainty principle 

if an EWA is to be institutionalized. 

• The current documentation of the DOD S&T program does not reflect the state of the art 

of information technology, is marginal for exercising stewardship over the program, and 

is inadequate for the conduct of an EWA. 

• DOD does not access systematically the breadth of scientific and technical knowledge 

and expertise that is potentially available to it. The establishment of and ready access to 

communities of experts that cover all DOD areas of scientific and technical interest 

would be of great value in itself and will be required for an EWA. 

• The current state of the art and expected advancements in information technology and 

library science offer the best hope of resolving the above concerns. 
vi

• An EWA will be viable only if it adds value to the parties involved and emerges as a 

byproduct of the normal process of doing business. 

Principal Recommendations 

• The principal investigators of the various DOD S&T projects/work units should be taught 

and required to document their projects/work units in terms of the very specific CEST 

issues that they are funded to resolve. These issues should be stated at the level at which 

people are actually performing work rather than in terms of general objectives. 

• S&T project documentation requirements should be standardized across DOD and should 

contain the information needed for an EWA. Funding for a project/work unit should not 

be released until the organization responsible for the project/work unit submits 

satisfactory documentation to the Defense Technical Information Center (DTIC). The 

DTIC database that results from proper documentation of the program will be broadly 

valuable across DOD (DDR&E, funding agencies, performers, etc.) independent of the 

EWA. The needed information is available at the performer level and can and should be 

documented routinely. This documentation is required as a matter of stewardship and will 

be needed for an EWA. An essential element of the documentation is a succinct statement 

of the critical enabling S&T (CEST) issues being addressed by the funded program so 

that an expert in the specific topical area being pursued can quickly assess the viability of 

the project/work unit meeting stated plans and milestones. 

• National communities of experts should be established for each of the specific CEST 

issues being addressed. The tools of information technology and library science must be 

utilized to accomplish this. Some DOD research investment may be required to resolve 

DOD specific issues in this regard. The establishment of the communities of experts will 

be of great value in itself and will be required for an EWA. 

• For an EWA, the approach of collecting the review teams at a central location for the 

purpose of holding hearings must be abandoned and replaced by a distributed assessment 

process that exploits the documentation of all of the CEST issues, advances in 

information technology and library science, and the readily available access to 

geographically distributed communities of experts. The EWA should be conducted as a 

targeted survey of the appropriate communities of experts. 

• A full EWA as envisioned by this study should not be initiated in the short term. Many 

details need to be addressed before such an assessment could be undertaken. Certain 

aspects of the proposed process should be examined via the conventional review process. 

For example, automated techniques for forming communities of experts could be tested 

by employing them as part of establishing the conventional review teams. Test cases 

should be conducted to clarify how to document the program in terms of CEST and to 

examine and resolve problems that confront the distributed assessment approach. These 

test cases should address classified distributed assessments (e.g., utilizing SIPRnet) and 

unclassified distributed assessments. DOD should be an active participant in the national 
vii

efforts that are underway to take advantage of the ongoing revolution in library science 

(e.g., text retrieval). 

If the processes described in this study could be implemented they would have a number of spinoffs. For example, the supporting DTIC database could be interrogated so as to compare the 

actual program of record with the DOD S&T investment strategy. It would also establish a broad 

database of experts covering most of the S&T areas of interest to DOD. It may be that the most 

valuable result of this approach to an EWA would be the establishment and maintenance of a 

detailed database for the funded S&T program and the development of a large community of 

experts who have a connection with and interest in DOD S&T. It is also worth noting that while 

the proposed methodology is directed at assessing the funded program of record it could, if 

successfully implemented, provide significant feedback to the important matter of program 

formulation.
viii
1. Introduction 

The U.S. Department of Defense (DOD) sponsors one of the world's largest science and 

technology (S&T) programs. In FY 2004 the DOD S&T appropriation was about $11 billion. 

This sum, while less than three percent of the DOD budget, is larger than the entire defense 

budgets of all but eleven of the world's countries and all but four of the NATO nations. This 

large investment has allowed the United States to maintain its technological advantage over 

potential adversaries. It has undoubtedly saved the lives of tens of thousands of U.S. men and 

women in uniform as well as prevented countless civilian casualties. Because of its impact, the 

matter of exercising stewardship over this large investment is of great importance. Stewardship 

in this context has many dimensions. These include, among others: program formulation; the 

appropriation of the needed funds; oversight of expenditures; and the review and assessment of 

the viability and productivity of the funded program. While all of these dimensions are important 

and interdependent, this paper focuses on the latter dimension—namely the review and 

assessment of the viability and productivity of the funded program. 

The very size of the program confronts this aspect of stewardship with a conundrum. This 

conundrum can be illustrated by examining the scale of the effort that would be required to 

assess the program in detail by conventional means at the level where the work is actually done. 

For S&T work, scientific and technical teams typically function in groups of three to five people. 

This corresponds to an average investment “unit” of about $1 million. An $11 billion S&T 

program will consist roughly of 1,100 such units. This overestimates the number of units because 

of funds that are expended on hardware purchases. However, the correction for such purchases 

will not substantially change the conclusions regarding conducting a complete detailed review of 

the program. A proper scientific and technical review involves examining the program at about 

the $1 million level. A conventional review process would convene a team of 5–10 experts to 

examine the program in detail. For each unit the review team would require perhaps one day of 

preparation, one day of hearings and interaction with the performers, and one day of travel. A 

single review team may review several related units, however, a proper expert review will add 

one day for each additional unit. Therefore, 1,100 S&T units would require 16,000 to 32,000 

reviewer work days and the participation of several thousand reviewers. Compensation for the 

review team’s workdays would cost between $20 million and $40 million. The travel and per 

diem costs would be in the $3 million to $6 million range. An estimate of the reviewer cost for 

the complete review of the program is therefore, in the $25–50 million or range. 

The other major cost lies in the preparation for the reviews. The scientists and engineers to be 

reviewed will begin preparation several months prior to the review. The experience of the 

authors suggests that each member of the S&T unit will spend 5–10 days of preparation for each 

day of review. Therefore, the low estimate of the number of workdays that would be spent in 

preparation for a complete review of the S&T program is about 80,000 workdays. This 

corresponds to a total preparation cost of about $100 million. Therefore, the total cost for a 

complete conventional review would be in the range of $125 million to $150 million. In relative 

terms this is only one percent of the DOD S&T program. In absolute terms however, it is a 

considerable amount of money to spend on a review process. In addition to the financial costs 

one must also consider the impact on the program itself of such a massive review. There is also 
2

the matter of finding several thousand expert reviewers who are in a position to devote so much 

time to such reviews. Furthermore, there is the non trivial matter of managing the logistical 

aspects of such a review. 

The practical consequence of the above is that the DOD has been forced to make tradeoffs 

regarding the types of reviews and the program coverage of reviews. A tradeoff can be 

envisioned through the simple empirical mathematical relationship: 

ΔT ΔP > C . (1.1) 

Here C is a constant related to the resources (funds, manpower, time, complexity, etc.) that one is 

willing to apply to the review process, ΔT represents the uncertainty that an assessment strives to 

achieve in its understanding of scientific and technical details, and ΔP represents the 

corresponding uncertainty that will result in the assessment’s ability to understand the total 

program. We will refer to this inequality as the assessment uncertainty principle (in analogy to 

the uncertainty principle of quantum mechanics). The assessment uncertainty principle says that, 

for a given value of C (i.e. allocation of resources to the review process) an assessment must 

trade off the detail at which the S&T program is examined against the amount of the program to 

be assessed. A very detailed examination of the S&T (small ΔT) will generally result in a large 

uncertainty in the total program coverage (i.e. large ΔP). One can, of course, simultaneously 

achieve, in principle, any desired value of ΔP and ΔT by investing sufficient resources to make C 

small enough. This is because the assessment uncertainty principle is not a physical law but 

results from limitations of resources (people, time, funds) available for conducting assessments. 

Such an undertaking, however, requires a very substantial investment of resources (funding and 

otherwise) that the DOD has not been willing to make. 

The practical consequence of the assessment uncertainty principle has been that the DOD 

conducts detailed scientific and technical assessments on small portions of the S&T program and 

conducts management oriented assessments of the full program (or large portions thereof). Both 

types of assessment are quite valuable and are done routinely and professionally by DOD. This 

situation does, however, lead to concerns regarding what is actually being done at the S&T level 

in this large and extraordinarily important program. This concern is felt especially keenly at the 

highest levels of DOD where stewardship for the entire program resides. In an attempt to redress 

this situation Dr. John Hopps, then Deputy DDR&E and DUSD (Labs), requested that the 

authors attempt to construct a methodology that would permit an enterprise-wide assessment 

(EWA) of the entire DOD S&T program. Of special interest was the alignment of the DOD 

laboratory system in terms of its people, programs and facilities against the transformational 

goals of OSD, the Services and the Agencies. It was agreed that a valid assessment of this 

alignment could only be done within the context of the overall DOD R&E enterprise. The term 

enterprise-wide assessment was, therefore, taken to include all performers (i.e. industry, DOD 

labs, universities, etc.). It is important to understand that a key objective of the study was to 

propose a methodology that provided coverage of the entire S&T program and not just selected 

pieces of the program. This has a profound impact on the options that are available. For example, 

techniques have been developed to assess the overall quality of the program by using statistical 

sampling methods for selecting the programs to be reviewed or for evaluating the impact of 

particular programs (e.g. see R.N. Kostoff, “An Assessment of the Basic Energy Sciences 
3

Program,” DOE/ER – 0123, March 1982; R.N. Kostoff “The Handbook of Research Impact 

Assessment” DTIC Technical Report ADA 296021, 1997). Such approaches are very valuable as 

indicators of overall quality. However, they do not satisfy the entire enterprise requirement of 

this study, nor do they provide an adequate forcing function regarding the documentation of the 

entire program. 

The early phases of the study brought to mind Edison’s famous comment when asked if he ever 

became discouraged working so long without results: "Results? Why I know 50,000 things that 

won't work". In this study, as with most others, it was found that things are the way they are for a 

reason. The first major finding of the study was that the database needed to undertake such an 

assessment did not exist. It had been hoped that the DD1498 documentation would provide a 

rudimentary database for some preliminary work and form a base from which to evolve the 

required documentation. However, the maintenance of that database was found to be no longer 

required. That was unfortunate, because that database was the only one that provided any 

detailed scientific and technical information on the entire program. It was quickly realized that 

an entirely new database would be required to support an EWA. This led to inquiries into 

whether the data formats for the conventional review processes and the methodology used by 

those processes could form the basis for an EWA. It ultimately became clear that the 

conventional review processes could not be morphed into a tool for an EWA. The S&T 

assessment uncertainty principle simply would not allow it. Fortunately, unlike the uncertainty 

principle of quantum mechanics, the S&T assessment uncertainty principle is not a law of nature. 

It is an empirical relationship that is determined by the assessment process itself and the tools 

that are employed to support that process. The study examined what it was about the 

conventional assessment methodology that resulted in inequality (1.1). It became clear that the 

ponderous, manpower-intensive character of the conventional review process and the current 

methods of program articulation and documentation were among the major driving factors that 

impose the inequality and determine the constant C. 

After considerable deliberation regarding alternate approaches, the study came to the conclusion 

that the most promising approach to an EWA involves assessing the entire program at a fine 

level of scientific and technical detail. This contradicts the assessment uncertainty principle 

(unless one is willing to drastically increase the resources available for the review process) and is 

completely orthogonal to the conventional review processes. This somewhat surprising 

conclusion has its basis in the reality of how one actually decides whether or not a particular 

scientific and technical approach to address a particular problem has merit. In the early phases of 

decision one does not convene a massive review. Rather, one identifies one or perhaps several 

individuals who have detailed knowledge and specific expertise in the scientific and technical 

areas being proposed. A brief conversation (by phone or meeting) is held during which the expert 

is given a brief description of the proposed approach and asked to provide an "on the spot" 

opinion regarding the viability of the proposed approach. If the individual is truly expert and if 

the problem/approach is carefully and succinctly stated then the "on the spot" opinion is often 

remarkably accurate. The secret of success here is to find the right individuals who truly 

understand the specific S&T involved and to pose the question to them properly and succinctly. 

The study ultimately focused on how the above approach might be extended/institutionalized for 

the purpose of an EWA. The remainder of this paper offers some observations and conclusions in 
4

this regard. The keys to success for an EWA will be found in the proper documentation of the 

program, the careful and succinct identification of the CEST issues that the projects are funded to 

resolve, the ability to rapidly establish "communities of experts" in these CEST areas, the use of 

a distributed assessment process employing survey techniques, and the employment of advanced 

information S&T (most especially advances in library science). Advances in library science are 

now occurring at a remarkable pace and will change forever how S&T knowledge is accessed 

and how S&T programs are assessed. It is significant that each of these “keys to success” would 

be valuable to the DOD S&T program independent of an EWA. It is also worth noting that, 

while the proposed methodology is directed at assessing the funded program of record it could, if 

successfully implemented, have significant feedback on the important matter of program 

formulation. 
5

2. Relationship of Current DOD S&T Documentation to 

Proposed Assessment 

Proper documentation of the DOD S&T program plays an essential role in the assessment 

methodology proposed in this paper. This section briefly examines current documentation from 

the perspective of what this study recommends regarding documentation. An essential element of 

the on-line assessment approach developed in this report is a searchable database that defines the 

critical descriptive elements and metrics needed for an assessment. Chief among these are the 

objectives of critical enabling S&T (CEST). Furthermore, these CESTs must be developed at the 

work unit level for a rapid and accurate assessment by an expert reviewer. In terms of existing 

S&T databases, the authors of this report have assumed that, in order to properly manage the 

S&T program, the data needed for an assessment already exists, at least at the performer level. 

The authors have also concluded that there has been no central, detailed, DOD S&T database at 

the work unit level since the requirement for submission of the DD 1498 data form was 

eliminated a few years ago. Therefore, the data at the work unit level exists at the performer level 

but is not formally provided to DDR&E. 

 

In response to the E-Government (E-Gov) Act of 2002, DDR&E in May 2005 launched an R&E 

Portal maintained by DTIC. Presently the portal will include links to the existing databases that 

DDR&E maintains, plus the following. 

y New E-Gov database of current R&E project summaries 

y Research and Development Descriptive Summaries (RDDS) 

y Budget Estimate Submission (BES) 

y Joint Warfare S&T Plans 

y Defense Technology Objectives (DTO) 

y Basic Research Plan (BRP) 

y Defense Technology Area Plan (DTAP) 

The portal will also include data on the S&T workforce and links to other DOD news and data. 

The New E-Gov database of current project summaries could be an appropriate vehicle to 

establish the database required for this study. At the present time, the data requested for this 

section of the report is designed to facilitate sharing of general information on the program in 

order to establish contacts between interested parties. The data required to assess the programs, 

especially the CESTs, are not called for. If DDR&E wishes to implement the assessment 

approach developed in this report, the data call requirements for the S&T submissions to DTIC 

for the R&E Portal could be modified to include the required level of project description needed 

to perform an assessment. The information required to implement the proposed assessment 

methodology is displayed in Appendix A as a sample Work Unit Summary form. If DDR&E 

alternatively wishes to use the proposed assessment process for selected topical reviews rather 

than a global assessment, individual data forms can be required. 
6

A brief description of the other databases linked to the E-Portal follows. The RDDS and BES are 

both budget submit documents used for the appropriation of funding by Congress. Work 

descriptions are brief and at a high level. CESTs are not discussed at this level of description. 

The Joint Warfare S&T Plans focus is on S&T vision, strategy and planning with limited 

technology description. The DTOs and the BRP comprise the most extensive current database of 

S&T. However, DTOs only address 40 percent of the program and are more focused toward 

objectives and milestones than to CESTs. The DTAP presents DOD objectives and investment 

strategy for ten technology areas critical to DOD acquisition looking across Service and Agency 

efforts. As with the other documents it is not written at the work unit level. 

There are a few other databases and program descriptions that are maintained by DOD or the 

Services. For example, the DOD maintains a Militarily Critical Technologies List (MCTL) as 

part of the Militarily Critical Technologies Program (MCTP). While the scope of this program is 

to determine existing technologies of use to potential adversaries there are features of this 

program of interest to this report. In particular the MCTP maintains a roster of about 1,000 

subject matter experts from the military Services, DOD, other federal agencies, industry, and 

academia. These subject matter experts are analogous to the community of experts proposed to 

assess the DOD S&T enterprise. The MCTP however does not use search engine technology to 

find experts and the scope of the MCTL is only a subset of the DOD S&T enterprise. Another 

feature of the MCTL is the generation of data sheets that require Critical Technology Parameters 

that are defined at the same level as this report recommends. Due to the nature of the MCTL 

however, these Critical Technology Parameters represent existing technology and are not future 

project objectives as defined in this study. 

Each of the Services documents its own programs. There is no common format and this 

information is not generally distributed. The Army produces a fairly comprehensive two volume 

report on its S&T program entitled Army Science & Technology Master Plan. While this plan is 

comprehensive, it only gives project descriptions for the Army Technology Objectives (ATOs 

were adopted by DDR&E as DTOs). As with DTOs, the AROs only represent part of the Army 

program and do not provide the CESTs necessary for the proposed assessment methodology. 

NASA has generated a promising database that has provided a rough template for this study. A 

brief description of the NASA Technology Inventory will be found at Appendix C. 

A key element for an EWA is a searchable database of the entire program that is generated at the 

performer level. A comprehensive, searchable database that documents the DOD S&T program 

at the performer level should exist independent of the desire for an EWA. Technology allows for 

the creation of such a database; routine documentation of the program at the performer level 

permits it and stewardship for the program requires it. 

 This study concludes that a comprehensive programmatic database, similar to the form DD1498, 

be established as a requirement for work units in DOD and be maintained by DTIC. Aside from 

the necessity of a comprehensive database to perform a meaningful EWA, better program 

documentation will provide other benefits such as more effective development of cohesive major 

S&T initiatives such as development of hypersonic flight vehicles. 
7

The authors contend that the information to generate an assessable database exists at the 

performer level, and the electronic means to collect and distribute the information needed for an 

assessment exists at DTIC through the E-Portal for S&T. However, the requirement to provide 

DDR&E with this level of work unit detail does not exist. Such a requirement is a necessary 

component of an EWA. 
8
9

3. Established Review Processes 

Background 

The purpose of this section is to briefly describe the current methods that the services, DDR&E, 

and Congress employ to assess the S&T program. As will be shown, there are several review 

processes in place in the services, DDR&E, and congressional staff. It is probably fair to say that 

at some level of detail every element of the S&T program funded by DOD is subjected to some 

type of review. However, there is no uniformity across or even within services as to the level of 

granularity and the composition of a review body. The purposes of these individual reviews vary 

according to the needs and management structure of the reviewing organizations. Also there is 

no uniformity in the documentation of reviews, and what documentation there is is often not 

generally distributed to the broad S&T community. Therefore, the sum of all current reviews and 

documentation does not represent an enterprise wide assessment (EWA). 

An effective EWA would contribute to DDR&E’s ability to provide oversight to enhance the 

content of the S&T portfolio through metrics of program quality, performance and relevance. It 

would increase collaboration between projects with aligned objectives and provide a knowledge 

base of the S&T program sufficient to justify the investment to the services and Congress. An 

effective EWA would also support the services in that it would focus their internal topical 

reviews to problem areas identified in the EWA. It would also provide a broader insight into the 

entire S&T program and the place of the services within it. This would result in greater program 

coordination and reliance on other service programs. Also, an EWA could form the basis of an 

interrelated community of experts across the entire S&T spectrum. 

The starting point for an EWA is a review of the current review processes and an assessment of 

strengths and shortcomings. In support of this objective, a framework is presented that 

characterizes current assessments and reviews of the S&T program by the purpose of the review; 

the level of detail or granularity of the subject matter; and the composition of the reviewing 

body. In a broad sense, program reviews can be divided into two categories: planning reviews, 

where the objective is to assess a program prior to initiation to determine if the work is worth 

funding (is this the right job?), and technical reviews, where the objective is to determine the 

quality and progress of the work (is the job being done right?). A further objective is to 

determine if the work is legitimately S&T or should be a higher category of funding. The 

program assessment methodology proposed in this study is focused on the technical reviews and 

assumes that the objectives of the program are in line with the needs of the military users. A third 

type of review that evaluates completed work done in the past provides hindsight lessons learned 

and program accomplishments. This type of review is not germane to this report. 

The following sections will include a broad description of the established review processes, 

followed by the details of current reviews by the services, DDR&E, and congressional staff. This 

will be followed by commentary on strengths and shortcomings and, finally, by conclusions. 
10

Types of Reviews and Purpose 

Table 3.1 portrays the current types of reviews in effect for the DOD S&T program. The three 

columns are in decreasing order of granularity and increasing order of comprehensiveness. The 

first column, Technical Subject, examines a project at the level where the work is actually done. 

The responsible briefer to a review panel is generally the principal investigator. Because of the 

many thousands of S&T projects at this level of granularity, the entire program cannot be 

reviewed at this level using conventional review procedures. At best, 20 percent of the S&T 

program is currently evaluated at this level. Increasing this fraction significantly would place 

unacceptable demands in terms of time and money on performers and reviewers. In addition, 

there are difficulties in assembling a review board with enough members that there is sufficient 

member expertise as well as a balance of strong opinions predicated on the members’ work or 

bias in the subject being reviewed. Therefore, the current Technical Subject Reviews, while 

sufficient in detail, lack the comprehensiveness for an EWA and are difficult and time 

consuming to conduct. This fact leads to consideration of a new type of review process, as 

discussed subsequently in this report, that aims to provide both the detail and comprehensiveness 

that lead to an EWA. 

Program element 

funding level 

adjustment

Program 

element 

funding level 

adjustment

Ad hoc, special 

interest, topical

Congress

(Congressional 

Armed Services staff)

Program execution, 

obligation, and 

expenditure

Program 

oversight

(TARA)

Ad hoc issue 

resolution

Department of 

Defense

(DDR&E)

Response to 

requirements

and emerging 

transition 

opportunities

Technology User 

Community

(customers of 

technology

e.g. Syscoms & 

operational forces)

Program execution, 

obligation, and 

expenditure

Resource 

balance 

between 

projects

Technical quality

Technical 

progress

Service Internal 

Management

(peer level technical 

experts & 

management)

Investment 

Oversight/ 

Audit

Subject 

Area 

Overview

Technical 

Subject

Table 3.1 Types of Conventional Reviews and Purpose

Types of Reviews

Conducted by
11

Despite the difficulties inherent in conducting an EWA at the scientific and technical subject, 

project, or work unit level (the terms are used interchangeably here), this is the appropriate level 

at which to evaluate the program. In S&T, the devil is in the details. The work unit is where the 

work actually gets done. At this level, an expert in the technical discipline under review can 

readily assess the technical merit of the approach and the feasibility of meeting time and cost 

metrics, evaluate whether the work is S&T in character, and evaluate the competence of the 

performers. Reviews held at higher levels of aggregation serve other purposes, such as ensuring 

program balance and response to requirements. Reviewers of programs at higher levels of 

aggregation are also often reviewing projects in which they are not experts. This leads to longer 

discussion and, therefore, more time needed to reach decisions. Only the subject level review 

gets at the actual details of the work with specialist reviewers, who can usually quickly and 

accurately determine the value of the work. Even at this level, the small number of specialists 

can result in parochial interests influencing the review. 

The next column, Subject Area Overviews, represents the collection of S&T projects that lie in a 

common discipline. For example, the DDR&E taxonomy of Platforms, Weapons, Space, 

Information Technology, Materials, Human Systems, Environments, and Sensors/EW forms a 

fairly comprehensive technology review format, although even at this level DDR&E does not 

conduct a comprehensive review. Another example of a Subject Area Overview is a review of a 

focused initiative (e.g., the National Aerospace Initiative), that incorporates many diverse 

technologies. At this level of granularity, much of the S&T program can be reviewed, although 

usually not in a given year. The DDR&E Technology Review and Assessment (TARA) process 

has reviewed the program at the Subject Area level on a rolling schedule of half the program 

every year (this process is under review and is subject to change). These reviews are a tradeoff 

between comprehensiveness and level of detail and represent DDR&E’s current review 

management tool. However, for an EWA, the TARA process fails to provide both enough detail 

and enough comprehensiveness. 

The final column, Investment Oversight/Audit, has the least technical content, is the most 

comprehensive, and is budgetary in nature. These reviews mostly deal with program execution, 

funding obligation rates, and expenditures. As such, they do not address the level of program 

detail necessary for an EWA. 

The four rows represent an ascending managerial level of program oversight. Program 

development and execution responsibility is primarily at the service level, so the most detailed 

and comprehensive reviews are conducted at this level. The services review the program at many 

levels of granularity including technical subject reviews focused on the quality of the work, the 

relevance of the work to service needs, the progress of the work in terms of cost and milestones, 

and evaluate whether the work is really S&T or in fact a higher category of funding. Technical 

experts and stakeholders in the program generally conduct these reviews. As mentioned earlier, 

there is no consistency in these reviews across services and even within services. The conduct of 

these reviews is generally given to the first or second level of service program management. Due 

to the overall size of even the individual service programs, senior service S&T managers are 

usually involved only in selected topical reviews and in issue resolution at the technical subject 

level. Senior service S&T oversight is more generally focused at the Subject Area level with the 

objective of decisionmaking across S&T priorities. The technology user communities, the 
12

customers of technology, are usually more involved in program formulation and approval than in 

technical quality and execution reviews. The user community’s primary concern is that 

technology is addressing service requirements. In addition, however, users are involved in 

program reviews of technologies near completion and transition to engineering development. 

These are generally 6.3 advanced development reviews. 

The preceding discussion of the customers and audience for current reviews raises the associated 

issue that, while there is currently no EWA, there is also no current service requirement or 

ownership for this product. To be useful, an EWA would need to strengthen the service programs 

but not significantly increase time and money spent on reviews. In this regard, the EWA 

proposed here might actually reduce the number of conventional topical reviews by highlighting 

only the potential problem areas for topical reviews. 

Conventional Review Process 

While table 3.1 portrays a matrix of types of reviews and purposes of reviews as well as 

differences in the details of individual review processes, there are common elements that 

conventional reviews share. First, an external review needs to be distinguished from routine 

supervisory management. The principal distinguishing factor of the external review is the review 

board composed of individuals outside the project being reviewed and selected from the 

community of experts in the discipline to be reviewed. The purpose of the review board is to 

strive for impartial judgment as well as an external perspective. For program credibility and 

program defense, an external review is essential. For a completely independent review, the 

review can be “contracted out” to an organization such as the National Research Council (NRC). 

Each of the services maintains study boards (e.g., the Army Study Board) that generally do 

studies and analyses but also provide program review boards. Predominately, however, the 

services develop their own stable of external reviewers. These reviewers are often technical 

experts with previous professional experience within the services programs. Retired military 

officers are also often solicited for their background in service requirements. 

The selection of the actual reviewers is critical to any review process and is a core issue of the 

EWA process analyzed in this report in which potential reviewers are identified by a computer 

search based on CEST objectives associated with the project to be reviewed. Individuals with 

technical competence developed through work in any field will have biases shaped by their 

experience and affiliations. To compensate for this, a sufficient pool of reviewers is required to 

balance out bias and provide a level of statistical sampling. Many current reviews suffer from 

choosing reviewers too close to the program that do not subject the program to difficult 

questions. 

External reviews tend to be very time consuming, hence the difficulty of performing an 

enterprise assessment. For an important review, one that can influence the direction and the 

funding of the project being reviewed, significantly more time is spent in preparation of briefing 

material, analysis, and documentation of results than in the review itself. Frequently, reviews 

require presentations in a prescribed format, resulting in the creation of dedicated briefing 

materials. The presenters often give dry runs of their presentations to their management and may 

redo their presentations as a result. Reviewers are often given read-ahead material to prepare 
13

them for the briefings. Scoring sheets are usually developed to correlate reviewer comments. 

Following a review, the review board customarily provides preliminary feedback followed by a 

report. 

The DOD S&T enterprise now exceeds $11B a year. If a representative project or work unit were 

about $1M a year (three or four investigators, plus project expenses) 11,000 projects would 

require reviews. DOD has not considered it a priority to perform 11,000 reviews at the project 

level and, therefore, from a corporate viewpoint, only looks at a fraction of its projects (about 20 

percent) at a detailed level each year. The enterprise is only viewed in its entirety in very large 

chucks at the program element level. An EWA at the detailed project level will require a process 

different from the conventional external review. 

Service and DDR&E Reviews 

Each of the services manages its S&T programs differently, particularly the 6.2 Applied 

Research program and the 6.3 Advanced Development program, but each service has a research 

office. The Army Research Office (a subordinate organization of the Army Research Lab), is 

under the authority of the Deputy Assistant Secretary for Research and Technology who reports 

to the Army Acquisition Executive. The director of the Air Force Office of Science Research 

reports to the Air Force Research Laboratory Commander. The Chief of Naval Research reports 

to the Assistant Secretary of the Navy for Research, Development and Acquisition. The Navy 

has had a tradition of funding a larger 6.1 account than the other services, and it manages all of 

its S&T, 6.1-6.3, under the Chief of Naval Research, Office of Naval Research. The Marine 

Corps S&T program, a small component of Naval Research (~ $30M / year), is also managed at 

the Office of Naval Research to provide synergy with related Navy S&T. The Air Force S&T 

program is executed by the Air Force Research Laboratory. The Army does not centrally manage 

S&T programs under a single organization. The Army has four Major Commands and one Field 

Operating Agency of Headquarters DA that own all of the laboratories. The predominant funding 

for Army S&T is managed by Army Materiel Command’s Research Development and 

Engineering Command, approximately 80 percent of the $1.7B annual Army S&T budget. Of the 

three services, the Air Force has given the highest priority to science and engineering education 

to its military officers hence the management and execution of the Air Force programs utilizes a 

higher proportion of service men and women than the other services (with the Navy having the 

highest proportion of civilians). With different management structures and personnel among the 

services it is to be expected that different assessment processes are in place together with 

different documentation procedures. 

Army Review Process. Most Army 6.2 and 6.3 programs are managed as Army Technology 

Objectives (ATOs) (DDR&E has adopted the Army framework of ATOs as DTOs for its own 

TARA reviews). These ATOs are 3–5 years in length and have defined milestones and 

deliverables. The Army reviews these programs annually at the Army S&T Executive level in a 

largely internal review process. The remainder of the Army S&T program is managed at the 

Army Research and Development Center and Laboratory level, generally employing a peer 

review process. The Army documents its program in the Army S&T Master Plan, which is 

published biennially. 
14

Air Force Review Process. The Air Force has the most comprehensive review process for its 

S&T programs. The Air Force Science Advisory Board reviews the program assessing five of the 

ten Directorates each year. The reviews are intensive, generally a week long, and the review of 

the five Directorates takes about two months every year. The Basic Research program is 

reviewed by the Science Advisory Board within this process when it reviews the AFOSR which 

is a directorate in AFRL. 

Navy and Marine Corps Review Process. The Navy differs significantly from the Army and Air 

Force both in management structure of S&T and consequently in reviews. Also, in recent years, 

the introduction of the Future Naval Capabilities (FNC) initiatives has changed both 

management and review processes for the Navy and Marine Corps. S&T is centrally managed at 

the Office of Naval Research for the Navy and Marine Corps. The 6.1 and part of the 6.2 

program are managed by six S&T departments, each employing peer review processes differing 

in detail and documentation. The FNC’s comprise the 6.3 program and the remainder of the 6.2 

program and are matrix managed in ONR. The FNC’s are strongly influenced by the user 

community who are active in its review process. The NRL has its own peer review process for its 

corporate S&T programs. 

DDR&E Review Process. The Technology Area Review and Assessment (TARA) process is 

currently DDR&E’s principal review process. TARA review panels are selected by DDR&E 

from a community of experts selected by the DDR&E staff responsible for a particular area 

review. Because of the broad nature of these reviews the board tends to be selected from 

technology generalists familiar with the DOD S&T program together with some specialists. 

While these area reviews are a week in length they only look in detail at Defense Technology 

Objectives (DTOs) that represent less than 40 percent of the entire program. Programs are 

selected as DTOs based on joint service participation and are most often 6.3 or late 6.2 in nature. 

While technology objectives are discussed in the TARA the level of detail is rarely at the CEST 

level. 

The TARA board is also provided a broad but brief overview of the remainder of the program at 

these system level reviews. 

Congressional Staff Reviews. As a part of the budgetary process, congressional staff members 

from the House and Senate Armed Service Authorization and Appropriations Committees review 

the service programs at the budgetary Program Element level. While these reviews are 

comprehensive they do not represent an EWA. Since the nature of these reviews is for funding 

justification, the services use these reviews to sell their programs, and problem areas are 

underplayed. Congress is left to make funding decisions without the context of the overall needs, 

opportunities, and priorities of the S&T enterprise. An EWA could contribute to better 

congressional understanding of the S&T program and, hopefully, to better funding decisions. 

Current Review Process Strengths and Shortcomings 

With the exception of the TARA review, the current multiple review processes are designed to 

meet individual service needs and management objectives. The principal strengths of the current 

reviews in practice are that the services take ownership of them, do not feel threatened by 
15

external oversight (hence, are more open in providing information), and are more inclined to act 

on internally generated recommendations and criticisms than on those recommended by 

DDR&E. The service reviews at the Army and Air Force Directorate or ONR levels are 

generally intensive and employ peer reviewers. An assumption inherent in this report is that, at 

the service performer level, the perception is that there is an adequate review process for their 

needs, and additional reviews would be a burden with little benefit to them. 

Despite these strengths, there are still serious shortcomings in the current review process. A 

significant shortcoming is that there is no EWA of the DOD S&T program. The DDR&E TARA 

process is the closest attempt but is incapable of examining the enterprise at a granularity 

sufficient to understand the program at the work unit level, nor is it comprehensive. Since the 

TARA process uses the conventional review method of a review board convened for several days 

of review, the overall size of the S&T program is too large to address conventionally, even 

looking at only half of the program every year. The lack of an EWA means that overall 

judgments of the program become qualitative, difficult to put into perspective, and subject to 

criticism. For example, there is limited ability to ensure jointness between related efforts, and it 

is difficult to compare work done by DOD to that of other agencies, such as DOE and NASA ,or 

to other performers both national and international. 

Another major shortfall is that the portfolio of current reviews is conducted at many different 

levels of granularity with nothing approaching a common format, even within individual 

services. Because of this, the individual reviews cannot be coherently combined into a 

comprehensive picture of the enterprise. Another shortfall is that the documentation of service 

reviews is not generally available across DOD, further limiting an enterprise-wide picture. 

Finally, current reviews have made little use of information technology tools that could reduce 

travel, speed the process, correlate material, and review results. 

Conclusions Regarding Conventional Reviews 

The desire to introduce an overall S&T assessment of the S&T enterprise at a level of granularity 

sufficient to justify and manage the program raises a conundrum, the so-called S&T uncertainty 

principle. On the one hand is the expectation that an enterprise-wide assessment would 

strengthen the program, explain the quality and relevance of the work, and support justification 

of the large DOD investment in S&T. On the other hand, the work load of such an assessment 

using conventional review techniques would create a massive overload of work on the performer 

base. It would be strongly resisted and would likely prove to be untenable. The search for a 

resolution of this conundrum forms the basis of this study. 

While extensive review processes exist, the sum of these reviews does not constitute an EWA. 

Differences in granularity, comprehensiveness, review-board composition, format, 

documentation and even the review objectives of current reviews do not result in a homogeneous 

understanding of the program. In addition, critical review material is generally not released by 

the services. An EWA of the S&T base would be of significant value to the DOD. A searchable 

database coupled with the powerful search engines already in use and under further refinement 

might offer a powerful new tool to DOD for program assessment while adding little to the work 
16

load of the S&T workforce. For example, an EWA applied to the major challenge posed by 

Improvised Explosive Devices (IED’s) might not only help fund promising ideas but might also 

eliminate investment in technological projects that experts would rule incapable of addressing 

the issues. A key tenet of this study is that the coupling of the best review expertise, obtained by 

computer search, to a work project completely within their field of knowledge will result in a 

very rapid and accurate assessment of the work effort. 

An EWA would create a vehicle for DOD to gain access to a “community” of scientific and 

technical experts that will not only contribute to assessing the program but will open DOD to the 

broader S&T done in other agencies, the private sector, and the global community. Even if not 

applied across the entire enterprise (especially in its first applications), this computer based 

review methodology might be of significant value in selecting reviewers and accessing related 

work for a more conventional board review. In fact, this application would be a good first test of 

its utility. 
17

4. An Approach to Enterprise-wide Assessment 

Background

The previous section suggests that it is unrealistic to approach an EWA of the total DOD S&T 

Program using the conventional review process. The scope of such an undertaking would be 

unmanageable and, if done, the volume of the resulting reports would be so large as to have little 

management value. A non-conventional approach must, therefore, be considered. Any 

assessment of the program must have available to it a database that documents the program at a 

level of detail, a level of completeness and in a format that is compatible with the desired 

assessment. Therefore, establishing the required documentation of the program will be necessary 

to any EWA. An assessment will also involve access to individuals who are experts in the topics 

being assessed. A means of identifying these experts must be in place before an assessment can 

proceed. The assessment process itself must be defined and put in place. Finally a means of 

displaying the data generated by the assessment must be developed so that understanding and 

actions can follow from the assessment. The following paragraphs will address each of these 

requirements. 

Program Documentation 

To come to grips with what documentation is necessary for an EWA, it is helpful to consider 

how S&T is actually done. In this regard, there often is confusion between S&T and capability. 

This results in a tendency to document the program in terms of desired capability rather than in 

terms of the details of the S&T that is actually being conducted. As an illustration of this point, 

consider that hypersonic flight is often viewed as a technology. However, there are many 

different sciences and technologies that contribute to hypersonic flight. Hypersonic flight is a 

capability rather than a technology. In general, scientists and engineers each work on a small part 

of the S&T needed to achieve the capability of hypersonic flight. The array of different scientific 

and technical disciplines needed to support a capability like hypersonic flight is quite large. The 

viability of achieving a desired capability will depend on the viability of the S&T programs 

aimed at resolving the CEST issues that prevent one from just getting on with building the 

desired capability. 

It should be clear from the above that properly defining, identifying, and documenting the CEST 

issues must be of high priority in preparing for an EWA. It is, therefore, worth spending some 

time addressing the matter of how CESTs should be defined and documented. For this purpose, 

we will continue with the example of hypersonic flight. We have chosen this example because it 

was of special interest to the DDR&E at the time the study was commissioned, it was recently 

examined by the NRC [“Evaluation of the National Aerospace Initiative,” National Academies 

Press, 2004] and it illustrates the complexity of properly defining CESTs such that they are 

useful in the EWA context. There are various levels at which one could identify the CESTs 

associated with hypersonic flight. For example, as mentioned above, one could define the CEST 

as “hypersonic technology.” If one chooses this definition, then it is straightforward to employ 

various readily available S&T databases and search engines to gain information (see figure 4.1 
18

and tables 4.1 through 4.3). The searches were done using the Engineering Compendex (Ei 

Compendex) database (http://www.ei.org/compendex.html). Articles related to hypersonic 

technology were identified from the Ei Compendex database for the years 1991 to 2004. The Ei 

Compendex provides abstracted information from significant engineering and technical literature 

from around the world. In addition to 4,500 journals, government reports, and books, the 

database holds over 480,000 records from the published proceedings of selected conferences. A 

more complete search could be done, e.g., by adding a Science Citation Index 

(http://scientific.thomson.com/products/sci/) search. However, for the purpose of this discussion 

the Ei Compendex search is adequate. 

From the unfiltered query results, conference papers were separated from the total number of 

articles. Several ranking functions were used to identify the most prolific authors, journals, 

institutions, and countries as measured by number of journal articles. To identify overall trends, 

the numbers of journal articles per year in this area were examined. A ranked list of authors and 

institutions was created from the residual list of conference articles and papers, as well as the 

number of conference papers written per country and per year for comparison with the trends in 

the journal papers. The unfiltered query returned 6,126 results (including journal articles and 

papers, conference articles and papers, conference proceedings, books, monographs, and 

dissertations,). Of the 6,126 total, 4,805 were journal articles, and about 1,200 were conference 

papers. The remaining 120 were a mix of conference proceedings, books, monographs, 

dissertations, etc. The time history is shown in figure 4.1. 

Figure 4.1 Comparison of Journal and 

Conference Articles Related to Hypersonics 

(Source: Ei Compendex, 1991-2003)

0

20

40

60

80

100

120

140

160

180

200

220

240

260

280

300

320

340

360

380

400

420

440

460

480

500

1991199319951997199920012003

Year

Number of Articles

JOURNAL

ARTICLES

CONFERENCE

ARTICLES &

PAPERS

Table 4.1 ranks the number of papers published by country. 
19

TABLE 4.1: NUMBER OF PAPERS PER COUNTRY

Source: Ei Compendex, 1991-2004

POLAND 8

SPAIN 9

SWITZERLAND 11

GREECE 11

BRAZIL 13

NETHERLANDS 20

AUSTRALIA 21

SOUTH KOREA 53

ISRAEL 55

CANADA 59

FRANCE 61

ITALY 79

INDIA 86

TAIWAN 108

GERMANY 175

ENGLAND 200

RUSSIA 205

CHINA 347

JAPAN 406

USA 1890

COUNTRY PAPERS

The five most published countries included the United States, Japan, China, Russia, and England 

with the United States having a substantial lead. This figure shows that the number of journal 

articles published decreased from approximately 460 in 1994 to about 270 in 1998, and then 

resurged to about 430 in 2003. This resurgence correlates with the establishment of the National 

Aerospace Initiative in 2001. The average number of journal papers from 1991–2003 is about 

362. The average number of conference papers per year is about 89. 
20

Table 4.2 orders the 24 most published organizations. 

TABLE 4.2: TOP 24 INSTITUTIONS

Source: Ei Compendex, 1991-2004

UNITED TECHNOLOGIES RESEARCH CENT USA 15

UNIV OF CINCINNATI USA 16

UNIV OF CAMBRIDGE USA 17

IOWA STATE UNIV USA 17

FLORIDA STATE UNIV USA 17

SANDIA NATL LAB, ALBUQUERQUE USA 18

NAGOYA UNIV JAP 18

KYUSHU UNIV JAP 18

NATL AEROSPACE LAB, TOKYO JAP 19

PENN STATE UNIV USA 20

UNIV OF FLORIDA USA 22

PRINCETON UNIV USA 24

UNIV OF TEXAS AT AUSTIN USA 25

NORTHWESTERN POLYTECHNICAL UNIV, CHINA USA 25

STANFORD UNIV USA 27

NORTH CAROLINA STATE UNIV USA 27

NASA LEWIS RESEARCH CENT, USA 27

VIRGINIA POLYTECHNIC INST USA 30

UNIV OF MARYLAND USA 31

BEIJING UNIV OF AERONAUTICS AND ASTRONAUTICS CHINA 34

OLD DOMINION UNIV USA 36

TOHOKU UNIV, JPN JAP 40

NASA AMES RESEARCH CENT USA 59

NASA LANGLEY RESEARCH CENT USA 158

INSTITUTION COUNTRY PAPERS

The U.S. Federal laboratories represented include three NASA centers and the Sandia National 

Laboratories. NASA Langley clearly has been the dominant force in this particular area. No 

DOD laboratories appear in the “Top 24 Institutions.” The other U.S. institutions are all 

universities, except for the United Technologies Research Center. 
21

Table 4.3 orders the top 20 journals. The top five journals listed include the AIAA Journal, the 

Journal of Spacecraft and Rockets, the Journal of Propulsion and Power, the Journal of Aircraft, 

and the Journal of Thermophysics and Heat Transfer. All of these are published by the AIAA. 

TABLE 4.3: TOP 20 JOURNALS

Source: Ei Compendex, 1991-2004

AERONAUTICAL JOURNAL 49

TEPLOFIZ VYS TEMP 52

JOURNAL OF SOUND AND VIBRATION 53

TRANS JPN SOC AERONAUT SPACE SCI 55

TEPLOFIZIKA VYSOKIKH TEMPERATUR 55

TRANSACTIONS OF THE JAPAN SOCIETY FOR AERONAUT 56

J GUID CONTROL DYN 65

INTERNATIONAL JOURNAL FOR NUMERICAL METHODS IN 67

COMPUT FLUIDS 70

IZV AN SSSR MEKH ZHIDK GAZA 81

JOURNAL OF FLUID MECHANICS 99

TUIJIN JISHU/JOURNAL OF PROPULSION TECHNOLOGY 108

NIPPON KIKAI GAKKAI RONBUNSHU, B HEN/TRANSACTI 132

JOURNAL OF THERMOPHYSICS HEAT TRANSFER 164

JOURNAL OF AIRCRAFT 219

JOURNAL OF PROPULSION AND POWER 355

JOURNAL OF SPACECRAFT AND ROCKETS 404

AIAA Journal 644

JOURNAL PAPERS

Figure 4.1 and tables 4.1 through 4.3 provide information regarding worldwide effort in the area 

of hypersonic technology and identify the centers of activity in the United States (at least in so 

far as they participate in the open literature). This material would be useful to and provides 
22

necessary background for an EWA. However, it is far too general in character to form the basis 

of an EWA. The reason for this relates to the large number of distinctly different scientific and 

technical disciplines encompassed by the descriptor “Hypersonic Technology.” If one attempted 

an assessment it would be extraordinarily difficult to identify and assemble a group of experts 

who were qualified to collectively provide a substantive assessment of the area. For example, the 

NRC study on the NAI noted that hypersonic technology involves several critical technologies. 

These include: 

• Propulsion 

• Thermal environment prediction, protection and management 

• Integrated airframe structures and cryogenic tanks 

• Vehicle design, optimization and simulation 

We will refer to the above as Level 2 descriptors of CESTs and “hypersonic technology” as a 

Level 1 descriptor. Further examination demonstrates that each of these Level 2 critical 

technologies subdivides into additional critical technologies. For example, Bowcutt 

(“Hypersonic Technology Status and Roadmap”, Presentation to AIAA HyTASP Program 

Committee, Dec. 18, 2003, http://www.aiaa.org/Participate/Uploads/TFAB%20HyTASP.pdf) 

showed that the Level 2 “Propulsion” CEST subdivides into Level 3 CESTs as follows: 

• Engine materials 

• Cooled engine panels 

• Scramjet combustors 

• Fuel injectors/Flame holders 

• Integrated flow-path-hydrogen 

• Integrated flow-path-hydrocarbon 

• Engine seals 

• Engine active control 

• Turbine-Based Combined Cycle (TBCC) hypersonic engine flow-path integration 

One can further subdivide the above list of Level 3 descriptors into Level 4 CEST issues, and so 

forth. This illustrates the difficulty of conducting a conventional assessment of the total DOD 

S&T program. Each of the subdivisions involves different disciplines and different communities 

of expertise. The number of experts that one would need to collect together for a conventional 

assessment is just unreasonable. As discussed in the previous section, this proliferation of 

disciplines presents a difficult problem even for a focused review of a small subset of the DOD 

S&T program. It presents an insurmountable problem for a conventional review approach to an 

EWA of the entire DOD S&T program. However, the next level (Level 4) subdivision 

approaches the level at which scientists and engineers actually work. This may present an 

opportunity for a non-conventional approach by exploitation of the fact that Level 4 approaches 

the point where the various subject matter experts can usually quickly make informed judgments 

regarding the state of the art and the prognosis for progress in S&T with which they are very 

familiar. This, however, places significant constraints on the documentation of the program. To 

explore this point we will examine the Level 4 detail for two of the CESTs identified by 
23

Bowcutt. In particular, we will discuss in some detail “Turbine-Based Combined Cycle (TBCC) 

hypersonic engine flow-path” and in lesser detail “Engine materials”. 

Future hypersonic air-breathing UCAVs for time-critical theater operations, and reusable launch 

vehicles for prompt global response and routine space access, will require 2–3 propulsion modes 

to operate across their design speed ranges. One of the most likely propulsion systems for such 

applications will be the TBCC propulsion plant, which consists of a supersonic (~Mach 4) 

turbine engine closely integrated with a dual-mode ramjet/scramjet engine. The study authors 

worked with Kevin Bowcutt of the Boeing Company to construct a hypothetical (but technically 

realistic) program that would address the key issues associated with the TBCC. It was found that 

it will be necessary to develop a TBCC engine inlet system that efficiently delivers air flow 

required by the turbine engine when operating alone, air flow required by the scramjet when 

operating alone and to both engines simultaneously during the transition from turbine to 

scramjet. It is also necessary to maintain inlet fluid dynamic stability during mode transition and 

in all other operational engine modes. One must develop a TBCC engine nozzle system that 

efficiently expands flow to produce thrust for the turbine engine when operating alone, for the 

scramjet engine when operating alone, and for both engines when operating simultaneously 

during mode transition. It is necessary to develop an aircraft and engine flowpath control system 

that adequately stabilizes and controls the aircraft and TBCC engine flowpaths during engine 

mode transition and all other flight phases. Lastly, it will be necessary to develop the material, 

structural, and mechanical design for the articulated TBCC inlet and nozzle systems, and a 

thermal management system for the turbine engine when it is shutdown but exposed to scramjet 

flowpath heating. In the following paragraphs we will outline the resultant hypothetical program 

aimed at achieving these objectives. One of the goals here is to illustrate how one might 

articulate the CESTs for a program of this nature and how one might document the program so 

as to be useful to an EWA. 

A reasonable approach to the above objectives would be to define TBCC inlet and nozzle 

geometry and articulation requirements based initially upon knowledge of existing supersonic 

inlets and nozzles, empirical and theoretical design guidelines and operability (i.e., stability) 

criteria, engine performance characteristics, and analysis results derived from engineering codes 

and computational fluid dynamics (CFD). This would be followed by building sub-scale inlet 

and nozzle models and testing them in wind tunnels, both statically and with dynamic 

articulation of the variable geometry components. One could then use the knowledge gained 

from testing and analysis to optimize inlet and nozzle designs. This would be aided by 

mathematical optimization and multidisciplinary design optimization techniques. 

It would also be necessary to develop inlet and nozzle control algorithms, and then test them via 

dynamic simulation using data derived from testing and CFD analysis. Vehicle stability and 

control during engine mode transition would be verified via dynamic simulation using CFDderived aero data in conjunction with inlet and nozzle test data. Wind tunnel tests would be 

conducted of an aircraft model with variable inlet to verify inlet stability during mode transition, 

and aero data would be gathered to verify vehicle stability and control during the transition 

event. 
24

The program would design the structure and mechanisms, including high-temperature seals, for a 

full-scale TBCC inlet and nozzle, and then bench test critical inlet and nozzle components to 

verify predicted operation and performance. It would also design a thermal management system 

(TMS) for the turbine engine flowpath and analyze it using fluid, thermal, and structural analysis 

codes, and then bench test critical TMS components to verify predicted operation and 

performance. 

The program outlined above involves several Level 4 CESTs that must be resolved for the total 

program to be successful. The prospects for conducting an EWA would be enhanced if the 

CESTs could be articulated such that experts in each CEST area could quickly comment on the 

likelihood of the CESTs being advanced in accordance with the stated program. It is suggested 

that the following articulation of the CESTs would suffice. 

1) Hypersonic engine seals for articulated (i.e., variable geometry) inlet and nozzle structural 

elements. 

 

2) A dual-flow-path TBCC hypersonic engine inlet that compresses and feeds air to the turbine 

and scramjet engines, both separately and concurrently. This entails complex variable geometry, 

high compression efficiency and stable operation for both the turbine and scramjet flow-paths 

operating separately and simultaneously. 

 

3) A dual-flow-path hypersonic engine nozzle that expands exhaust gases from the turbine and 

scramjet engines, both separately and simultaneously. This entails complex variable geometry of 

hot structures and high expansion efficiency for both engine flow-paths operating separately 

and simultaneously. 

 

4) A post-shutdown hypersonic turbine engine thermal management system (TMS) that 

maintains turbine material temperature limits when the turbine is shut down and the scramjet is 

operating adjacent to the turbine flow-path. 

 

5) A hypersonic engine flow-path control system and associated logic and algorithms (i.e., 

software) that maintain stable engine flow-path operation and thrust performance, and concurrent 

aircraft stability and control during TBCC mode transition from ramjet to scramjet operation, and 

vice versa. 

The program with the CESTs outlined above becomes complete when milestones, metrics, and 

funding are assigned to each CEST. At that point the program can be assessed regarding the 

realism of the milestones, metrics, and funding. Continuing with the above hypothetical example, 

we propose the milestones, metrics and funding for each CEST illustrated below. We employ a 

format similar to that used by the NASA Technology Inventory (see Appendix C), and have 

combined CESTs 2 and 3.
25

TABLE 4.4 CEST 1: Hypersonic engine seals (U)

Technology Metrics:

1000’s for <2000o F

100’s at 2500o F

10’s at 3000o F

1000’s for metallic

10 – 100 for 

ceramic

# of cycles before 

replacement

Durability/life

1800o F (metallic)

2500-3000o F (ceramic)

1400o F (metallic)

18000 F (ceramic)

Temperature o Maximum use F

temperature

Metric Metric Unit SOA Planned Value

Milestones:

Design and fabricate seals and 

complete critical component testing 

in a relevant environment

6 2008

TRL Year Description

*Current TRL = 2 – 3

Funding:

Funding $1200K $1500K $1800K $1500K

Profile

Fiscal Year 2005 2006 2007 2008

Technical Work Years

Technical Work 3 4 4 3

Years

Fiscal Year 2005 2006 2007 2008

Suggested Experts:

NASA Glenn 

Research Center

NASA Glenn 

Research Center

Last Name* First Name* Organization Email Phone

*names deleted
26

Table 4.5 CEST 2: Dual flow-path TBCC hypersonic engine inlets and 

nozzles (U)

Technology Metrics:

2 at SOA thrust 

coefficient

# of nozzle operating 1

simultaneously at high 

performance

Integrated dual nozzle 

operability

2 at SOA inlet recovery 

and operability/stability

# of inlets operating 1

simultaneously at high 

performance

Integrated dual inlet 

operability

Metric Metric Unit SOA Planned Value

Milestones:

Update inlet and nozzle designs, and 

tunnel test inlet on a sub-scale aircraft 

model with an articulated inlet

5 2008

Wind tunnel test sub-scale TBCC inlet 

and nozzle

4 2007

TBCC inlet and nozzle design and 

analysis

3 2006

Year Description

TRL

*Current TRL = 2 – 3

Funding:

Funding Profile 0 $3000K $4000K $3000K

Fiscal Year 2005 2006 2007 2008

Technical Work Years

Technical Work Years 0 8 8 8

Fiscal Year 2005 2006 2007 2008

Suggested Experts:

Email Phone

NASA Langley (Retired)

TechLand Research, Inc. 

NASA Langley (Retired)

AF Research Laboratory/Propulsion Directorate

Johns Hopkins Applied Physics Laboratory

Last Name* First Name* Organization

*names deleted
27

Table 4.6 . CEST 3: High-speed turbine engine thermal management system (TMS); for turbine 

engine during operation and when shutdown/cocooned during scramjet operation

Technology Metrics:

No impact to engine thrust-to-weight via highly 

integrated turbine and scramjet TMS 

TMS system Weight

weight

Maximum of 350-600o F; optimum value must be 

determined by trade study (limited by fuel coking, 

lubricants, seals, engine structure, electronics)

160-200 o

F

Maximum turbine Temperature

engine soak 

temperature

Metric Metric Unit SOA Planned Value

Milestones:

4 2008 Critical TMS component bench testing and analysis update 

3 2007 TMS design and analysis complete

Conduct trade study to establish optimum turbine engine soak temperature, and 

create high-temperature engine component technology development roadmap (which 

become additional TBCC critical enabling technologies) 

2 2006

TRL Year Description

*Current TRL = 2-3

Funding:

Funding Profile N/A $1000K $2000K $3000K

Fiscal Year 2005 2006 2007 2008

Technical Work Years

Technical Work Years 0 3 4 5

Fiscal Year 2005 2006 2007 20080

Pratt and Whitney, retired

Air Force Research Lab

Pratt and Whitney

Pratt and Whitney

Pratt and Whitney

Last Name* First Name* Organization Email Phone

Suggested Experts

*names deleted
28

Table 4.7. CEST 4: Hypersonic Engine Control System Development

Technology Metrics:

6 db gain and 45o

phase margins

Control system phase N/A

and gain

Aerodynamic 

dynamic stability 

during engine mode 

transition

Inlet stability during Stability margin N/A 5%

engine mode 

transition

Metric Metric Unit SOA Planned Value

Milestones:

Use inlet control laws in wind tunnel test of integrated inlet/airframe. Use 

tunnel data to update control laws and simulations.

4 2008

Develop inlet, nozzle and aircraft control laws using aerodynamic propulsion 

analysis of same. Perform dynamic simulations.

3 2007

TRL Year Description

*Current TRL=2.3

Funding

Technical Work Years 3 3 3 4

Fiscal Year 2005 2006 2007 2008

Technical Work Years

Funding $1000k $1500k $1500k $2000k

Fiscal Year 2005 2006 2007 2008

Suggested Experts:

Boeing, St. Louis

Air Force Research Laboratory

NASA, Dryden

Boeing, Huntington Beach

Last Name* First Name* Organization Email Phone

*names deleted

The above examples suggest a documentation format for a DOD S&T work unit. The basic 

theme behind the format is the careful identification by the principal investigators of all the 

CESTs that the work unit is funded to address. The active involvement of the principal 

investigators in preparing the proper documentation is essential. They are the individuals who 
29

best know the program. In this regard it should be noted that the above examples have included a 

list of recommended assessors. The principal investigator should provide such a list as part of the 

program documentation. An outline of the suggested format is presented and table 4.8. If the 

entire DOD S&T program were documented as shown in table 4.8, the resulting database might 

form the basis for an EWA. The logical custodian for this database is DTIC. For purpose of 

illustration, Appendix A provides a completed version of table 4.8 for the hypersonic example 

summarized in tables 4.4 – 4.7. 

Work Unit Summary Form XX-XXXX 

1. Title: 2. Date of Summary: 

3. Responsible Organization 

a. Name: b. Address: 

c. Principle Investigator: 

Name Phone Email Address 

 

4. Classification Level: 

a: Summary Level: b: Work Level: 

5. Military need: 

6. Technical Objective: 

7. Progress: 

8. Critical Enabling Science and Technology (CEST) 

a: CEST 1: 

Approach 

Technology Metrics 

Metric Metric Unit SOA Planned Value 

 

Milestones 

TRL Year Description 

 

 

*Current TRL = 

Page 1 

Funding

Funding 

Profile

2005 2006 2007 2008 2009

Fiscal Year

Technical Work Years

Technical 

Work Years

Fiscal Year

9. Suggested Experts:

Last Name First Name Organization Email Phone

Click here to add an

additional CEST

Click here to add an

additional CEST

10. Total Funding:

Total 

Funding

Fiscal Year

11. Program Participants:

Government organization

Grants with whom

Contracts with whom

12. DoD Subject Areas: http://www.dtic.mil/dtic/subcatguide/

13. Key Words: 

Page 2

Table 4.8 : Suggested Work Unit documentation format

Table 4.8 has been streamlined so as to highlight those elements that are most needed for an 

EWA. The table would ultimately incorporate additional fields (e.g., program element numbers) 

to allow traceability to appropriations, etc. The database that would result would be useful 
30

independent of an EWA. The NASA Technology Inventory (NTI) database provides a good 

illustration of such utility. This point is discussed in Appendix C, which provides a brief 

description of the NTI. It also provides several displays that this study was able to easily 

construct from the online NTI. These displays are intended to show that the existence of this 

database and its functionality allow authorized users to inquire about the NASA Technical 

Program from the highest level of aggregation down to the individual work units with the click 

of a mouse. This functionality could and should be available to DOD decision makers and 

performers regarding the DOD S&T program. 

Establishing Communities of Experts 

The following paragraphs will outline an approach to identifying the needed communities of 

experts. While the establishment of these communities of experts is a critical aspect of the EWA 

methodology discussed in this report, it is also in the best interests of DOD independent of an 

EWA. Science and technology will continue a trend towards increased sophistication and 

complexity. Keeping current in these developments will become increasingly difficult. This will 

be exacerbated by the inevitable trend toward globalization of S&T. In order to cope with these 

developments, DOD must reach out in a constructive fashion to engage the larger U.S. scientific 

and technical communities regarding S&T for national security. This must be done within the 

context of a national purpose rather than through some competitive free market process. It is 

unfortunate that the constructive engagement between the DOD and the U.S. scientific 

community has greatly diminished over the past thirty years. This situation needs to be rectified. 

Most American scientists and engineers are well aware that the nation’s long-term strength 

(military and economic) depends upon the federal government’s stewardship of the nation’s long 

term S&T investments. Many of these scientists and engineers would be honored to be 

considered members of a community of experts that is contributing to this national purpose. A 

public policy to accomplish this should be established, and a public relations program should be 

mounted to articulate this policy to the nation’s scientific and technical communities. If 

successfully done it would be therapeutic for the S&T community and would provide the type of 

access to this community that can be achieved only by tapping into people’s commitment to the 

nation’s health and wellbeing. Such an undertaking would be a strong motivator for individual 

scientists and engineers to participate in the EWA discussed below. 

Tables 4.4 through 4.7 reduce the program to a succinct statement of CESTs, metrics, 

milestones, funding, and technical work years. A database of such tables for the entire DOD 

S&T program might form the basis for an EWA. However, in order to do this the program 

documentation must enable the finding of appropriate assessors. One would like to identify a 

large number of assessors in order to gain adequate scientific and technical coverage to avoid 

some of the “small sample” problems that often occur in the conventional review process. The 

CESTs may provide the key to doing this. For example, if the CESTs are properly articulated 

then, by employing available databases and search engines, the tables might form the basis for 

identifying a pool of active scientists and engineers from which a community of experts could be 

formed by. It is important to understand that each CEST will have its own community of experts. 

We will apply this line of reasoning to the CESTs identified above. The first was hypersonic 

engine seals for articulated inlet and nozzle structural elements. The broad search of the field of 

hypersonic technology reported in table 4.3 indicates that the AIAA journals have published the 
31

largest number of papers in the area of hypersonic technology. It would, therefore, seem 

reasonable to query the AIAA Electronic Library (http://www.aiaa.org/content.cfm?pageid=406) 

regarding publications in the area of hypersonic engine seals in order to identify individuals who 

publish in this area. A query on “hypersonic engine seals” for the period 1995 through 2005 

found 25 records that identified 20 U.S. authors along with their affiliated institutions. The same 

query of the Google search engine identified a number of sites including one that provides the 

proceedings of the “2002 NASA Seal/ Secondary Air System Workshop.” A simple search of the 

proceedings table of contents identified 17 additional U.S. authors along with their institutions. A 

more sophisticated search using the Google Scholar search engine with the same query for the 

same timeframe yielded 103 U.S. authors along with their affiliations. It is interesting to note that 

the Google Scholar search identified papers in the AIAA Electronic library that were not 

identified by the direct search of the AIAA Electronic Library. Also, the direct search of the 

AIAA Electronic library found some papers not identified by Google Scholar. The total number 

of individuals identified who had U.S. affiliations was 129. A complete listing of these 

individual’s organization will be found in Appendix B, table B1. The names of the individuals 

are not listed for privacy reasons. This group could be considered to be a potential community of 

experts for the CEST “Hypersonic engine seals.” The time that it took to generate the input for 

the entire listing of the potential community of experts for hypersonic engine seals was less than 

a second (plus perhaps another 15 minutes of manual work to eliminate individuals who were 

clearly inappropriate). The search engines used above did not offer the option of automatically 

extracting all the authors and their affiliations. This had to be accomplished manually and was 

unacceptably time consuming for an EWA. However, since the author names and affiliations are 

machine readable, it should be straightforward to automate this extraction process. Once this is 

done, arriving at the potential community of experts for each CEST could be totally automated 

and performed in a matter of seconds. 

A large number of search engines are available for scientific and technical inquiries. Some can 

be utilized at no cost, while others can be quite costly. The latter search engines are best suited 

when a very complete analysis of an area is desired. However, the establishment of a potential 

community of experts in an area does not require a complete analysis of all work that has been 

done in that area. It requires only that a large enough sample be found of individuals who have 

published recently in the area in question. In the cases examined in this study it was found that 

the free search engines, such as Google Scholar, provided satisfactory potential communities of 

experts. It is expected that, in the coming years, these free search engines will improve 

considerably and that the ability to find satisfactory potential communities of experts rapidly and 

at no cost will improve accordingly. 

We have used the term “potential community of experts” because the listing in table B1 contains 

individuals who will be well matched to assessing the program outlined in table 4.4 and others 

who will not be well matched. One could narrow down the community of experts by several 

methods. At one extreme one could proceed by asking the principal investigator for a particular 

CEST to identify from table B1 those individuals best suited to perform an assessment (this 

request could be automatically prompted upon the PI providing documentation of the program). 

At the other extreme one could interrogate databases such as the Community of Science 

Expertise database (http://expertise.cos.com/about/expertise.shtml) for each individual listed in 

table B1 in order to find the best matches for the desired assessment. At the current state of 
32

technology this latter approach likely would involve an unacceptable level of human 

intervention. An intermediate approach would be to have the principal investigator responsible 

for a particular CEST identify, in his/her formal documentation, a number of individuals 

believed to be appropriate assessors for the CEST (the “suggested community of experts” shown 

in tables 4.4 through 4.7) and have the principal investigator, prior to submitting the 

documentation, gain their agreement to participate in an assessment. One could then 

automatically provide these individuals a list such as shown in table B1 (with the names 

included) and ask them to identify individuals from that list whom they believe are well suited to 

conduct an assessment. This approach has the benefit of putting some distance between the PI 

and the assessors and having the assessors selected by experts. Another approach would be to use 

the full list given in table B1 and have the members of the potential community of experts decide 

as individuals whether or not they are appropriate to participate in the assessment. 

The same approach used to construct table B1 was used to identify potential communities of 

experts for the other CESTs identified in tables 4.5 through 4.7. In these cases the searches were 

limited to Google Scholar. In case of the CEST “Dual flow path TBCC engine inlets and 

nozzles,” a potential community of experts of 28 individuals was found. For the CEST “Post 

shutdown turbine engine thermal management System (TMS)” a potential community of experts 

consisting of 70 individuals was found. For the CEST “Hypersonic Engine Control System 

Development” a potential community of experts consisting of 134 individuals was found. The 

names and affiliations of the potential communities of experts will be found in Appendix B, 

tables B2 through B4. In each of the above cases a potential community of experts was found 

that is large enough to suggest that meaningful statistics could emerge if a reasonable fraction 

(say 50 percent) of the potential community of experts agrees to participate in the assessment. In 

general, a lower limit of about 10 would need to be placed on the size of the final community of 

experts in order to obtain meaningful statistics. 

The turbine based combined cycle (TBBC) engine flow path example discussed above is typical 

in character of many of DOD programs funded at the advanced 6.2 and 6.3 stage. Such programs 

often have a systems character to them. Our study finds that, if the CESTs are properly specified, 

it is straightforward to find a potential community of experts for the various CESTs. The 

searches we conducted produced 10s to 100 or more potential members for the community of 

experts associated with a particular CEST. These searches were all performed in less than one 

second of search time. The DOD S&T program, however, also supports a substantial amount of 

work that can be generic in character and is often funded at the 6.1 and early 6.2 levels. For 

example, work on materials as often in this category. This work may investigate materials 

properties with the objective of identifying material systems that might be useful in some of the 

stressful environments encountered by DOD systems. This study has found that the basic 

approach outlined for the TBBC example is applicable to the more generic S&T. The TRL 

assignment is not particularly useful here because, for these programs, it usually stays in 1-2 

range for the program duration. However, the idea of identifying the critical S&T issues being 

addressed by these programs remains valid. The concepts of the state of the art and planned 

advances in the state of the art remain valid, as do the concepts of milestones and communities of 

experts. For these programs, the importance of clearly identifying the CEST is also found to be 

key. In order to demonstrate some of these points we will consider briefly the topic of hypersonic 

engine materials. Furthermore, to illustrate the variety of search engines that are readily 
33

available, we will use results obtained from the SCIRUS search engine 

(http://www.scirus.com/srsapp/). Because of the high temperatures encountered in hypersonic 

propulsion, ceramic matrix composites have been of great interest. If we declare “ceramic matrix 

composites” as a CEST then, for the period 1995–2005, SCIRUS identifies 1635 records for a 

full document search. An examination of these records shows that the description “ceramic 

matrix composites” covers a large number quite different material systems and investigations. It 

would be difficult to easily extract a viable potential community of experts from this broad 

listing. Therefore, “ceramic matrix composites” is a Level 1 designation and is not a satisfactory 

CEST. One must inquire at a more specific level. A significant subfield of ceramic matrix 

composites is that of fiber reinforced ceramic matrix composites. A full document search at this 

level and returns 602 records. Examination of these records once again demonstrates many 

different material systems and investigations are covered by this level of description. This is a 

Level 2 designation and is not a satisfactory CEST. A scientist or engineer will typically be 

working on a particular material system such as silicon carbide fiber reinforced silicon carbide 

(referred to by the community as SiC/SiC). A title search on “SiC/SiC” for the period 1995–2005 

identified 112 files. While this is approaching the level where an assessment might be done, it is 

actually a level 3 designation. Even within this narrow field, scientists and engineers will be 

working on some particular critical issue, such as the interphase, that transitions the silicon 

carbide fiber to the silicon carbide matrix. A title search on “SiC/SiC interphase” returns 39 

records. This is a Level 4 designation. If one undertakes to identify a community of experts at 

this level, then one should find individuals who could quickly and authoritatively comment on a 

DOD S&T program that is conducting working in this narrow area. We suggest that this is the 

proper level at which to identify the CEST (e.g. “SiC/SiC interphase” for this example). 

The results from a search to identify a potential community of experts for a particular CEST can 

be very dependent on how the CEST is stated. For example, when the term SiC/SiC in the above 

mentioned title search was replaced with its full name “silicon carbide fiber reinforced silicon 

carbide” then the SCIRUS search returned only 9 records instead of the 112 returned for the 

abbreviated name. The reason for this is that the community that works in this area has embraced 

the designation “SiC/SiC” in their literature. The current search engines are not sophisticated 

enough to figure this out. One must, therefore, not only define the CEST at the proper level but 

must also properly name the CEST. The individuals best qualified to do this identification of the 

CESTs are the principal investigators for the various projects. The principal investigators should 

be required to identify the CESTs as part of the formal documentation of the program. They 

should also be required to conduct searches on their proposed CESTs to verify that their 

definitions identify an adequate number (10–100) of potential members of a community of 

experts for each CEST and that the listing they are finding includes names that they would 

expect including some of those they have identified as “suggested experts.” They will likely need 

to iterate the naming of the CEST several times to get it right. This should not be burdensome, 

because each search typically takes less than a second to accomplish. The proper identification of 

the CESTs must be included in the formal documentation if there is to be any hope of conducting 

an EWA. This is because the exact statement of the CEST will form the basis for identifying the 

individuals who will be asked to perform the assessment. If the CEST is not properly identified, 

then a valid community of experts will not be found. It would be helpful to get feedback from the 

expert assessors regarding whether the CEST was properly identified/stated. 
34

Assessment Process 

In this section the enterprise-wide assessment methodology is described sequentially. Five major 

steps are envisaged including the preparation of Work Unit Summary data sheets, extraction of 

CESTs by DTIC and search for potential reviewers, selection of assessors from the list of 

potential reviewers, program assessment by the review board and analysis and reports of the 

assessments. The two highest risk elements in the execution of this proposed process are the 

formulation of the CESTs in the Work Unit Summaries at the appropriate level and the 

generation of a community of experts using advanced search engine technology. The sequence is 

as follows: 

1. The PI of the responsible organization prepares a Work Unit Summary document (table 4.8) 

and submits it electronically to DTIC where it is entered into the appropriate database. 

A hypothesis of the proposed review process is that the reviewers, based upon detailed expert 

knowledge of the particular CESTs of the work unit, can make rapid and accurate assessments of 

the maturity, impact and feasibility of the work unit. The generation of the CESTs by the 

principal investigator is therefore, the critical element of the Work Unit Summary. If the CESTs 

are properly developed at the level that the work is actually performed, search engine technology 

should be able to retrieve a community of experts from the published literature in that field of 

knowledge. The PI will also suggest experts that will augment the automated search. Finally, as a 

test, the PI will run a search of the CESTs to determine if an appropriate community of experts 

(including some of the suggested experts) is retrieved by the search engine. If not, the CEST 

description must be modified to produce the desired results. 

2. The DTIC automated system extracts the CESTs and conducts an automated literature search 

to identify a potential community of experts for each CEST. 

DTIC is designated as the custodian of the S&T Work Unit Summary database. DTIC will 

receive the data sheets from the investigating agency and electronically inform DDR&E that it 

has received the data. DTIC will then use search engine technology to produce a potential 

community of experts. While this will duplicate to some extent the search done by the PI it will 

not be time intensive and should be free from any conflict of interest. 

3. The suggested experts listed in the Work Unit Summary select the most qualified members 

from the DTIC generated potential community of experts. 

The list of the potential community of experts is automatically made available, via a password 

controlled web site, to the suggested experts listed in the Work Unit Summary. The responsible 

organization will have certified at the time of submission of the Work Unit Summary document 

that the suggested experts are U.S. citizens and have agreed to participate in an assessment. The 

suggested experts place a check mark next to those individuals on the list who they consider 

appropriate to act as assessors for the CESTs in question. This input combined with the 

suggested experts will constitute the communities of experts for the CESTs. 
35

4. The selected expert review members complete the appropriate worksheets and submit them to 

DTIC. 

The selected communities of experts will be provided electronically a survey worksheet (see 

table 4.9) to complete and return to DTIC. It is expected that the expert reviewers will be able to 

perform the assessment in a few minutes using their previous experience, the compact 

assessment worksheet, and the limited information provided to them. 

Table 4.9: S&T Assessment Worksheet Assessor:

Background Information

1.Work Unit Title (include classification of title):

2.Responsible Organization: 

3.Critical enabling S&T (include classification of 

entries):

a.Approach

b.Metrics

c.Milestones

d.Funding

e.Technical work years

Assessment:

On a scale of 1-10 (1 is low, 10 is high) score the 

following:

Adequacy of funding

Likelihood of achieving 

milestones (item 3.c)

Level of maturity of the S&T 

being pursued

Impact if milestones are met

Level of S&T Challenge

Realism of timeframes

CEST Properly Stated

Topic Score Comment

5. The DTIC automated system will collect and analyze the input from the various communities 

of experts, identify areas for in depth review, and prepare summary reports for the appropriate 

DDR&E staff and others (for example the service S&T executives and the organizations 

responsible for the work being assessed). 

If the on-line review indicates the need for an in depth review, the same experts designated for 

the on-line assessment are appropriate members for an in-depth review. An additional spin-off of 

the process described here for generating a community of experts is that this process should be of 

value for selecting reviewers for a conventional review at the detailed technology level. While a 

search engine would pull up too many names for a broader subject area review, it could still be 

of value in selecting subject experts to augment the generalists who are typical of the 

conventional review process.
36

A simplified flow chart of the envisioned process is presented in figure 4.2: 

STEP 1

Responsible organization

submits Work Unit

summary to DTIC

STEP 2

DTIC extracts CESTs and

automatically generates

a Potential Community

of Experts (PCE)

STEP 3

The responsible organization

suggested experts select

most qualified from the PCE 

leading to the final communities

of experts (COEs)

STEP 4

The COEs complete the

appropriate survey worksheets

and submits them to DTIC

STEP 5

DTIC automated system

collects and analyzes the

Results, identifies areas for

further review, and prepares

Summary reports for DDR&E

and others

Figure 4.2 EWA Process Flow Chart

The state of information technology today is not advanced to the point where it would allow full 

automation of the process outlined above. It is, however, surprisingly close to allowing such an 

EWA to be undertaken. There are some minor issues that must be resolved such as those 

associated with extracting individual authors and their affiliations from the archived publications 

and finding their e-mail addresses. More serious issues must also be resolved such as increasing 

the search engine capabilities beyond key word searches to the point where they can reliably 

perform context searches. In this regard the DOD should participate in those efforts that are 

ongoing to resolve such issues. Active DTIC participation in the NIST and DARPA sponsored 

Text REtrieval Conference series (TREC) (http://trec.nist.gov/) is recommended for the purpose 

of benchmarking various approaches to an EWA. In the short term the techniques outlined above 

could be profitably applied to the conventional review process where manual intervention would 

be substituted for those processes that are not easily automated with current technology. This 

would allow the refining of the data documentation requirements. 
37

Displaying Assessment Results 

The assessment methodology outlined above will generate a large amount of data. One must, 

therefore, develop schemes for analyzing the data and displaying it in ways that provide insights 

into the program. The last three topics in table 4.9 are especially interesting in this regard. They 

form axes of the three-dimensional data cube shown in figure 4.3. The data could be displayed as 

three-dimensional scatter plots ranging from that associated with a particular CEST, to a 

combined scatter plot for all CESTs within a given work unit, all the way up to a scatter plot for 

all CESTs in the entire DOD S&T program. These scatter plots would provide a quick sense of 

where in the data cube (likelihood, impact, maturity) the DOD S&T program resides as a 

function of the level at which it is examined. 

 

Figure 4.3 Data Cube 

Three-dimensional scatter plots, while they contain all of the data are difficult to 

interpret/visualize. It is often useful to project the data on to the planes that form the data cube. 

An especially interesting display is the projection of the three-dimensional scatter plot onto the 

Impact-Maturity plane. This plane could be broken into quadrants as shown in figure 4.4. It 

would be quite illuminating to see where in this plane the program would cluster. Clustering in 

the lower left-hand quadrant or the lower right-hand quadrant would likely be viewed as 
38

undesirable. Clustering in the upper left-hand or upper right-hand quadrants are more interesting 

cases. There will be very different opinions regarding in which of these two quadrants the 

program should reside. The discussion that this data would precipitate would be quite valuable. 

The projection of the three-dimensional data onto the other two planes would also provide 

valuable insights regarding the sense of the scientific and technical community relative to the 

program’s health and viability. Impact

Maturity 0 10

10

S&T Assessment

Figure 4.4 Impact-Maturity Plane Template 

The survey results could be used as a basis for requesting in-depth reviews of particular areas. 

For example, programs that fell into the lower two quadrants of figure 4.4 could be automatically 

identified for further examination. It would also be useful to correlate the Maturity scores with 

the “color” (6.1, 6.2, and 6.3) of the money used to fund the program. A similar correlation 

should be done of the maturity levels assigned by the principal investigators and the maturity 

levels assigned by the assessors. In this regard, it would be important that the assessors are not 

privy to the maturity levels assigned by the principal investigators. A low degree of correlation in 

either of the above could form the basis for requesting additional review. 

In order to provide a concrete example of the data displays suggested above, the study conducted 

a simulated assessment of the hypersonic program summarized in Appendix A. The approach 

used was to establish a reasonable estimate of the mean scores (Impact, Maturity and Likelihood) 

that the study directors would expect for each of the CESTs if a detailed assessment were made. 

The simulation then assumed 50 assessors whose scores were distributed normally along each 

axis (Impact, Maturity, Likelihood) about the means with a standard deviation of 1 (on a scale of 

1–10). A small random component was then added to the various scores to avoid the artifact of 

all points clustering along a line in each two dimensional projection. Figure 4.5 shows the 

resultant scatter plots for the CEST “Hypersonic engine seals”. 
39

4.5a Impact / Maturity

0

1

2

3

4

5

6

7

8

9

10

0 1 2 3 4 5 6 7 8 9 10

Maturity

Impact

4.5b Likelihood / Maturity

0

1

2

3

4

5

6

7

8

9

10

0 1 2 3 4 5 6 7 8 9 10

Maturity

Likelihood
40

Figure 4.5 Scatter plots for the simulated assessment of CEST “Hypersonic engine seals” 

From the above figures one sees that the simulated assessment places this CEST in the upper 

left-hand quadrant of the Impact-Maturity and Likelihood-Maturity planes and in the upper righthand quadrant of the Impact-Likelihood plane. The authors would view this as satisfactory for a 

mature 6.2 or early 6.3 task. 

4.5c Impact / Likelihood

0

1

2

3

4

5

6

7

8

9

10

0 1 2 3 4 5 6 7 8 9 10

Likelihood

Impact
41

Figure 4.6 present the scatter plots for the CEST “Thermal Management System”. 

4.6a Impact / Maturity

0

1

2

3

4

5

6

7

8

9

10

0 1 2 3 4 5 6 7 8 9 10

Maturity

Impact

4.6b Likelihood / Maturity

0

1

2

3

4

5

6

7

8

9

10

0 1 2 3 4 5 6 7 8 9 10

Maturity

Likelihood
42

Figure 4.6 Scatter plots for the simulated assessment for CEST “Thermal Management” 

These scatter plots place the program in the upper left quadrants of the Impact-Maturity and 

Impact-Likelihood planes and in the lower left quadrant of the Likelihood-Maturity plane. The 

authors would interpret these results as showing a satisfactory 6.2 – 6.3 task but one with 

considerably more risk than the “hypersonic seals” task. The risk is associated with the technical 

challenges confronting this CEST. It is important to note that reaching this conclusion requires 

reviewing all the scores that are required by table 4.9 and not just the scores displayed in the 

scatter plots shown above. 

The scatter plots (not shown) for the simulated assessment of the “dual flow-path” CEST place 

that CEST in the upper right-hand quadrant for all three 2D projections of the 3D data cube. 

Such an assessment suggests that the program involves technology that has high impact, is 

relatively mature, and is relatively low risk. One would expect such a program to be funded 

primarily with 6.3 funds. 

The scatter plots (not shown) for the simulated assessment of the “control systems” CEST place 

the CEST in the upper left-hand quadrants for the Impact-Maturity projection and the ImpactLikelihood projection, but in the lower left quadrant for the Likelihood-Maturity projection. This 

implies that the program involves considerable risk. The high-risk assessment in this case 

resulted not from the technical challenge confronting this CEST but rather from a concern that 

the funding provided was inadequate for the stated plans and milestones. Here again it should be 

noted that reaching this conclusion involves scores not shown by the data cube. 

4.6c Impact / Likelihood

0

1

2

3

4

5

6

7

8

9

10

0 1 2 3 4 5 6 7 8 9 10

Likelihood

Impact
43

The 2D scatter plots that result from combining the simulated assessments for all four CESTs are 

shown in figure 4.7. 

4.7a Impact / Maturity

0

1

2

3

4

5

6

7

8

9

10

0 1 2 3 4 5 6 7 8 9 10

Maturity

Impact

4.7b Likelihood / Maturity

0

1

2

3

4

5

6

7

8

9

10

0 2 4 6 8 10

Maturity

Likelihood
44

Figure 4.7 Scatter plots for combined scores from the simulated assessment of the Work 

Unit “Hypersonic Turbine-Based Combined Cycle Engine (Mach 4 turbine and Mach 3.5 

to 7 scramjet engine)”

This figure suggests that the program that is summarized in Appendix A is one of high impact 

and moderate maturity and one that involves some considerable risk. As pointed out above, 

understanding this risk requires access to data collected as part of the assessment but not shown 

in figure 4.7. Since the viability of the EWA approach suggested here requires that the entire 

process be automated it will be necessary that simple algorithms be developed to flag concerns. 

For example, a program that is deemed risky because of the level of the technical challenges 

confronting it may very well be a satisfactory program. On the other hand, a program that is 

deemed to be risky because of inadequate funding would likely not be viewed as satisfactory. 

Similarly, a program that is assessed to be low impact and low maturity might be a satisfactory 

program because it involves such new S&T that it is difficult to judge its impact. History has 

demonstrated many times that new S&T that is originally deemed to be not relevant has turned 

out to be very important. On the other hand, a program that is assessed to be low impact and high 

maturity should probably not be part of the DOD S&T program. The development of algorithms 

to sort out such issues would be required as part of an EWA. These algorithms could be based 

upon the type of reasoning given above. Figure 4.8 provides a simple decision template based 

upon the above considerations. 

4.7c Impact / Likelihood

0

1

2

3

4

5

6

7

8

9

10

0 1 2 3 4 5 6 7 8 9 10

Likelihood

Impact
45

Maturity 

10 

10 0 

Unsatisfactory 

Call to attention of 

responsible 

organization 

•OK if maturity centroid 

is less than 3 

•If greater than 3 call to 

attention of responsible 

organization 

•OK if maturity centroid 

is less than 7 

•If greater than 7, call to 

attention of responsible 

organization 

OK 

I 

m

p 

a 

c 

t 

Likelihood 

10 

0 10 

Probably not S&T 

Call to attention of 

responsible 

organization 

Unsatisfactory 

Call to attention of 

responsible 

organization 

•OK if maturity centroid 

is less than 7 

•If greater than 7 

probably not S&T. Call 

to attention of 

responsible 

organization 

•OK if due to technical 

challenge 

•Not OK if due to 

inadequate funding or 

unreasonable time 

frames. Call to attention 

of responsible 

organization 

I 

m 

p 

a 

c 

t 

4.8a Impact / Maturity

4.8b Impact / Likelihood
46

The above discussion covers but a few of the ways that the data from an EWA could be 

displayed and analyzed. For example, there are ten data cubes that could be created (such as a 

cube with the axes: Adequacy of funding, Realism of time frames, and Level of scientific and 

technical challenge). Decision criteria could be established for each of these data cubes. Also, 

evaluation of the first few moments of the rating distributions would be necessary. The matter of 

how to aggregate the data from the individual CEST scores is not trivial, especially as one moves 

from a few CESTs (like the case considered above) to large numbers of CESTs (such as for the 

total program). These issues would best be sorted out by conducting limited testing of the ideas 

presented above. A reasonable place to start would be to incorporate some of the EWA 

considerations into the conventional review process and/or to conduct limited tests of the EWA 

approach on small segments of the program. Such testing would provide guidance as to the 

practicality and value of the EWA that emerged from this study. It would also be an important 

step in gaining acceptance of the approach if it turned out be practical and valuable. 

Maturity 

10 

10 0 

Unsatisfactory 

Call to attention of 

responsible 

organization 

•OK if due to technical 

challenge 

•Not OK if due to 

inadequate funding or 

unreasonable time 

frames. Call to attention 

of responsible 

organization 

•OK if maturity centroid 

is less than 7 

•If greater than 7, 

probably not S&T. Call 

to attention of 

responsible 

organization 

OK 

L 

i 

k 

e 

l 

i 

h

o

o

d 

4.8c Likelihood / Maturity 

Figure 4.8. Decision criteria for assessments at the individual work unit 

level or the individual CEST level 
47

5. Conclusions and Recommendations 

Four major conclusions have emerged from this study: 

• Many different and valuable reviews of the DOD S&T program are currently conducted. 

However, the aggregation of these reviews does not constitute an EWA. There is no 

obvious way to modify the current review structure so as to produce an EWA as it has 

been defined for this study. The underlying problem is the sheer magnitude and breadth 

of the program. A new methodology will be required if one is to institutionalize an EWA. 

• The current documentation of the DOD S&T program does not reflect the state of the art 

of information technology, is marginal for exercising stewardship over the program, and 

is not adequate for the conduct of an EWA. The recent introduction of the DDR&E 

“R&E Portal” will improve this situation. However, the planned documentation of the 

program will remain inadequate for the conduct of an EWA. 

• DOD does not access systematically the breadth of scientific and technical knowledge 

and expertise that is potentially available to it. The establishment of and ready access to 

communities of experts that cover all DOD areas of scientific and technical interest 

would be of great value. 

• The current state of the art in and expected advancements in information technology and 

library science offer the best hope of resolving the above concerns. 

These four conclusions lead to several recommendations: 

• The principal investigators of the various DOD S&T projects should be taught to and 

required to document their projects in terms of the very specific CEST issues that they 

are funded to resolve. These issues should be stated at the level at which people are 

actually performing work rather than in terms of general objectives. 

• S&T project documentation requirements should be standardized across DOD and should 

contain the information needed for an EWA. Funding for a project should not be released 

until the organization responsible for the project submits satisfactory documentation to 

DTIC. The DTIC database that results from proper documentation of the program will be 

broadly valuable across DOD (DDR&E, funding agencies, performers, etc.) independent 

of the EWA. The needed information is available at the performer level and can and 

should be documented as a routine matter. This documentation is required as a matter of 

stewardship. 

• The underlying strategy for an EWA should be to carefully and succinctly state the CEST 

issues being addressed by the funded program so that an expert in the specific topical 

area being pursued can quickly assess the viability of the program’s meeting stated plans 

and milestones. 
48

• National communities of experts should be established for each of the specific CEST 

issues being addressed. The tools of modern information technology and library science 

must be utilized to accomplish this. Some DOD research investment in library science 

may be required to resolve DOD specific issues. The establishment of the communities of 

experts will be of great value independent of the EWA. 

• For an EWA, the approach of collecting the review teams at a central location for the 

purpose of holding hearings must be abandoned and replaced by a distributed assessment 

process that exploits the documentation of all of the CEST issues, the advances in 

information technology and library science, and the readily available access to 

geographically distributed communities of experts. The EWA should be conducted as a 

targeted survey of the appropriate communities of experts. 

• DTIC should be the custodian of the new DOD S&T database and for the automation of 

and the routine implementation of the EWA. The assessment should ultimately be a 

routine byproduct of doing business. 

• A full EWA as envisioned by this study should not be initiated in the short term. Many 

details need to be addressed before such an assessment could be undertaken. Certain 

aspects of the proposed process should be examined via the conventional review process. 

For example, automated techniques for forming communities of experts could be tested 

by employing them as part of establishing the conventional review teams. Test cases 

should be conducted to clarify how to document the program in terms of CEST and to 

examine and resolve problems that confront the distributed assessment approach. These 

test cases should address classified distributed assessments (e.g. utilizing SIPRnet) and 

unclassified distributed assessments. DOD should be an active participant in the national 

efforts that are underway to take advantage of the ongoing revolution in library science. 

If the process described in this report could be implemented it would have a number of spin-offs. 

For example, the supporting DTIC database could be interrogated so as to compare the actual 

program of record with the DOD S&T investment strategy. It would also establish a broad 

database of experts covering most of the S&T areas of interest to DOD. It may be that the most 

valuable result of this approach to an EWA is the establishment and maintenance of a detailed 

database for the funded S&T program and the development of a large community of experts who 

have a connection with and interest in DOD S&T. 
A1 

APPENDIX A 

Work Unit Summary Hypothetical Example 

1. Title: Hypersonic Turbine-Based Combined Cycle Engine (Mach 4 turbine and Mach 3.5 to 7 

scramjet engine) (U)

 2. Date of Summary: 12/08/05

3. Responsible Organization 

a. Name: Some DOD Activity b. Address: Somewhere, USA 

c. Principle Investigator: 

Name Phone Email Address 

John Doe 123-456-7891 jdoe@----.mil 

4. Classification Level: 

a: Summary Level: U b: Work Level: U

5. Military need: Technology relevant to candidate system solutions for Prompt Global Strike, Long 

Range Strike, and Assured Access to Space: 

• Responsive global power projection 

• Responsive and persistent ISR 

• Responsive global payload delivery 

• Responsive space payload delivery 

6. Technical Objective: Future hypersonic air-breathing UCAVs for time-critical theater operations, and 

reusable launch vehicles for prompt global response and routine space access, will require 2-3 

propulsion modes to operate across their design speed ranges. One of the most likely propulsion 

systems for such applications will be the turbine-based combined-cycle engine (TBCC), which consists 

of a supersonic (~Mach 4) turbine engine closely integrated with a dual-mode ramjet/scramjet engine. 

In order to proceed along this path it will be necessary to develop a turbine-based combined-cycle 

(TBCC) engine inlet system that efficiently delivers air flow required by the turbine engine when 

operating alone, air flow required by the scramjet when operating alone, and to both engines 

simultaneously during the transition from turbine to scramjet. It is also necessary to maintain inlet fluid 

dynamic stability during mode transition and in all other operational engine modes. One must develop 

a TBCC engine nozzle system that efficiently expands flow to produce thrust for the turbine engine 

when operating alone, for the scramjet engine when operating alone, and for both engines when 

operating simultaneously during mode transition. It is necessary to develop an aircraft and engine 

flow-path control system that adequately stabilizes and controls the aircraft and TBCC engine flowpaths during engine mode transition and all other flight phases. Lastly, it will be necessary to develop 

the material, structural and mechanical design for the articulated TBCC inlet and nozzle systems, and a 

thermal management system for the turbine engine when it is shutdown but exposed to scramjet flowpath heating 
A2 

7. Progress: New project

8. Critical Enabling Science and Technology (CEST) 

CEST 1: Hypersonic engine dynamic seals for articulated (i.e. variable geometry) inlet and nozzle 

structural elements. (U) 

Approach: Design and test high-temperature seals for a full-scale TBCC inlet and nozzle (Mach 4 

turbine and Mach 3.5 to 7 scramjet engine). Consideration will be given to both metallic and ceramic 

seals.

Technology Metrics: 

Metric Metric Unit SOA*

 Planned Value 

Maximum use 

temperature 

Temperature o F 1400o F (metallic) 

18000

 F (ceramic) 

1800o

 F (metallic) 

2500-3000o

 F (ceramic) 

Durability/life # of cycles before 

replacement 

1000’s for metallic 

10 – 100 for 

ceramic 

1000’s for <2000o F 

100’s at 2500o F 

10’s at 3000o

 F 

Milestones: 

TRL Year Description 

6 2008 Design and fabricate seals and 

complete critical component testing in 

a relevant environment 

*Current TRL = 2 - 3 

Funding: 

Fiscal Year 2005 2006 2007 2008 

Funding 

Profile 

$1200K $1500K $1800K $1500K 

Technical Work Years 

Fiscal Year 2005 2006 2007 2008 

Technical 

Work Years 

3 4 4 3 

 *

 State of the art. 
A4 

Suggested Community of Experts: 

Last Name* First Name* Organization Email Phone 

 NASA Glenn 

Research Center 

 

 NASA Glenn 

Research Center 

 

*names deleted 

CEST 2: Dual flow-path TBCC hypersonic engine inlets and nozzles (U) 

Approach: Define TBCC (Mach 4 turbine and Mach 3.5 to 7 scramjet engine) inlet and nozzle 

geometry and articulation requirements based initially upon knowledge of existing supersonic inlets 

and nozzles, empirical and theoretical design guidelines and operability (i.e., stability) criteria, engine 

performance characteristics, and analysis results derived from engineering codes and Computational 

Fluid Dynamics (CFD). This would be followed by building sub-scale inlet and nozzle models and 

testing them in wind tunnels, both statically and with dynamic articulation of the variable geometry 

components. The knowledge gained from testing and analysis will be used to optimize inlet and nozzle 

designs. This will be aided by mathematical optimization and multidisciplinary design optimization 

techniques. 

Technology Metrics: 

Metric Metric Unit SOA Planned Value 

Integrated dual inlet 

operability 

# of inlets operating 

simultaneously at high 

performance 

1 2 at SOA inlet recovery 

and operability/stability

Integrated dual nozzle 

operability 

# of nozzle operating 

simultaneously at high 

performance 

1 2 at SOA thrust 

coefficient 

Milestones: 

*Current TRL = 2 – 3 

TRL Year Description 

3 2006 TBCC inlet and nozzle design and 

analysis 

4 2007 Wind tunnel test sub-scale TBCC 

inlet and nozzle 

5 2008 Update inlet and nozzle designs, and 

tunnel test inlet on a sub-scale aircraft 

model with an articulated inlet 
A4 

Funding: 

Fiscal Year 2005 2006 2007 2008 

Funding Profile 0 $3000K $4000K $3000K 

Technical Work Years 

Fiscal Year 2005 2006 2007 2008 

Technical 

Work Years 

0 8 8 8 

Suggested Community of Experts: 

Last Name* First Name* Organization Email Phone 

 Johns Hopkins 

Applied Physics 

Laboratory 

 

 Air Force Research 

Laboratory/Propulsion 

Directorate 

 

 NASA Langley 

(Retired) 

 

 TechLand Research, 

Inc. 

 

 NASA Langley 

(Retired) 

 

*names deleted

CEST 3: Post shutdown turbine hypersonic engine thermal management system (TMS) (U) 

Approach: Design a thermal management system (TMS) for a TBCC engine (Mach 4 turbine and 

Mach 3.5 to 7 scramjet engine) flow-path and analyze it using fluid, thermal and structural analysis 

codes, and then bench test critical TMS components to verify predicted operation and performance. 

Technology Metrics: 

Metric Metric Unit SOA Planned Value 

Turbine bay 

temperature during 

scramjet operation 

Temperature N/A Max temperature 

capability of lowest 

temperature turbine 

material (~ 1500o

 F) 
A5 

 Milestones: 

TRL Year Description 

3 2007 TMS design and analysis 

4 2008 Critical TMS component 

bench testing and analysis 

update 

Funding: 

Fiscal Year 2005 2006 2007 2008 

Funding Profile NA $1000k $2000k $3000k 

Technical Work Years 

Fiscal Year 2005 2006 2007 2008 

Technical 

Work Years 

0 3 4 5 

Suggested Community of Experts: 

Last Name* First Name* Organization Email Phone 

 Pratt and Whitney 

 Pratt and Whitney 

 Pratt and Whitney 

 Air Force Research Lab 

 Pratt and Whitney, retired 

*names deleted

CEST 4: Integrated high speed airframe and engine control system development (U) 

Approach: Develop inlet and nozzle control algorithms for a hypersonic Turbine-Based Combined 

Cycle Engine (Mach 4 turbine and Mach 3.5 to 7 scramjet engine), and then test them via dynamic 

simulation using data derived from testing and CFD analysis. Vehicle stability and control during 

engine mode transition will be verified via dynamic simulation using CFD-derived aero data in 

conjunction with inlet and nozzle test data. Wind tunnel tests will be conducted of an aircraft model 

with variable inlet to verify inlet stability during mode transition, and aero data will be gathered to 

verify vehicle stability and control during the transition event. 
A6 

Technology Metrics 

Metric Metric Unit SOA Planned Value 

Inlet stability during 

engine mode transition 

Stability margin N/A 5% 

Aerodynamic dynamic 

stability during engine 

mode transition 

Control system phase 

and gain 

N/A 6 db gain and 45° 

phase margin 

Milestones 

TRL Year Description 

3 2007 Develop inlet, nozzle and aircraft 

control laws using aerodynamic 

propulsion analysis of same. 

Perform dynamic simulations. 

4 2008 Use inlet control laws in wind 

tunnel test of integrated 

inlet/airframe. Use tunnel data to 

update control laws and 

simulations 

*Current TRL = 2-3 

Funding 

Fiscal Year 2005 2006 2007 2008 

Funding 

Profile 

$1000 k $1500 k $1500 k $2000 k 

Technical Work Years 

Fiscal Year 2005 2006 2007 2008 

Technical 

Work Years 

3 3 3 4 

Suggested Community of Experts: 

Last Name* First Name* Organization Email Phone 

 Boeing Huntington Beach 

 NASA Dryden 

 Air Force Research Laboratory 

 Boeing St. Louis 

*names deleted
A7 

9. Total Project Funding: 

Fiscal Year 2005 2006 2007 2008 

Total 

Funding 

$2200k $7000k $9300k $9500k 

10. Program Participants: 

Contracts with whom XYZ Corp. 

Grants with whom TBD 

Government organization Some DOD Activity 

11. DOD Subject Areas: (http://www.dtic.mil/dtic/subcatguide/ ) 

• 01-01-Aerodynamics 

• 01-03-Aircraft 

• 01-04-Flight control and instrumentation 

• 21-05-Jet and gas turbine engines 

12. Key Words: Hypersonic turbine-based combined-cycle engine; High-temperature seals; 

Supersonic inlet; Supersonic nozzle; Turbine engine; Scramjet engine; Supersonic inlet control system; 

Variable geometry inlet; Variable geometry nozzle. 
APPENDIX B 

Potential Communities of Experts 

The potential community of experts shown here were obtained by on-line computer searches of various 

databases using search engines such as Scirus, Google, Google Scholar, and the AIAA Electronic Library 

Table B1: Potential community of experts for CEST 1: Hypersonic Engine Seals 

Number of Individuals 

Identified

Organization Organization Location

1 Advanced Components & Materials, Inc East Greenwich, RI 

2 Aerojet Sacramento, CA 

2 Aerothermal and Aero-Optics Evaluation 

Center 

Buffalo, NY 

1 Albany Techniweave, Inc. Albany, NY 

8 ATK Thiokol Propulsion Corporation Brigham City, UT 

2 BF Goodrich Aerospace Chula Vista, CA 

1 California Institute of Technology Pasadena, CA 

4 Case Western Reserve University Cleveland, OH 

1 Connecticut Reserve Technologies Stow, OH 

3 Drexel University Philadelphia, PA 

1 Embry-Riddle Aeronautical University Daytona Beach, FL 

4 FDC/NYMA, Inc. Hampton, VA 

3 General Applied Science Laboratories, Inc. Ronkonkoma, NY 

2 General Electric Cincinnati, OH 

2 General Electric Global Research Center Niskayuna, NY 

1 JIAFS, The George Washington University Hampton, VA 

9 Johns Hopkins University, Applied Physics Laurel, MD 

1 Modern Technologies Corporation Middleburg Heights, OH 

3 Mohawk Innovative Technology, Inc. Albany, NY 

2 NASA Ames Research Center Moffett Field, CA 

10 NASA Glenn Research Center Cleveland, OH 

1 NASA Headquarters Washington, DC 

28 NASA Langley Research Center Hampton, VA 

10 NASA Lewis Research Center Cleveland, OH 

2 NASA Marshall Space Flight Center Huntsville, AL 

2 Ohio Aerospace Institute Beavercreek, OH 

4 Orbital Sciences Corporation Dulles, VA 

2 Pennsylvania State University University Park, PA 

1 Refractory Composites Inc. Glen Burnie, MD 

1 Rensselaer Polytechnic Institute Troy, NY 

1 Saint-Gobain Advanced Ceramics Northboro, MA 

2 Sverdrup Technology, Inc. Brook Park, OH 

1 The Advanced Products Company North Haven, CT 

3 The Boeing Company Long Beach, CA 

1 Tribos Engineering, P.C Niskayuna, NY 

7 U.S. Air Force Research Laboratory Wright-Patterson Air Force Base, OH 

1 U.S. Army Research Lab Adelphi, MD 

1 Universal Technology Corporation Beavercreek, OH 

3 University of Akron Akron, OH 

1 University of Minnesota Minneapolis, MN 

1 University of Toledo Toledo, OH 

1 Virginia Tech, Mechanical Engineer Dept. Blacksburg, VA 

B1 
Table B2: Potential community of experts for CEST 2: Hypersonic Turbine-Based Combined 

Cycle Engine (Mach 4 turbine and Mach 3.5 to 7 scramjet engine) 

Number of Individuals 

Identified

Organization Organization Location

2 Allison Advanced Development Co. Indianapolis, IN 

2 Astrox Corporation College Park, MD 

3 Boeing Corporation St. Louis, MO 

1 Brigham Young University Provo, UT 

1 Johns Hopkins University Columbia, MD 

1 Johns Hopkins University, Applied Physics Laurel, MD 

8 NASA Langley Research Center Hampton, VA 

1 NASA Marshall Space Flight Center Huntsville, AL 

1 Pyrodyne Inc. Glenwood, MD 

1 SAF/AQRT Washington, DC 

2 Spiritech Advanced Products Inc. Jupiter, FL 

2 U.S. Air Force Research Lab Wright-Patterson AFB, OH 

1 University of Florida Gainesville, FL 

2 University of Maryland College Park, MD 

Table B3. Potential community of experts for CEST 3: High-speed turbine engine thermal 

management system (TMS) 

Number of Individuals Identified Organization Organization Location

2 Aerojet Sacramento, CA 

5 Air Force Research Laboratory Wright Patterson AFB, OH 

2 Allison Advanced Development Co. Indianapolis, IN 

1 Andrews Space Inc. Seattle, WA 

1 Brigham Young University Provo, UT 

4 FDC/NYMA, Inc. Hampton, VA 

1 Flight Unlimited Flagstaff, AZ 

4 Georgia Institute of Technology Atlanta, GA 

1 Johns Hopkins University Columbia, MD 

2 NASA Dryden Flight Research Center Edwards, CA 

7 NASA Glenn Research Center Cleveland, OH 

1 NASA Headquarters Washington, DC 

15 NASA Langley Research Center Hampton, VA 

1 NASA Marshall Space Flight Center Huntsville, AL 

1 Parks College St. Louis, MO 

1 Pratt & Whitney Space Propulsion West Palm Beach, FL 

6 SAIC Huntsville, AL 

3 SAIC Torrance, CA 

3 Space Works Engineering, Inc. Atlanta, GA 

2 Spiritech Advanced Products, Inc. Jupiter, FL 

3 The Boeing Company Long Beach, CA 

3 The Boeing Company Seal Beach, CA 

1 University of Florida Gainesville, FL 

B2 
Table B4. Potential community of experts for CEST 4: Hypersonic Engine Control System 

Development 

Number of Individuals Identified Organization Organization Location

5 Air Force Research Laboratory Wright-Patterson AFB, OH 

4 Andrews Space, Inc. Seattle, WA 

1 Boeing Company St. Louis, MO 

1 Boeing Phantom Works Huntsville, AL 

2 Boeing Phantom Works St. Louis, MO 

1 Brigham Young University Provo, Utah 

2 Enercon Systems Cleveland, OH 

3 FDC/NYMA, Inc. Aerospace Sector Hampton, VA 

1 Flight Unlimited Flagstaff, AZ 

1 Flow Parametrics New Castle, DE 

3 General Applied Science 

Laboratories 

Ronkonkoma, NY 

1 General Electric Cincinnati, OH 

1 General Electric Lynn, MA 

6 Georgia Institute of Technology Atlanta, GA 

1 International Space Systems, Inc. Huntsville, AL 

2 Johns Hopkins University Columbia, MD 

1 Modern Technologies Corporation Middleburg Heights, OH 

2 NASA Ames Research Center Moffett Field, CA 

3 NASA Dryden Flight Research 

Center 

Edwards, CA 

9 NASA Glenn Research Center Cleveland, OH 

1 NASA Headquarters Washington, DC 

32 NASA Langley Research Center Hampton, VA 

6 NASA Lewis Research Center Cleveland, OH 

1 NASA Marshall Space Flight Center Huntsville, AL 

3 Ohio State University Columbus, OH 

1 Parks College St. Louis, MO 

2 Pennsylvania State University University Park, PA 

3 Princeton University Princeton, NJ 

1 Proton Aerospace Corporation Jupiter, FL 

1 QSS Group, Inc. Cleveland, OH 

8 SAIC Huntsville, AL 

2 Spiritech Advanced Products, Inc. Jupiter, FL 

2 TechLand Research, Inc. North Olmstead, OH 

4 The Boeing Company Seal Beach, CA 

1 University of Alabama Huntsville, AL 

1 University of Florida Gainesville, Florida 

2 University of Maryland College Park, MD 

B3 
C1 

APPENDIX C 

NASA Technology Inventory 

This information presented in this appendix is taken from several sites that describe the NASA 

Technology Inventory. The purpose of the NASA Technology Inventory is to document the full scope 

of NASA's technology investment in order to improve management and communications inside and 

outside the agency. Each of the 5 NASA Enterprises manages a technology program that is focused on 

the needs and objectives of that Enterprise. In addition, the enterprises support technology activities 

within their Advanced Development, and Research and Analysis Programs. Other NASA programs 

contribute technology as well, including the Cross Enterprise Technology Development Program, the 

Small Business Innovation Research Program, the Center Director's Discretionary Program, safety and 

engineering programs, and educational programs. Collectively, all of the technology activities in 

NASA amount to about $1.6 billion. 

The Tech Inventory seeks to capture this spectrum of activities in a common, searchable format. Each 

input is linked to the Enterprises that would benefit, so that program managers can quickly identify 

technologies that support their programs. Keywords and a search engine enable technologists to 

identify related work. Technology Readiness Levels help describe the maturity of programs, and help 

separate near-term from far-term work. Enterprise-specific fields help NASA managers track their 

technology programs. Partnership information is collected as a NASA GPRA metric that helps 

measure the extent of collaboration with other agencies. 

For the purposes of this database, technology is defined as the practical application of knowledge to 

create the capability to do something entirely new or in an entirely new way. This can be contrasted to 

"scientific research," which encompasses the discovery of new knowledge from which new technology 

is derived, and engineering, which uses technology derived from this knowledge to solve specific 

technical problems. When investments are made in a particular technology, it begins to mature—a 

process of testing and analysis that progressively reduces the programmatic risk of selecting that 

technology for an application and increases the readiness of that technology for use in a mission. 

Technology may be described in terms of maturity within a scale of Technology Readiness Levels, 

which reflect the extent to which the technology has been proven in a realistic situation. 

Technology Readiness Levels (TRLs) are a systematic metric/measurement system that supports 

assessments of the maturity of a particular technology and the consistent comparison of maturity 

between different types of technology. The TRL concept is based on a general model for technology 

maturation that includes: (a) research in new technologies and concepts (targeting identified goals, but 

not necessary specific systems), (b) technology development addressing specific technologies for one 

or more potential identified applications, (c) technology development and demonstration for each 

specific application before the beginning of full system development of that application, (d) system 

development (through first unit fabrication), and (e) system launch and operations. 
C2 

Technology Readiness Levels Summary

TRL 1 Basic principles observed and reported 

TRL 2 Technology concept and/or application formulated 

TRL 3 Analytical and experimental critical function and/or characteristic proof-of-concept 

TRL 4 Component and/or breadboard validation in laboratory environment 

TRL 5 Component and/or breadboard validation in relevant environment 

TRL 6 System/subsystem model or prototype demonstration in a relevant environment (ground or 

space) 

TRL 7 System prototype demonstration in a space environment 

 

Discussion of Each Level 

The following paragraphs provide a descriptive discussion of each technology readiness level, 

including an example of the type of activities that would characterize each TRL. 

TRL 1 

Basic principles observed and reported 

This is the lowest "level" of technology maturation. At this level, scientific research begins to be 

translated into applied research and development. Examples might include studies of basic properties 

of materials (e.g., tensile strength as a function of temperature for a new fiber). 

TRL 2 

Technology concept and/or application formulated 

Once basic physical principles are observed, then at the next level of maturation, practical applications 

of those characteristics can be invented or identified. For example, following the observation of 

high critical temperature superconductivity, potential applications of the new material for thin film 

devices (e.g., SIS mixers) and in instrument systems (e.g., telescope sensors) can be defined. At this 

level, the application is still speculative: there is not experimental proof or detailed analysis to support 

the conjecture. 

TRL 3 

Analytical and experimental critical function and/or characteristic proof-of-concept 

At this step in the maturation process, active research and development (R&D) is initiated. This must 

include both analytical studies to set the technology into an appropriate context and laboratory-based
C3 

studies to physically validate that the analytical predictions are correct. These studies and experiments 

should constitute "proof-of-concept" validation of the applications/concepts formulated at TRL 2. For 

example, a concept for High Energy Density Matter (HEDM) propulsion might depend on slush or 

super-cooled hydrogen as a propellant: TRL 3 might be attained when the concept-enabling 

phase/temperature/pressure for the fluid was achieved in a laboratory. 

TRL 4 

Component and/or breadboard validation in laboratory environment 

Following successful "proof-of-concept" work, basic technological elements must be integrated to 

establish that the "pieces" will work together to achieve concept-enabling levels of performance for a 

component and/or breadboard. This validation must be devised to support the concept that was 

formulated earlier, and should also be consistent with the requirements of potential system 

applications. The validation is relatively "low-fidelity" compared to the eventual system: it could be 

composed of ad hoc discrete components in a laboratory. For example, a TRL 4 demonstration of a 

new fuzzy logic approach to avionics might consist of testing the algorithms in a partially computerbased, partially bench-top component (e.g., fiber optic gyros) demonstration in a controls lab using 

simulated vehicle inputs. 

 TRL 5 

Component and/or breadboard validation in relevant environment 

At this, the fidelity of the component and/or breadboard being tested has to increase significantly. The 

basic technological elements must be integrated with reasonably realistic supporting elements so that 

the total applications (component-level, sub-system level or system-level) can be tested in a simulated 

or somewhat realistic environment. From one-to-several new technologies might be involved in the 

demonstration. For example, a new type of solar photovoltaic material promising higher efficiencies 

would at this level be used in an actual fabricated solar array blanket that would be integrated with 

power supplies, supporting structure, etc., and tested in a thermal vacuum chamber with solar 

simulation capability. 

TRL 6 

System/subsystem model or prototype demonstration in a relevant environment (ground or space) 

A major step in the level of fidelity of the technology demonstration follows the completion of TRL 5. 

At TRL 6, a representative model or prototype system or subsystem which would go well beyond ad 

hoc, patch-cord or discrete component level breadboard would be tested in a relevant environment. At 

this level, if the only relevant environment is the environment of space, then the model/prototype must 

be demonstrated in space. Of course, the demonstration should be successful to represent a true TRL 6. 

Not all technologies will undergo a TRL 6 demonstration: at this point the maturation step is driven 

more by assuring management confidence than by R&D requirements. The demonstration might 

represent an actual system application, or it might only be similar to the planned application, but using 

the same technologies. At this level, several-to-many new technologies might be integrated into the 

demonstration. For example, a innovative approach to high temperature/low mass radiators, involving 

liquid droplets and composite materials, would be demonstrated to TRL 6 by actually flying a working, 

sub-scale (but scaleable) model of the system on a Space Shuttle or International Space Station pallet.
C4 

In this example, the reason space is the relevant environment is that microgravity plus vacuum plus

thermal environment effects will dictate the success/failure of the system and the only way to validate 

the technology is in space. 

TRL 7 

System prototype demonstration in a space environment 

TRL 7 is a significant step beyond TRL 6, requiring an actual system prototype demonstration in a 

space environment. It has not always been implemented in the past. In this case, the prototype should 

be near or at the scale of the planned operational system and the demonstration must take place in 

space. The driving purposes for achieving this level of maturity are to assure system engineering and 

development management confidence (more than for purposes of technology R&D). Therefore, the 

demonstration must be of a prototype of that application. Not all technologies in all systems will go to 

this level. TRL 7 would normally only be performed in cases where the technology and/or subsystem 

application is mission critical and relatively high-risk. Example: the Mars Pathfinder Rover is a TRL 7 

technology demonstration for future Mars micro-rovers based on that system design. Example: Xvehicles are TRL 7, as are the demonstration projects planned in the New Millennium spacecraft 

program. 

The NASA Technology Inventory is available to government employees and a restricted version is 

available to government contractors. This study accessed the inventory and found its functionality to 

be quite good. It was straightforward to extract information from the inventory and reconfigure it for 

specific display purposes. The following sequence of charts represents one such reconfiguration and is 

meant to illustrate how one can move from the highest level of description in the inventory to its 

lowest level. Each chart in the sequence was arrived at by a mouse click on a desired part of the 

previous chart. The first chart is the highest level of aggregation and the last chart is the lowest level. 

The example sequence tracks the Technology Discipline “Power”. The specific details of the program 

have been removed since they are not relevant to the purpose here where only functionality is of 

interest. 
C5 

NASA Technology Disciplines: Funding in $K

Advanced Concept Analysis

Aero & Aerothermod

Autonomy 

Avionics

Biotechnology

Communications

Cryogenics

EVA Education

Manufacturing/Processing 

Information Systems

Instrumentation Instruments

Life Support

Medicine/Health & 

Performance 

Power

Propulsion

Robotics

Space Environmental Effect 

Structures/Materials

Systems Engineering

Thermal Control

Thermal Protection Systems 

Training Technology

Total Funding = $3,951,991K
C5 

NASA Technology Discipline Power: Funding

Technology Discipline 

Generation 

Storage

Distribution & Management 

Total Funding = $60,097K
Power: Generation Funding Breakdown ($K) Task ID: 5827 

Task ID: 5832 

Task ID: 6366 

Task ID: 6286 

Task ID: 6427 

Task ID: 6488

Task ID: 6534

Task ID: 6923 

Task ID: 7580 

Task ID: 7695

Task ID: 7778 

Task ID: 7915

Task ID: 8123 

*Task ID: 8966 

Task ID: 9141

Task ID: 9424 

Task ID: 9552 

Task ID: 10264 

Task ID: 10289 

Total Funding = $24,162.14K

C7 
*Task ID: 5827 Task Name: *Advanced Thin Film Technology for Ultra-lightweight Solar Cell Arrays 

 

 

*Center *Sponsor 

GRC AST

 

Point of Contact 

 First Name Last Name E-Mail Phone 

Task POC 

Site POC 

COTR 

 

*Description 

*Objective & Description

 

*Technology Disciplines (Specific) 

 

FY04 

Funding 

$K

FY04 

FTE

Power 

 generation 

 

 

*Technology Application Areas & Special Categories 

• Terrestrial Atmosphere/Surface Systems 

• Space Technology/Spacecraft/Platform 

• Space Technology/Constellations 

• Space Technology/Planetary Atmosphere/Surface Systems 

 

Technology Metrics 

Technology Discipline: Power 

Metric Technology Metric 

Units

Current State of the Art 

Value

Planned 

Value

No metrics entered for this discipline 

Technology Discipline: Power | generation 

Metric Technology Metric 

Units

Current State of 

the Art Value

Planned 

Value

Efficiency % 

Mass Specific 

Power w/kg 

Areal Specific 

Power w/m2 

Specific Cost 

($/w) 

Metric Comments:

Not Specified 

 

C8 

Sample Task Description 
Milestones 

TRL Year Description

1 2004

1 2004

2 2004

3 2005

6 2012

*Maturity Time Frame: Far 

* Current TRL = 2 

 

FY 

*NASA Profile in $K 

Civil Service FTE 

 

NASA Distribution Funding 

Contracts With Whom 

Grants With Whom 

NASA Field Center 

Other (Space Act, etc.) 

Partnership Distribution Funding 

NASA Contributions 

Other Contributions 

 

*Link to other site 

 

*Attached Image 

None 

 

C9