December 2013

NASA/TP–2013-218145

Technology Estimating: A Process to Determine 

the Cost and Schedule of Space Technology 

Research and Development

Stuart K. Cole

NASA Headquarters, Office of Evaluation, Cost Analysis Division

Washington, D.C.

John D. Reeves and Julie A. Williams-Byrd

Langley Research Center, Hampton, Virginia

Marc Greenberg and Doug Comstock

NASA Headquarters, Office of Evaluation, Cost Analysis Division

Washington, D.C.

John R. Olds, Jon Wallace, Dominic DePasquale, and Mark Schaffer

SpaceWorks Enterprises, Inc., Washington, D.C.
NASA STI Program . . . in Profile

Since its founding, NASA has been dedicated to the 

advancement of aeronautics and space science. The 

NASA scientific and technical information (STI) 

program plays a key part in helping NASA maintain 

this important role.

The NASA STI program operates under the 

auspices of the Agency Chief Information Officer. 

It collects, organizes, provides for archiving, and 

disseminates NASA’s STI. The NASA STI 

program provides access to the NASA Aeronautics 

and Space Database and its public interface, the 

NASA Technical Report Server, thus providing one 

of the largest collections of aeronautical and space 

science STI in the world. Results are published in 

both non-NASA channels and by NASA in the 

NASA STI Report Series, which includes the 

following report types:

 TECHNICAL PUBLICATION. Reports of 

completed research or a major significant phase 

of research that present the results of NASA 

Programs and include extensive data or 

theoretical analysis. Includes compilations of 

significant scientific and technical data and 

information deemed to be of continuing 

reference value. NASA counterpart of peerreviewed formal professional papers, but 

having less stringent limitations on manuscript 

length and extent of graphic presentations.

 TECHNICAL MEMORANDUM. Scientific 

and technical findings that are preliminary or of 

specialized interest, e.g., quick release reports, 

working papers, and bibliographies that contain 

minimal annotation. Does not contain extensive 

analysis.

 CONTRACTOR REPORT. Scientific and 

technical findings by NASA-sponsored 

contractors and grantees.

 CONFERENCE PUBLICATION. 

Collected papers from scientific and 

technical conferences, symposia, seminars, 

or other meetings sponsored or cosponsored by NASA.

 SPECIAL PUBLICATION. Scientific, 

technical, or historical information from 

NASA programs, projects, and missions, 

often concerned with subjects having 

substantial public interest.

 TECHNICAL TRANSLATION. 

English-language translations of foreign 

scientific and technical material pertinent to 

NASA’s mission.

Specialized services also include organizing 

and publishing research results, distributing 

specialized research announcements and feeds, 

providing information desk and personal search 

support, and enabling data exchange services.

For more information about the NASA STI 

program, see the following:

 Access the NASA STI program home page 

at http://www.sti.nasa.gov

 E-mail your question to help@sti.nasa.gov

 Fax your question to the NASA STI 

Information Desk at 443-757-5803

 Phone the NASA STI Information Desk at 

443-757-5802

 Write to:

 STI Information Desk

 NASA Center for AeroSpace Information

 7115 Standard Drive

 Hanover, MD 21076-1320
National Aeronautics and 

Space Administration

Langley Research Center 

Hampton, Virginia 23681-2199

December 2013

NASA/TP–2013-218145

Technology Estimating: A Process to Determine 

the Cost and Schedule of Space Technology 

Research and Development

Stuart K. Cole

NASA Headquarters, Office of Evaluation, Cost Analysis Division

Washington, D.C.

John D. Reeves and Julie A. Williams-Byrd

Langley Research Center, Hampton, Virginia

Marc Greenberg and Doug Comstock

NASA Headquarters, Office of Evaluation, Cost Analysis Division

Washington, D.C.

John R. Olds, Jon Wallace, Dominic DePasquale, and Mark Schaffer

SpaceWorks Enterprises, Inc., Washington, D.C.
Available from: 

NASA Center for AeroSpace Information 

7115 Standard Drive 

Hanover, MD 21076-1320 

443-757-5802 
Abstract 

As the premier leader in space technology research and development, NASA is investing in new 

technologies (1) that include 14 primary technology roadmap areas, and aeronautics (2). Understanding the 

cost for research and development of these technologies and the time it takes to increase the maturity of 

the technology is important to the support of the ongoing and future NASA missions. Specifically, there 

exists a need to provide estimating capabilities for technologies in the Technology Readiness Level 

(TRL) range of 2 to 6. Overall, technology estimating may help provide guidance to technology 

investment strategies initially for NASA’s Game Changing Development Program Office and similar 

stakeholders help improve evaluation of technology affordability, and aid in decision support. 

A year-long research effort was initiated to help identify a systematic process for estimating the 

cost and schedule of low TRL technology research and development projects. This research project effort 

was complimented by phased work group sessions, multiple meetings, data acquisition and analysis, and 

discussions with experts in the field of technology maturation, NASA executives, technologists, analyst, 

principal investigators, engineers, and project/program managers. 

This research report provides a summary of the framework development of a Technology 

Estimating process where four technology roadmap areas were initially selected to be studied. The 

framework includes definition of terms, discussion for narrowing the focus from 14 NASA Technology 

Roadmap areas to four, and further refinement to include technologies in the TRL range of 2 to 6. 

Included in this paper is a discussion to address the evaluation of 20 unique technology parameters that 

were initially identified, evaluated and then subsequently reduced in number determined suitable for use 

in characterizing these technologies. Further, a discussion of data acquisition effort and criteria 

established for data quality are provided. A summary of the findings obtained during the research 

including gaps identified, description of a spreadsheet-based estimating tool initiated as a part of the 

Technology Estimating process, and the examination of a case study are provided. Suggestions for 

continued investigations in this research are also addressed. 
Page 2 of 88 

Table of Contents 

Introduction and background 3 

Purpose 4 

Benefits 5 

Determination of a Cost Estimating Process 6 

Scope of Technology 7 

1. Technology Tracking Not Standardized 8 

2. Naming System or Taxonomy of Technologies 9 

3. Taxonomy of Technologies Between NASA Organizations 10 

4. Endpoint or Disposition of the Technology 10 

Technology Areas 11 

Parameters Used for Characterization of Technology 11 

Definition of Terms 14 

Technology Estimating 14 

Risk 15 

Data 16 

1. Acquisition 17 

2. Sources of Data 19 

3. Pre-Completed Data 23 

4. Data Quality 25 

5. Parameter Evaluation and Screening 25 

Estimating Tool 30 

Findings 34 

Suggestions 36 

Conclusion 37 

Acknowledgements 38 

References 39 

Appendices 

Appendix A Definitions 42 

Appendix B Survey Form 54 

Appendix C List of Data Sources 56 

Appendix D Data Collection Sheets 59 

Appendix E Technology Estimating Data Quality and Screening Methodology 60 

Appendix F Parameter Evaluation Narrative 63 

Appendix G Technology Cost and Schedule Estimating (TCASE) Tool 74 

Appendix H User Manual Technology Cost and Schedule Estimating - Beta Version 75 

 
Page 3 of 88 

Introduction and Background 

The need for technology estimating was addressed to the NASA Cost community in 

2011Cost Symposium, where Sefcik et al., 2011 identified: 

 No current Agency/Glenn Research Center database exists for estimating technology 

development projects, 

 Data to populate such a database is in technologist’s head or spread out amongst many 

people or in long term storage or missing/discarded due to personnel turnover or ...., 

 No known good method to estimate the cost of Technology Reference Level (TRL)* 

advancement that is supported by actual data, 

 Reduction in Agency technology projects since 2005 results in little recent data that 

would be available, and 

 Estimates tend to be worked to the year’s budget versus planning through project 

completion. (3)

Note: * = TRL is defined in Appendix A. 

In response to the NASA Cost Community’s call for implementation of a capability to provide 

technology estimating, this research project was initiated and sponsored by the Office of Evaluation, Cost 

Analysis Division in collaboration with the Space Technology Mission Directorate, Game Changing 

Development Program Office. The goal was to improve NASA’s ability to estimate the cost of 

technology development and maturation. This technology estimating research involves identifying 

parameters that are expected to influence the cost of a technology, collecting actual data for completed 

technology efforts, and then conducting mathematical and statistical analyses to generate cost estimating 

relationships between the parameters and use these in an analysis to determine the cost and schedule of 

technology development.(4)

The procedural process diagram for the conduct of the research was prepared by the Cost 

Analysis Division and presented in Figure 1. The process was identified to include a series of workshops 

and internal reviews within the NASA Cost Community to elucidate issues and the development of the 

Technology Estimating research, provide the necessary community input and acceptance of the process, 

and provide a platform for the required internal peer review. 
Page 4 of 88 

Figure 1. Technology Estimating Process Diagram, 2012-2013 

Purpose 

Technology has been defined in a number of ways. NASA defines it as: 

“a solution that arises from applying the disciplines of engineering science to synthesize a device, 

process, or subsystem, to enable a specific capability.” (1) (5) 

Industry definitions include: 

By Boer, 1999 “the application of knowledge to useful objectives. It is usually built on previous 

technology by adding new technology inputs or new scientific knowledge” (6), or by Van Wyk, 1999 

“Created competence as manifested in devices, procedures, and acquired human skills”.

(7) 

Fundamentally, given any definition, it is recognized that technology will possess a monetary cost 

for its research and development, and will exhibit the duration in time to achieve its full maturation in 

preparation for inclusion in subsequent flight projects. Both are fundamental metrics that should be 

known. What is the purpose then for Technology Estimating, what are the benefits, and how do we do it? 

The intent of this paper is to report the information accumulated and process developed so as to document 

the technology estimating effort. 

Beltramo, 1988 identified the primary purpose of a cost estimating model is to support a decision 

at some level. The choice may be a “go” or “no go”, alternative A versus B, design trade off at the 

component level, or at some other level. Under all circumstances, the user must carefully consider the 

assumptions inherent in the estimating to determine they are appropriate for the intended purpose.(8)

Gaining an understanding of the cost and schedule of developing a technology could be critical for those 

who acquire technology research and development or to those who need to manage specific technology 

projects and technology advancement programs that manage a portfolio of technology projects. Clearly, 

an increased understanding of technology cost(s) or when management is involved in technology 

investments and how money may be spent on a single project, or either in a limited or broad portfolio of 
Page 5 of 88 

specific technologies, technology estimating should help in decision making. Oberlender et al., 2001, 

recognized that early estimating of projects is vital for business unit decisions that include strategies for 

asset development, potential project screening, and resource commitments. (9) 

Benefits 

We must acknowledge that the motivation for delivering or pursuing new technology research 

and development activities may be distinctly different for each stakeholder, and each has an opportunity 

to enjoy different benefits. NASA benefits were identified and discussed with technologists and the cost 

estimating community for this research and they are presented in Table 1. 

Table 1 

Benefits to NASA - Technology Estimating 

NASA Benefit 

Projects and Programs 

Pre-Phase A, to satisfy 

NPR 7120.5E, 5.f. Basis 

of Estimate (cost and 

schedule) 

“The project also develops and maintains the status of a set of programmatic and technical 

leading indicators to ensure proper progress and management of the project. These 

include..... cost trends”. 

Analyze technology 

costs to satisfy NPR 

7120.5E, 8.0 

Technology Readiness 

Assessment and 

Development 

“Provide technology development schedules, including intermediate milestones and 

funding requirements, during Phases A and B for each identified technology development 

to achieve TRL 6 by PDR. Describe expected status of each technology development at 

SRR, MDR/SDR, and PDR.” 

Analyze technology 

costs to satisfy NPR 

7120.5E, 3.5 

Technology 

Development Plan 

where it is required 

“describe how the project will transition technologies from the development stage to the 

manufacturing and production phases.” Technology Estimating would provide the data to 

evaluate the cost and schedule during the phases. 

Support NPR 7120.8 

NASA Research and 

Technology Program 

and Project 

Management 

Requirements, R&T 

program Commitment 

Agreement (PCA) 

Template 

Provide the estimated cost range for the program for the five-year period beginning in the 

current fiscal year at a level of detail that identifies the approved individual projects. 

Identify the constraints and assumptions used to develop this estimated cost range and 

specifically identify those assumptions that drive the range. This cost range should contain 

all costs necessary to perform the program, including, but not limited to, standard project 

activities, required technology developments, facilities costs, infrastructure costs, 

operations and sustainment, data analysis, and disposal. Reference the annual budget 

contained in the Integrated Budget and Performance Document (IBPD) for cost phasing. 

The cost range should be updated when program content changes, such as the addition of 

new projects entering implementation.” Appendix D, 6.0 Cost Commitment 

Task Managers, Program 

Managers, Principal 

Investigators 

Provide capability and dataset to benchmark cost and schedule estimate plans against prior 

technology investments. 

Increase confidence in technology cost/schedule thus increasing capability to manage 

technology costs. 

Help analyze factors that influence technology cost and thereby reduce cost. 

Technologists Provide a capability and dataset to better anticipate the cost and schedule resources needed 

to mature a technology to a target TRL level. 

NASA-wide 

Supports need to independently assess the likelihood that various technology cost and 

schedule plans can be successfully met. 

Provides a tool and dataset that addresses gap in existing cost analysis methods for 

technologies TRL 2 to 6. 

Knowledge of Technology cost(s) and schedule increases understanding as to whether it 

can be affordable. 

Establishes framework for future data collection to further enhance estimating capabilities. 
Page 6 of 88 

How you perform technology cost and schedule estimating can be a challenge. The initial 

challenge comes from the decision for which cost model or estimating process should be selected. The 

considerations made for selection must include what ultimate process is best for evaluating complex 

technologies, each with their own set of unique or potentially abstract conditions. Added to the challenge 

are potential differences in the characterization of a technology, or from one technology to another where 

the difference can be significant. 

The difficulties and considerations made, for how to estimate technology has been addressed by 

Hazelrigg et al., 1991 who recognized the difficulties in estimating the cost of space technologies (10), Roy 

et al., 2005 recognized the need in commercial applications where a systematic approach to cost 

estimating of technologies in the automotive industry was needed(11), Ingalls et al., 2004 who recognized 

that the use of parametric estimating in budgeting, scheduling, and control of (Technology driven) 

projects would enhance the ability of project management organizations to effectively and efficiently 

utilize valuable resources,(12) and Cyr, 1988 who investigated parametric cost estimating methods for 

advanced space systems in the conceptual phase. (13) The Rand Corporation, 1992 while engaged in 

technology estimating of complex aerospace technologies, identified the need to perform estimating using 

parametric analysis for complex systems early in the development of advanced technologies. (14) From this 

community, it was evident that a technology, during its early stages of research and development, must be 

characterized, not only by its maturity, but by how much definition a technology was given by a thorough 

characterization. A basic assumption taken in this research was that all technologies have inherent 

variability.

Determination of a Cost Estimating Process 

Knowing that there was variability in the very thing we are trying to estimate, and given 14 plus 

one technology areas, the derivation of credible estimates seems to be a daunting task. How then do we 

begin to select a cost estimating process and what NASA technologies do we address? To answer these 

questions for technology estimating in this research, the Research Team proceeded in two steps: 1) 

summarize the types of cost estimating processes available; then 2) to make simple refinement of the 

NASA technology data based on TRL. In the first step, a short summary has been identified for five cost 

estimating processes, and they are presented in Table 2. 

Table 2 

Cost Estimating Processes – Advantages and Disadvantages 

(Hazelrigg et al. 1991(10), Roy Et al. 2005 (11), Shishko et al. 2003 (15), Dumas 1979 (16), Cavalieria, 2003(17) ) 

Cost Estimating Process Advantages Disadvantages 

Analogy-based technique (10) Estimate is based on degree of similarity between 

projects. 

Used when little or no data is available. 

Degree of similarity is difficult to assess. Costs 

are subjectively adjusted up or down depending 

on the project complexity. 

Engineering approach (11)

(also known as “bottoms up”) 

Method is systematic and based on elementary 

components. 

Design or technology must be very well defined. 

Requires large amount of time for preparation. 

 

Real Options Valuation (15) Valuing projects in an environment of extreme 

uncertainty. 

Requires extensive systems thinking and 

engineering analysis to build a credible 

simulation of alternative sample paths. 

Parametric (16) Objective, unbiased, consistent. 

Takes less time. 

Suitable for use early in project formulation with 

limited project definition and limited data 

Assumption is that same condition(s) which 

governed costs in the past will be the same on 

each project. 

May exacerbate cost growth. 

Artificial neural network (17) Non-linear and parallel, can “infer” from knowledge. Requires training, and provides “black box” 

solutions 
Page 7 of 88 

After a literature review and review of the multiple analytical methods, the formalized parametric 

estimating techniques and data collection and analysis as outlined by the International Cost Estimating 

and Analysis Association in the Parametric Estimating Handbook (18) was determined to be the best option 

for further investigation. The selection of parametric estimating as the best choice for use in cost 

estimating technology is also substantiated by Thibault, 1992 who indicated that the “US Defense 

Contract Audit Agency believes now, as it always believed, that parametric estimating techniques using 

cost-estimating relationships are acceptable in the appropriate circumstances for proposing costs on 

government contracts.”(19) Fad et al., 1988, identified that parametrics is only the cost estimating method 

that can function within the limited data and estimating turn-around time constraints of the new business 

environment.(20) The Federal Acquisition Regulations (FAR) 15.404-1(b)(2)(iii) and 4(c)(i)(C) states that 

parametrics are an acceptable method for performing price analysis.(21) Based on the comparison of 

advantages and disadvantages of various accepted estimating processes and literature review, parametric 

estimating provides the best possible solution for satisfying this technology estimating research given 

the need to perform analysis early in project definition and possessing limited data. 

Scope of Technology 

The second step in the cost estimating process is that we refined the NASA technologies by 

separating them into two classes, i.e. TRL 2 to 6; and technologies greater than TRL 6. This separation of 

technologies into classes was made to simplify and reduce the scope of technologies to be investigated. It 

is commonly recognized that technologies in the TRL range of 2 to 6 normally have limited detailed 

project information, while the extent of detail or information for technologies with TRL > 6 usually 

possesses project information in much greater detail. Technologies with TRL > 6 were also identified to 

have significant or extensive amount of project related information and this information has been 

typically and accumulated in and represented by many cost estimating cost models and data sets adopted 

by NASA Cost Community such as NAFCOM and REDSTAR. The investigation of TRL 1 projects was 

deferred in this initial study to help limit the number of potential projects and reduce the scope of the 

research effort. 

In the Phase I Workshop, technologies with TRL range from 2 to 6 were identified to include 

those systems, sub-systems, parts, components and materials that were non- flight technologies. Figure 2 

presents the primary scope of technologies that were investigated and provides a matrix indicating the 

technologies of interest in this research study were TRL in the range of 2 to 6. 
Page 8 of 88 

Figure 2. TRL Versus Technology Scope Classification – Presented at the Phase I Workshop 

Before the Phase I Workshop, the Research Team deliberated on the extent of data that may be 

available in each of the respective technology areas and that several challenges may exist. Later these 

challenges were found to become evident during the development and investigation of the TRL 2 to 6 

technology projects scope. Each of these “challenges” is listed below with a Research Team observation: 

1. Technology Tracking Not Standardized 

Meaning: Standardization of Technology Tracking at NASA would allow an understanding of 

technology and TRL progression and help refine an understanding of native problems or difficulties in 

certain technologies, and to help identify common issues. Tracking of technologies in successive phases 

or similar technologies between programs would help to understand the progression of a technology, and 

possible impacts to duration. 

Research 

Observations: 

a. The deployment of NASA’s technology roadmap was not brought to light until at least the period of 

time after February 2012, when the Steering Committee for NASA Technology Roadmaps; National 

Research Council of the National Academies published “NASA Space Technology Roadmaps and 

Priorities: Restoring NASA's Technological Edge and Paving the Way for a New Era in Space”.(2) 

After this publication, NASA’s technologies would then be assigned into specific Technology Areas 

(TAs) and these assignments were in the early process of being established. Prior to this publication 

therefore, no previous technology project data before February 2012 would be expected to 

consistently contain the formalized NASA Technology Area Roadmap designations. 
Page 9 of 88 

b. NASA’s new technology information resource (TechPort) was published internally to NASA on 

October 10, 2012. TechPort was constructed for tracking technologies and providing help 

standardization of the tracking of technology. After an initial survey of the data contained in TechPort

during the first phase of this research project, only five completed projects were found to be included 

in the first publication, although this number was found to grow with time. Further, only five of the 

20 technology estimating parameters were found common in TechPort, and many of these were not 

fully completed and provided as placeholder to be completed later. 

c. Tracking technology in its early progressive stages along the path of development or where the 

early technology has branched to other Technology Roadmap Areas (TAs) would be a significant 

building block to Technology Estimating. Systematic tracking of technology cost and schedule 

efforts using the technology characterization parameters identified in this research as they could be 

applied to NASA and other data sources. These data sources may include the New Technology 

Reporting https://invention.nasa.gov/, TechPort https://techport.nasa.gov/techport/home.action, Small 

Business Innovation Research/ Small Business Technology Transfer (SBIR/STTR) 

http://sbir.gsfc.nasa.gov/SBIR/SBIR.html, Earth Science Technology Office (ESTO) Techfolio 

http://www.estotechnology.us/techportfolio/, One NASA Cost Engineering (ONCE), and other 

governmental organizations. 

2. Naming system or Taxonomy of Technologies 

Meaning: A naming system for technologies helps to identify where a technology fits or is 

attributed to the larger system or mission. Whether the particular technology is a material, part, 

component, etc. of a larger system, a naming system or convention was perceived to be especially helpful 

when multiple items are included in the construct of the technology. 

Research 

Observations: 

a. The NASA Systems Engineering Handbook identifies systems, components, etc. as shown ballooned 

in red in Figure 3. (5) Beyond the Work Breakdown Structure (WBS), which is unique to each 

technology, NASA does not appear to have a standard taxonomy to name overall systems or 

components. Further, with respect this research, technology projects in the range of TRL 2 to 6 do not 

normally have a WBS. 
Page 10 of 88 

Figure 3. Reference to naming system 

(From NASA Systems Engineering Handbook (5), page 295) 

b. For convenience, the basis for the “Technology Scope Classification” shown on the axis of abscissas 

identified in Figure 2 and as presented in the Phase I Workshop was formulated by the Cost Analysis 

Division, and with an eye towards International Standards Organization (ISO) 27026 (22). The ISO 

27026 issued a formative naming taxonomy and tree structure for cost breakdown structure in 

Appendix E of ISO 27026, which was used to help formulate the “Technology Scope Classification”. 

3. Taxonomy of Technologies Between NASA Organizations 

Meaning: A naming system for technologies that is common between NASA organizations 

would help translate technologies and communication across organizations. Common naming systems 

would improve technology estimating and data management. 

Research 

Observation: NASA has multiple technology organizations and naming of systems between them 

was not observed to be standard. The common naming of items between organizations would help to 

retrieve information and cost and schedule data for technologies between different NASA organizations. 

4. Endpoint or Disposition of the Technology 

Meaning: The identification for what status the technology takes at its conclusion is important to 

understanding of the technology progression or if the technology has been dead-ended. The disposition of 

the technology is a clarification if the technology is to be tracked further or identified for how it can be 

progressed. 
Page 11 of 88 

Research 

Observation: NASA does require technology projects to identify the end point or disposition of the 

technology. Identifying this information would help to track technology and illuminate the cost and 

schedule impact on future similar or same projects. 

The Phase I meeting confirmed that the focus of this research will include the scope for 

technologies with TRL range of 2 to 6 and that those technologies would be comprised of parts, materials, 

components, sub-systems or systems. Further, it was widely recognized within the NASA Cost 

Community that NASA currently has no cost estimating tool(s) for the scope of technologies within the 

range of TRL 2 to 6. Therefore, the focus of this Technology Estimating research project has been 

identified for technologies with TRL range of 2 to 6. 

In summary, multiple cost estimating processes were identified and evaluated with respect to 

the advantages and disadvantages presented, and considering the scope of technology addressed, 

parametric estimating offers the best option for use in this research to provide cost estimates for 

technologies in the TRL range of 2 to 6. However, based on the extent of retrospective data received 

during this phase of research, the analogy based methods will be used until the data can be evaluated 

further for Costing Estimating Relationship (CER) development and parametric analysis using those 

CERs.

Technology Areas 

 In 2004, the Steering Committee for NASA Technology Roadmaps; National Research 

Council of the National Academies published NASA Space Technology Roadmaps and Priorities: 

Restoring NASA's Technological Edge and Paving the Way for a New Era in Space. On December 2012, 

NASA rolled out the NASA strategic Space Technology Investment Plan (1). These reports identified 

NASA’s 14 technology roadmap to space technology. As a finding of the Phase I Workshop, four 

technology areas were identified to be investigated for Technology Estimating. It was resolved at the 

Phase I Workshop that the following technology areas, listed in Table 3, provided the highest opportunity 

for providing a large enough number of technology projects that could be investigated for use in the 

research. In addition, it was recognized that any attempt to include all 14 technology areas would be 

inextricably difficult within the one year term planned for this research effort. Further, the acquisition for 

data for all of the 14 technology areas would be a challenge within the time frame of this research. 

 

Table 3 

Phase I Workshop Results 

NASA Technology Roadmap Areas Identified to be Investigated in the Technology Estimating 

Research

TA03 Space Power and Energy Storage 

TA04 Robotics, TeleRobotics, Autonomous Systems 

TA08 Science Instruments, Observatories, Sensor Systems 

TA12 Materials, Structures, Mechanical Systems, Manufacturing 

Parameters Used for Characterization of Technology 

 Describing technology such that it can be adequately measured for use in Technology Estimating 

was one of the principle framework topics addressed in the Phase I Workshop. The Research Team 

engaged a three step process for development of parameters that would be used in characterizing the 
Page 12 of 88 

technology. First, a literature search was conducted to determine those parameters that could best 

characterize NASA technology. The second step was to evaluate all the technology parameters in the 

Phase I Workshop that could best be applied for use in the cost estimating process and data collection. 

The third step included an evaluation of the parameters during the data collection phase, and for use in the 

Technology Cost and Schedule Estimating (TCASE) tool development. 

It was anticipated by the Research Team that the CERs will include multiple parameters that 

significantly influence the cost of technology development. These included parameters such as: 

Technology Readiness Levels (TRL), Key Performance Parameters (KPPs), Complexity, Research & 

Degree of Difficulty (R&D3

), Advanced Degree of Difficulty (AD2), Implementation Readiness Level 

(IRL), etc. Initially, there were 13 parameters presented for consideration at the Phase I Workshop, see 

Figure 4. 

Figure 4. Initial Phase I Workshop Parameters Presented for Consideration 

 The advantages and disadvantages for prospective parameters presented for consideration in the 

Phase I Workshop and are provided in Table 4 for convenience, and also included are those parameters 

evaluated in the Phase II workshop. 
Page 13 of 88 

Table 4 

Parameter - Advantages and Disadvantages 

Parameter Advantages Disadvantages 

Technology Readiness 

Level 

Experienced procedure; Assessment can affect Cost 

Estimate; Analysis using TRAT tool; Fast, iterative 

process easily repeated during development 

Highly subjective; No defined process for defining 

TRL values; Characterize what drives the time to 

Technology Maturity; Difficult to measure; Risk to 

quality not fully developed; Identify future success? 

Key Performance 

Parameters 

Measurable performance goals that a given technology 

must attain to enable mission objectives; A significant 

advancement in the state-of-the-art 

Unique to each specific system, therefor application 

not well defined across multiple technologies 

Research and 

Development Degree of 

Difficulty 

Compliments TRL by providing an accurate and effective 

assessment of the difficulty required for TRL 

advancement 

Maturity of application & approach; Subjective? 

Manufacturing 

Readiness Level 

Adds analysis to determine manufacturability and 

identify related risks; Predicts whether or not we will be 

able to produce the product in the timeframe and at the 

rate desired with the desired quality; Identifies risks for a 

program office to work on. 

Subjective; No defined process for defining MRL 

values; Information must be dug out of 

manufacturer 

Criteria for comparisons must be established 

Advancement Degree of 

Difficulty 

Prior demonstration of implementation with AD2 analysis 

tool Little demonstrated usage beyond AD2 tool 

Schedule 

Defined schedule showing maturity increasing/adequate 

analysis and testing; High risk items, work around – 

Contingency development 

Technology development probability of failure is 

similar to any project; Need defined WBS, 

requirements, schedule, cost, etc. 

Integration Readiness 

Level 

Enables consistent comparison of the maturity of 

different integration points 

Insufficient for evaluating the developmental state, 

potential, or risk of a system 

System Readiness Level 

(SRL) 

Common platform for system development and 

technology insertion evaluation; Hierarchal view of 

technology insertion/system development assessment 

System readiness (capability, maturity) may be too 

complex concept; Calculated from other metrics 

rather than objectively measured, may yield 

inaccurate information and non-intuitive results 

Cost* Cost is fundamental to requirements for estimating 

new technologies. All costs may not be included due to shift in WBS. 

Estimate vs. Actual* Actual costs are preferred for parametric analysis. Estimates do not meet criteria for parametric 

analysis. 

Funding Sources* Provides understanding of possible program funding 

impact to technology cost and schedule. 

Multiple funding sources creates difficulties in 

tracking the technology costs 

Push vs. pull* Identifies the impetus for technology Application of classification is subjective 

Evolutionary vs. 

Revolutionary* Identifies the level of innovation Application of classification is subjective 

Technology Area* Segregates the technology into pre-defined road map 

areas 

Cross referencing of technology areas can be 

confusing 
Page 14 of 88 

Note: * = Identified during and after Phase I Workshop 

Definition of Terms 

 Initiation of this research required an assessment of the language terms that the Research Team 

used with one another in discussions and during the many briefings made throughout this research 

project. It was quickly recognized that a need existed to define terms, especially prior to the Phase I 

workshop. The construct of the definitions were intended to have a positive impact and to help elicit 

meaningful discussions amongst the workshop participants and provide a near community acceptance for 

how to communicate issues. In addition, the definitions were intended to be provided as a supplement in 

the data collection process. It is noteworthy that the definitions have not gained acceptance outside this 

research. The definitions were changed and modified multiple times over the course of the research to 

evolve into those definitions now used by the Research Team and they are presented in Appendix A. 

Technology Estimating 

 For the purposes of this research, the phrase “technology estimating” implies the estimating of the 

cost and schedule requirements for the development of technologies. This differs from the interpretation 

of “technology estimating” used throughout most of the NASA community, for the broader technology 

estimating term often implies the analysis and estimation of the improved technical performance of a 

system or architecture due to the inclusion of a given technology. The usage here of technology 

estimating to mean the estimating of the cost and schedule for technology maturation is limited to the cost 

and schedule estimating communities. 

Table 4 - Continued 

Parameter - Advantages and Disadvantages 

Parameter Advantages Disadvantages 

Hardware vs. software* Helps to illustrate the technology complexity Software can be developed as its own parameter 

Defining System 

Characteristics* Characterizes technology to a greater detail Difficult to apply consistent rationale 

Level of Effort* Defines the number of staff involved and potentially a 

good indicator of cost Students, contractor staff not well identified 

Unplanned Events* Helps to illustrate emergent costs Difficult data to acquire and information is 

currently not well documented 

Organization(s) 

Involved* 

Identifies the level of specialties required for a 

particular technology Difficult to establish as a cost driver 

Facilities and 

Infrastructure* Illustrates the basis for cost Difficult data to acquire and information is 

currently not well documented 

Team Experience* Exposes relation of technology complexity and ease of 

administration/accomplishment May be subjective 

Budget Constraints and 

Disruptions* Helps to illustrate emergent costs Difficult data to acquire and information is 

currently not well documented 
Page 15 of 88 

Risk 

Risk may be defined as a measure for the loss of an asset or capability. Technology estimating 

provides the fundamental metrics that should help the technology manager reduce, eliminate or accept the 

risks pertaining to the development cost and schedule of a technology. The asset or capabilities that could 

be lost within technology development efforts include the success of the technology development effort 

itself. Once these risks are revealed or better understood, it will both enlighten and embolden the 

technology or technology portfolio manager and potentially help to support the foundation for partnership 

building. Partnerships that are founded on an understood cost and schedule risk will position the 

partnership to propel success in a technology’s research and its ultimate development. NASA has clearly 

indicated that partnerships for technology development, transfer and mutual benefit is a key objective in 

the Space Technology Program.(23) As an instrumental supporting role, technology estimating capability 

could help reduce risk by providing cost and schedule input data to decision support systems such as 

those identified by Smith et al., 2004(24) and Weisbin et al., 2004, for complex technology portfolios.(25)

Recognizing the full impact of cumulative risks in technologies within portfolio management could 

improve overall risk management and further the opportunity for success in the mission’s goal. 

Risk for technology projects has been addressed in the literature. (24) (25) (26) (27) Although it is recognized 

that technology risk is a driver to cost and schedule of technology, due to the results of the Phase I 

Workshop, it was determined that there would be a limited or no data captured in a meaningful way 

for retrospectively reporting the risk data in the TRL< 6 projects. Therefore, risk as a parameter for use 

in analyzing individual technology projects would not be addressed in this research. 

Upon completion of the Phase I Workshop, and from input by technologist during the course of 

the research, a total of 20 parameters evolved and were then ultimately established for characterizing 

technologies and then used in the data collection. Figure 5 presents the 20 parameters identified. The 

parameters were identified into generalized classes to include: 

 Project characterization, 

 Project results, 

 Development metrics, 

 Project execution, and 

 Programmatic factors. 

Each classification was devised to help capture the overall technology project diversity, 

variability, and technical detail. These parameters have been defined in Appendix A. 
Page 16 of 88 

Figure 5. Technology Estimating Parameters Identified in Phase I Workshop 

Used for Characterization of Technology and Data Collection 

 

Data 

Essential to this research project was the technology project data. Because limited historical data 

was believed to exist, or was anticipated to be available for the parameters which were expected to be 

significant drivers of technology development cost and schedule, a retrospective effort to gather historical 

data was required. Therefore, the focus of data for this research would be past projects that contained the 

best available information that could be readily found from online sources, embedded data in costing 

programs, or other Nasa Cost Analysis Division supported software. 

 A significant focus in this research was to obtain historical project data, retrospectively 

generated, which would be used to help develop a key outcome of the research. This key outcome is a 

consensus in the cost and technology communities about those parameters which are determined to be 

key drivers for technology cost. Once this consensus is obtained, the parameters would be analyzed for 

development of cost estimating relationships. Once these relationships and parameters could be 

decided upon, then future collection of data related to these parameters can then be incorporated into 

ongoing databases and data collection efforts such as TechPort, One NASA Cost Engineering (ONCE) 

and Nasa Cost Instrument Model (NICM). Over time, existing CERs can be refined and new CERs can 

be developed as the amount of available data to conduct the requisite analysis expands. 
Page 17 of 88 

1. Acquisition 

The acquisition of data for this research was broken down into three tasks for Technology 

Projects in the TRL range of 2 to 6 and the four technology roadmap TAs (3, 4, 8 and 12). These data 

acquisition tasks included: 

 Identification of a NASA community preference for which TAs, the NASA programs might 

prefer to be studied first; 

 Identification of the technology funding programs currently being administered by NASA as of 

2012; and then 

 Determine which of the available projects that could be identified by a survey given at the Phase I 

Workshop, and by survey of all data sources developed by exhaustive research. 

The first component to data acquisition was the Research Team’s task to determine if there 

existed an individual or most requested preference for which of TAs areas were to be studied and 

subsequently evaluated for different NASA programs or organizations. Based on a random survey of 

several technologists via informal interviews with different NASA programs, no preference was identified 

for one of the TAs was to be evaluated over another. 

The second component to the data acquisition was the identification of technology funding 

programs throughout NASA, and from these, it was necessary to identify which of those would 

potentially possess the necessary population of technical projects with data or information that might 

yield names of Technology Managers, and who could provide the requisite retrospective data. 

Approximately 42 NASA funding programs were identified initially for use by the Research Team for 

retrospective data generation by the Technology Managers. Table 5 lists the funding programs identified 

in the research to determine the availability of data. The list was derived from those programs identified 

and listed in TechPort. 
Page 18 of 88 

Table 5 

Funding Programs Identified to Be Surveyed for Technology Estimating Data

Number Centers & Facilities Programs 

1 Advanced Exploration Systems 

2 Astrophysics Research and Analysis 

3 Centennial Challenges 

4 Center Independent Research & Developments: GSFC IRAD 

5 Center Independent Research & Developments: JPL IRAD 

6 Center Independent Research & Developments: JSC IRAD 

7 Center Independent Research & Developments: KSC IRAD 

8 Center Independent Research & Developments: LaRC IRAD 

9 Center Innovation Fund 

10 Center Innovation Fund: ARC CIF 

11 Center Innovation Fund: DFRC CIF 

12 Center Innovation Fund: GRC CIF 

13 Center Innovation Fund: GSFC CIF 

14 Center Innovation Fund: JPL CIF 

15 Center Innovation Fund: JSC CIF 

16 Center Innovation Fund: KSC CIF 

17 Center Innovation Fund: LaRC CIF 

18 Center Innovation Fund: MSFC CIF 

19 Center Innovation Fund: SSC CIF 

20 Discovery Program 

21 Earth Science Technology Office 

22 Exoplanet Exploration 

23 Flight Opportunities Program 

24 Fundamental Aeronautics Program 

25 Game Changing Development 

26 Ground Systems Development and Operations 

27 HELIOS 

28 Mars Exploration 

29 NASA Electronic Parts and Packaging Program 

30 NASA Innovative Advanced Concepts 

31 OSMA Software Assurance Research Program 

32 Planetary Science Research 

33 Planetary Science Technology Program 

34 SBIR/STTR Programs 

35 Small Spacecraft Technology Program 

36 Space Life and Physical Sciences Research and Applications 

37 Space Technology Research Grants 

38 Technology Demonstration Missions 

39 Phase I SBIR 

40 Phase II SBIR 

41 Phase III SBIR 

42 Constellation ETDP (Exploration Technology Development Program) 
Page 19 of 88 

The third component for data acquisition was an exhaustive survey of the available technology 

projects that may be available for use in the research. To help identify possible technology projects with a 

potentially rich data to help satisfy the retrospective data generation by the Technology Managers, a 

survey was conducted at the Phase I Workshop. The survey included a written document (form) which 

was developed and provided to the Phase I Workshop participants. The form is included in Appendix B. 

The information obtained from the survey during the Phase I workshop was promising, however the 

resulting number of individual Technology Projects revealed as a result of this survey were ultimately 

found to be limited. In addition to this survey at the Phase I Workshop, a listing of 76 prospective data 

sources was developed by the Research Team and is included in Appendix C. This listing was based on 

research of NASA programs, literature search, and from cost estimating models common to NASA and 

commercial industry typically used for TRL > 6 missions and projects. 

 To help acquire the needed data for the research, a Data Collection Sheet was prepared by the 

Research Team and consisted of an Excel® spreadsheet in three printable sheets and included in 

Appendix D. In preparation of the collection sheet format, the Research Team generally followed the 

NASA’s TechPort for its sequence of information as presented on the web page. In addition, the data 

collection effort generally followed the principles as outlined by Meisel, 1988, who identified three key 

requirements for obtaining the correct model input data: (1) finding the right experts, (2) asking the right 

questions, and (3) transforming the question responses correctly into data compatible with the model 

input format. (28)

The format of the data collection sheet along with the instructions for preparing answers was 

discussed and modified at least four times by the Research Team. After resolving many concerns, the data 

collection sheet was then used in at least three separate and individual interviews with Technology 

Managers for preliminary testing. The format of the data collection sheet consisted of pull down menus 

where possible so as to aid the Technology Manager in making a response. File extension naming of the 

data collection sheets also provided very helpful in identifying the correct projects with the respective 

Technology Manager and accountability. The data collection sheets were also made suitable for data 

retrieval into an Integrated Technology Project Database which was used as a data source to the 

Technology Cost and Schedule Estimating (TCASE) tool. 

 Overall, data collection followed guidelines as identified by established methodologies (29)(30)

including structure in the document, pre-testing with sample technologist, and including wording or 

sequence to help reduce bias. 

2. Sources of Data 

The existing data sources identified in the first phase of the research were compiled and 

investigated in detail. Those with reliable cost and schedule information were selected for use as 

supporting data for the TCASE framework. The specific data sources are identified below, along with the 

year the data source was produced, and the number of entries (i.e. technology data projects) contained in 

each. Included are those 3,178 data found as listed in Table 5, by SpaceWorks, 2012 (31) 
Page 20 of 88 

Table 5 

Data Sources Used in this Research (31)

Short Name Description Year Entries 

ETDP NASA Exploration Technology Development Program 2012 42 

ESTO NASA Earth Science Technology Office 2012 138 

SBIR Current NASA SBIR Technologies 2012 51 

GCD Game Changing Development Program Office Technology 

Projects 2012 35 

ESMD NASA Exploration Systems Mission Directorate ESAS 

Technologies 2005 304 

Tech Tool Box ATLAS (Advanced Technology Lifecycle Analysis System) 2002 6 

Ext Tech Data Tauri Group research into External Government Technologies 

(EGT within the MATCH relational database) 2012 28 

Cx ETDP Constellation Exploration Technology Development Program 2007 19 

NTID NASA Tech Inventory Database 2004 991 

Hist SBIR STTR Historical SBIR and STTR data 2012 1191 

RLV Tech DB NASA RLV Technology Database 1993 64 

HEOMD 

TechDev Team Estimates of HEOMD needed technology developments 2012 78 

ETDP from 

MATCH Mapping Applicable Technology To Exploration Challenges 2007 21 

External from 

MATCH Mapping Applicable Technology To Exploration Challenges 2007 192 

JSS TBCC NASA/Air Force Joint Systems Study TBCC Technologies 2010 18 

SpaceWorks, 2013, provided a frequency analysis of 643 technology projects investigated for all 

technology areas, Figure 6 to identify if the source of data were rich with the 18 parameters of interest, 

and then only for the four selected technology areas were presented in Figure 7. 
Page 21 of 88 

Figure 6. Data Frequency – 20 Parameters 

(Figure from SpaceWorks, (31) 2013) 

Figure 7. Data Richness - Data in four Technology Areas (3, 4, 8 & 12) and 18 Parameters 

 (Figure from SpaceWorks, (31) 2013) 

As indicated in Figures 6 and 7, many parameters were found to be absent in the historical data 

for all the data sources investigated and specifically these included: team experience, system hierarchy, 

hardware vs. software, R&D3

, AD2, push vs. pull, evolutionary vs. revolutionary, and unplanned events. 
Page 22 of 88 

Upon completion of an evaluation by the Research Team of all the data sources, it was revealed 

that the SBIR Phase III, ETDP, ESTO and GCDPO programs retained the highest potential sources of 

data to be used for retrospective data generation to be used in the research and for evaluation of the 20 

parameters for possible CER development. This was due to the nature that much of the project 

information was available on-line or as a downloaded document with cost and schedule data furnished to 

the Research Team. These four sources of data for these projects were identified in Table 6. These 

specific programs were found to have many of the data and parameters of interest to the research which 

included published: cost and schedule data, TRL, project descriptions, technology area identified, and the 

names of Principal Investigators or Project Managers and contributor contractors. Initially, 229 

technology projects were identified individually as containing potentially high value data. Multiple 

contacts were made with technologists where the Technology Managers reside to request assistance for 

their support to supply the needed data. 

Table 6 

Sources of Technology Projects TRL 2 to 6 Used for Evaluation of the 20 Parameters 

SBIR Phase III, EDTP, ESTO and GCPDO Programs

Funding 

Program 

Media Point of Contact Notes 

SBIR Phase II 

& III 

List from 

Electronic 

Handbook 

Provided by Mederos, L. and Jahns, G. 

C. 

Cost and schedule 

data provided 

 

ETDP CD Fitzgerald, J. and Law, R. C. provided 

duplicate copy of the ETDP information. 

Formal copy of the data was included in 

WindchillTM as indicated by Hall, K. L. 

but not readily accessible. 

Partial cost and 

schedule data 

available. ETDP 

data was headed to 

National Archives. 

 

ESTO On Line On Line at 

http://www.estotechnology.us/techportfo

lio/ and Komar, G. J. 

Cost and Schedule 

data provided 

separately. 

 

Game 

Changing 

Development 

Program 

Office 

NX Koca, M. E. and Brooks, C. E. Cost and schedule 

data available. 

Beginning May 21, 2013, 178 of the Technology Managers for 229 targeted Technology Projects 

were provided a data collection sheet to be used for retrospective data generation. The balance of the 51 

technology projects were found to have Technology Managers who were no longer associated with the 

project or not found, or no records were found to be available to make for a viable data collection 

response making these data of no value. As of June 20, 2013, 37 data collection sheets were prepared by 

the Technology Managers and returned to the Research Team. A copy for each of the electronic data 

collection sheets for responses received from the Technology Manager are included in Appendix D. 

Where telephone contact with the Technology Managers were necessary to aid in the completion of the 

data collection sheets, GAO structured interviewing techniques and guidelines (28) were implemented to 

help provide explanations that were uniform and to provide clear explanations or instructions to avoid 

disruption of data quality, avoid introduction of bias, or to not interfere with the cognitive decision 

process of the Technology Manager. (28-29)
Page 23 of 88 

An informal survey of the Technology Managers, who provided their input to the data collection 

sheet, or their facilitators related to the data collection activity generally, indicated the following: 

 “This is a very important process, and we need it”, 

 “Insufficient time to accomplish this data collection effort with all the other work being 

done”, 

 “It takes from one to two hours to complete the data collection sheet for a project”, 

 “The technology project is old and I cannot remember, or files not available”, 

 “The project is 10 years old”, and 

 “A WBS will be needed to complete the data collection sheet”. 

Data collection from established sources in this research proved exceedingly difficult and the 

uniform documentation of the data in a useable format overall was found not well established. A notable 

exception, the data acquired from the SBIR Electronic Handbook, was superior to this research in that it 

contained a systematic categorization of the many parameters of interest and the data were presented on 

an Excel® spreadsheet. The Game Changing Development Program Office data were found to contain the 

highest level of parameters of interest to the Research Team and the technology project data were found 

archived as slides in MicrosoftTM Power Point® on LaRC’s NX on-line project archival system. The 

ETDP data was found to be located on NASA’s Windchill® system and designated to be headed for the 

National Archives. An attempt was made to intercept it for use in this research and fortunately, a copy of 

the ETDP data was made available to the Research Team before its’ archival. The ESTO project 

information was rich in data and found on an on-line web site quad sheets in AdobeTM PDF® format. 

These ESTO project data were then converted to Microsoft Word® documents for extraction to the 

Integrated Data Base. 

Appendix B of the Parametric Handbook identifies that data should be collected for all 

technology projects centrally and maintained in a manner that provides a complete audit trail with 

dates, so that cost can be adjusted for inflation. (21) A good example for how project metrics are 

documented for large mission level projects was NASA’s ONCE and CADRe. It was unfortunate 

however that the ONCE/CADRe system did not afford a suitable data base for this research. ONCE and 

CADRe is supported by requirements included with NASA policy for NASA Procedural Requirements 

(NPR) 7120.5 projects in excess of $250 million. It is recognized that Technology projects of TRL range 

of 2 to 6 are below that dollar threshold and may be managed at a different level under multiple funding 

programs. Each of these lower TRL technology projects has been founded on programs that have different 

documentation requirements and none were required to be centrally located in a data base. It was also 

observed that Technology Projects with TRL < 6 in this research do not normally have a work breakdown 

structure in which data can be acquired. Further, programmatic information related to skill of the 

technology team, infrastructure requirements, delay(s) to the project, significant change(s) in the project, 

and tracking the project were obscure. 

3. Pre-Completed Data 

Contact with several NASA organizations revealed that the time for generation of the 

retrospective project data into the data collection sheet by the Technology Manager was a perceived 

burden to their time applied to their required duties. In an effort to help reduce this time burden, the 

Research Team attempted to obtain total project costs and the beginning and end dates for the project 

from the respective funding organization. In addition, an effort was made to retrieve the technical paper(s) 
Page 24 of 88 

published for 82 technical projects. The technical paper(s), where found were researched to determine if 

they contained any one of the 20 project parameters to be evaluated for characterization of the 

technology. The information contained in the technical paper(s) was then translated back to the project 

Data Collection sheets along with the cost and schedule information as provided. The Technology 

managers were then asked to back check the pre-completed data collection sheet for accuracy. 

 

It is important to note that the NASA Aeronautics and Space Database was the primary research 

source for investigating technical publications related to the technology project along with the LaRC 

OCIO Technical Library resources and E-Databases. At times, this effort was augmented by courtesy use 

of the Old Dominion University Library on-line resources. 

The following was learned from this pre-completion effort for the data collection sheet: 

 The technology presentation for each project was researched. Some technology projects only had a 

power point presentation associated with the technology project as if they had been given to a 

professional technical group or symposium. The value for obtaining parameters from this source was 

found to be of low or no/questionable data value, and the data collection sheet was not pre-completed. 

 The technology paper for each project was researched. Technology projects at times had one or more 

technical papers associated with the technology project. Most papers were found to consistently 

contain rich information with respect to the 20 parameters of interest. The value for obtaining data for 

the estimating parameters and retrospectively generating the data into the data collection sheet was 

found to be of high to medium data value. 

 During the research, the technical papers did not always possess the same title as the funded research 

project title. 

 Tracking the technical paper to the parent funded research project title was at all times not possible. 

The funding contract number did not appear in the technical paper. 

 A question was identified that could not be resolved within the scope of this effort to understand if a 

series or successive technical papers were connected to the parent funding contract. Currently, 

progression of the technology could only be tracked by the technical paper’s publication date. It was 

unresolved in this limited time to determine if the primary technology branched into other technology 

applications or Technology Areas. 

 The TRL progression and cost per year was input as an estimate to the pre-completed Data Collection 

sheets by the Research Team as a straight-line interpolation, 

 Review of the technical papers did not reveal if a cost was associated with use of a facility for the 

project. When costs are reported in pre-completed data collection sheets for use of facility, they were 

estimated by the Research Team. 

 Review of the technical papers did not reveal how many Full Time Employees (FTEs) were included 

for a technology. The numbers of FTEs were estimated on the Data Collection Sheets by the Research 

Team based on the number of technical paper contributors. 
Page 25 of 88 

 Typically, the pre-prepared data collection sheet, if not checked or confirmed by the by the 

Technology Manager, was ultimately identified to be of “no/questionable value” data and was not 

used. 

 Between the initiation of data collection in March, and through June 2013, approximately 17% of the 

Technology Managers responded even though the Data Collection Sheets were pre-completed for 

their review and back-checking. By Mid-September 2013, the response rate was approximately 31%. 

The participation by the Technology Managers was gained from direct contact via emails and phone 

calls. 

4. Data Quality 

Data to be used in the research must be satisfactory for use in the development of the cost 

estimating relationships that are included in the parametric analysis. Multiple screening criteria were 

established in a written document titled Technology Estimating Data Quality and Screening Methodology

and the data was checked for general accuracy. The Data Quality and Screening Methodology document 

has been included in Appendix E. 

5. Parameters Evaluation and Screening 

During the Phase I workshop, 20 parameters (See Figure 4) were established for the 

characterization of Technology projects. These parameters were classified into categories for Project 

characterization and results, development metrics, project execution, and programmatic factors. 

Recognizing the variability of technology, and the goal of determining which one of these parameters 

drive cost an effort was made to understand and confirm what it is we are trying to accomplish. The 

Research Team focused on a path to understand what the fundamental drivers to technology cost and 

schedule were and how best do we help assign these drivers a place in the parametric analysis to provide 

an estimate and schedule. 

 The parameters were evaluated to determine if they retained either a qualitative or quantifiable 

objective and the results are provided in Table 7. The qualitative and quantitative aspect of each 

parameter was analyzed with a discussion indicated. 
Page 26 of 88 

Table 7 

Technology Characterization Parameters – Analysis 

Project 

Characterization 

Parameter Value Metric 

(Quantifiable 

or 

Qualitative) 

Discussion 

Technology Area (TA) 1 through 15 Quantifiable Easy parameter to assess for first level TA, however the secondary, tertiary 

levels of TA were at times difficult to apply. 

System Hierarchy Hardware/Material, 

Component/Part, Assembly, 

Sub-system, System 

Qualitative The hierarchy was easy to apply and helps to classify the technology. 

Hardware versus 

Software 

All hardware/No software, High 

hardware/Low software, 

Medium hardware/Medium 

software, Low hardware/High 

software, No hardware/All 

software 

Qualitative The amount of hardware versus software was found difficult to quantify with 

great accuracy. The level of programming hours, lines of code, extent of 

computer applications must be addressed in order to determine its total cost 

driving implications to the Technology. Parameter should be expanded to 

include software as a separate parameter. 

Push versus Pull Either Qualitative The determination of the intent whether a project was push versus pull could 

not be fully discerned from the data. 

Evolutionary versus 

Revolutionary 

Either Qualitative The determination of the intent whether a project was evolutionary versus 

revolutionary could not be fully discerned from the data. 

Defining System 

Characteristics 

Description of project in terms 

of length, mass, volume, power, 

etc. 

Quantifiable Parameter was a significant contributor to the analysis as a means to describe 

the technology however at the level of limited data for a specific TA. 

Project Results

Key performance 

Parameter (KPP) 

Description of project 

performance or distinct 

capability 

Quantifiable The parameter was a significant contributor to the analysis as a means to 

describe the technology however at the level of limited data for a specific TA. 

Level of Effort Full time employees Quantifiable The parameter was considered a significant contributor to the research. 

Cost Total cost in dollars Quantifiable The parameter was considered a significant contributor to the research. 

Schedule Total project duration in years Quantifiable The parameter was considered a significant contributor to the research. 

Development Metrics

Technology Readiness 

Level (TRL) 

1 through 9 Qualitative The parameter is a significant contributor to the analysis however the TRL 

determination may is subjective. 

Research & Development 

Degree of Difficulty 

(R&D3

) 

1 through 5 Qualitative The parameter was found difficult to apply.

Advancement Degree of 

Difficulty (AD2

) 

1 through 9 Qualitative The parameter was found difficult to apply. 

Project Execution

Funding Sources Number and identification Qualitative The parameter was found easy to apply. 

Organizations Involved Identification of NASA 

organizations, Universities, 

contractors, etc. 

Qualitative The parameter was found easy to apply. 

Facilities and 

Infrastructure 

Description of laboratories, 

aircraft, or other special facilities 

required. 

Quantifiable The parameter was found easy to apply. 

Team Experience Little or no team experience, 

Experience conducting research 

but little or no experience with 

specific technology, Experience 

with research of this specific 

technology, Experience with this 

specific technology and 

innovative team 

Quantifiable The parameter was found easy to apply 

Programmatic Factors

Estimates versus Actuals Either Quantifiable The parameter was found easy to apply. 

Budget Constraints and 

Disruptions 

Description of any budget 

constraints or disruptions which 

may have influenced the project 

Qualitative The parameter was found easy to apply. 

Unplanned Events Description of any events which 

may have influenced the project 

Qualitative The parameter was found easy to apply. 

An analysis was conducted of the 20 parameters further using a weighting scheme and the results 

are presented in Figures 8a and 8b. 
Page 27 of 88 

Figure 8a. Parameter Analysis Using a Rating Scheme 

Figure 8b. Parameter Analysis Using a Rating Scheme 

Each parameter was individually evaluated and documented with an evaluation narrative and 

included in Appendix F. The evaluation conducted used the premise that a schedule and cost driver was a 

parameter that tends to increase or decrease schedule/cost or is highly correlated with schedule/cost, and 
Page 28 of 88 

to qualify as a cost driver, a parameter must be a significant predictor of schedule/cost at least in the 

statistical sense. The following is a detailed summary of the parameter evaluation: 

A statistical analysis was conducted to evaluate the parameters and the results are included in 

Figure 9. 

Figure 9. Analysis of Variance Results 

With the addition of retrospective data, additional analysis is expected to be conducted to 

determine the significance of the parameters to be evaluated further. 

Based on the analysis conducted, the parameters were screened and the recommended parameters 

were then identified in the Phase II Workshop. Based on the workshop results, the following parameters 

are recommended to be used for Technology Estimating, Figure 10, and those items identified with a 

“star” are already included in TechPort. 
Page 29 of 88 

Figure 10. Preliminary Recommended Parameters 

Several parameters were identified to be evaluated further. The parameter hardware versus 

software has been identified to have the hardware term eliminated and the software parameter term could 

include: 

1. Brief description of software and its use, 

2. Source lines of code, 

3. Function point count, 

4. Object class count, 

5. Number of screens, 

6. Language, 

7. Application type, 

8. Build environment, 

9. Target environment, 

10. Reliability, Maintainability, 

11. Degree of reuse or COTS, 

12. Number FTEs, and 

13. Team experience. 
Page 30 of 88 

Estimating Tool 

 Development of a tool to aid in the preparation of an estimate must include a sober consideration 

for its intended accuracy. Oberlander et al., 2001, identified that the accuracy of an estimate is measured 

by how well the estimated cost compares to the actual total installed cost. Oberlander’s research identified 

that the accuracy of an early estimate depends on four determinants: (1) who was involved in preparing 

the estimate: (2) how the estimate was prepared; (3) what was known about the project; and (4) other 

factors considered while preparing the estimate. Further, his research team defined the estimate as the 

point in time when the estimate was performed, i.e. estimate that has been prepared from inception of the 

project up to and including funding approval.(9) The Technology Estimating research has determined 

that the estimating process presented herein is initially conceived as a predictive estimate performed at 

the inception of the technology, and based on the level of confidence indicated with the number of 

similar projects included and with reference to their respective analogies of projects. Ultimately, it is 

desired that the process will lead to development of Cost Estimating Relationships and Parametric 

analysis to be developed later. 

The credibility of an estimate is a highly desirable but intangible characteristic that is often touted 

but can only be achieved through demonstrated performance or through verification or validation. 

Determining the credibility of an analogy based or parametric model whether (1) to verify the cost of a 

program for a higher command, or (2) to develop a cost estimate, or (3) to evaluate a cost proposal 

requires the same type of information. The responsible person, or group(s), accepting the output of the 

model must feel comfortable with the information being provided. Fad, et al. 1988, indicated that this can 

only come from a moderate understanding the model being used, information pertaining to its technical 

validity, confidence in the people using it, and the track record of both. (20) Colmer, et al., 1999, 

recognized that although cost estimation techniques can make forecasts using very little data, it is a 

mistake to place great reliance on these estimates without a firm grasp of the estimate limitations, 

assumptions, and risks. All too often a number (or cost estimate) created has a life of its own and 

continues to mislead. It should be borne in mind that new technologies have been developed before. 

Experience on how this affected cost is still relevant, especially as it is likely that some parts of the 

product will be little changed by these developments. A sufficiently long term historical perspective and a 

broad view of industry may well reveal trends and analogies of value to the estimator.(32) It is recognized 

in this research that development of a technology estimating process and tool will evolve to a higher 

levels of credibility once the commitment in this research continues and the level and number of data 

required on which the technology estimates are based becomes increasingly established. The high level 

of credibility anticipated to be required for users will continue as a function of the long term and 

perpetual commitment to data acquisition and its maintenance. 

The TCASE tool was developed in Microsoft Excel® from the modeling architecture functional 

diagram developed and presented in Figure 11. The tool allows the user to define a new technology 

project for the purpose of cost and schedule estimation by entering inputs for the different estimating 

parameters identified during the study. To familiarize the user with the tool, a User’s Manual is included 

in Appendix H. Based on these inputs; the tool presents analogous historical technology projects from the 

database of collected project data. From these analogies, a statistical distribution of cost and schedule 

predictions can be generated. 
Page 31 of 88 

Figure 11. Architecture of the Technology Estimating Tool TCASE 

(Figure from SpaceWorks, (33) 2013) 

The front end of the TCASE tool was updated to reflect the development of the analytical 

framework. The Project Inputs section reflects the latest technology cost and schedule estimating 

parameters, and the analogies section was expanded to include the parameters associated with the 

analogous technology projects. The front end is shown on Figures 12a and 12b. 
Page 32 of 88 

Figure 12a. Front End View of TCASE 

(Figure from SpaceWorks Phase II Workshop, (33) 2013) 

Figure 12b. Front End View of TCASE 

(Figure from SpaceWorks Phase II Workshop, (33) 2013) 
Page 33 of 88 

Importing and Exporting Technology Project Data Collection Worksheets was addressed by the 

Research Team in consideration of current and future applications for updating the TCASE with data. The 

data collection sheet can be used by the technologists for recording the parameters associated with their 

technology projects. In order to easily incorporate new data into the existing dataset, the processes of 

importing data from the datasheets to the database, and exporting data from the database to the datasheets, 

were automated through a set of Excel® VBA macros. 

To import data from a completed data collection sheet, the VBA macros parse the datasheet, 

collect the parameter data from the different fields, and store this data in memory. The macros then print 

the data into a new row of the database, with each field from the datasheet corresponding to a column of 

data in the database. To export data from the database, this process is reversed. The macros read the 

parameter data from a single row in the database, and print this data into the appropriate fields in the 

datasheet. The macros are designed to import and export multiple technology projects simultaneously. To 

import multiple datasheets, all of the datasheet files are placed in the same folder; the macros open every 

datasheet identified in the folder and print the parameter data to multiple rows in the database. Conversely 

if multiple projects are to be exported from the database, the macros read multiple rows from the 

database, create a new datasheet file for each row, and print the parameter data for the projects into their 

associated datasheets. 

 Calibration, verification, validation and accreditation of the model were addressed by the 

Research Team. As this research will introduce the viability of the technology estimating process and 

TCASE tool for only four TAs, full accreditation was not anticipated to be fulfilled. Verification and 

validation of the TCASE however will be accomplished by continued investigation of case studies.

Documentation on model verification and validation is usually critical in convincing users of the 

“correctness” of a model and its results, and should be included in the simulation model documentation.

Calibration is the process of adjusting a model’s parametric values to reflect an organization's cost and 

product history. The calibration process converts a general model into one developed exclusively for the 

organization’s application. Validation is the process or act of demonstrating a model’s ability to function 

as a credible forward estimating tool or system. Validation is performed for all parametric estimating 

techniques (e.g., CERs, models). (18)

Verification and validation steps generally considered and applied the following after Sargent, 

2000 to TCASE: 

1. Apply a test case as a study to identify that the cost and schedule estimates are at least bracketed 

within the range of costs for analogous projects. 

2. The amount of accuracy required of the model’s output variables for the model’s intended 

application is dependent upon the input data. 

3. Test, wherever possible, the assumptions and theories underlying the model. 

4. In each model iteration, perform at least face validity analysis of the conceptual model. 

5. In each model iteration, at least explore the model’s behavior using the computerized model. 

6. In at least the last model iteration, make comparisons, if possible, between the model and system 

behavior (output) data for several sets of experimental conditions. 

7. Develop validation documentation for inclusion in the simulation model documentation. 

8. Over a period of time, a schedule model upgrades will be ultimately developed for periodic 

review of the model’s validity. (34)

The TCASE analytics was based on work by Opricovic et al. (2006) VIKOR method for multicriteria decision making analysis which is an effective tool in multi-criteria decision making, particularly 
Page 34 of 88 

in situations where the decision maker is not able, or does not know to express his/her preference at the 

beginning of system design. The obtained compromise solution could be accepted by the decision makers 

because it provides a maximum group utility of the ‘‘majority’’, and a minimum individual regret of the 

‘‘opponent’’. The compromise solutions could be the base for negotiation, involving the decision makers’ 

preference by criteria weights. (35)

The Exact Match parameter can be set to either yes or no. If set to yes, then that particular search 

term will be treated as a filter, meaning that any project in the database that does not contain that search 

term in the specified field will not be returned as a viable result. 

The Weighting parameter can be used to establish the relative importance of each search term 

(e.g. assigning a weighting = 1.0 for Title Keywords and a weighting of 2.0 for Primary TA would 

indicate that matching the Primary TA search term is twice as important as matching the Title Keywords). 

The two parameters listed under Search Settings (Minimum Score for Analogy Matching and 

Maximum Number of Analogies in Results) are optional parameters that can be used to limit the number 

of results returned by a particular search. Lowering the minimum score or raising the maximum number 

of analogies will increase the number of results returned, and vice versa. 

 

Findings 

The following findings were provided: 

1. A basic assumption taken in this research was that all technologies have inherent 

variability. 

2. Parametric estimating provides the best possible solution for satisfying this technology 

estimating research. 

3. The technologies of interest in this research study were TRL of range 2 to 6. 

4. Tracking technology in its early progressive stages along the path of development or where 

the early technology has branched to other Technology Roadmap Areas (TAs) would be 

paramount as a significant building block to Technology Estimating. 

5. Multiple cost estimating processes were identified and evaluated with respect to the 

advantages and disadvantages presented, and considering the scope of technology 

addressed, parametric estimating offers the best option for use in this research to provide 

cost estimates for technologies in the TRL range of 2 to 6. However, based on the extent of 

retrospective data received during this phase of research, the analogy based methods will 

be used until the data can be evaluated further for Costing Estimating Relationship (CER) 

development and parametric analysis using those CERs.

6. Four technology areas were identified to be investigated for Technology Estimating. 

7. Although it was recognized that technology risk is a driver to technology cost and schedule, 

due to the results of the Phase I Workshop it was determined that there would be a limited 
Page 35 of 88 

or no data captured in a meaningful way for retrospectively reporting the risk data in 

completed projects. Therefore, risk as a parameter for use in analyzing individual 

technology projects would not be addressed in this research. 

8. A significant focus in this research was to obtain historical project data, retrospectively 

generated, which would be used to help develop a key outcome of the research. This key 

outcome is a consensus in the cost and technology communities about those parameters 

which are determined to be key drivers for technology cost. Once this consensus is 

obtained, the parameters would be analyzed for development of cost estimating 

relationships. Once these relationships and parameters could be decided upon, then future 

collection of data related to these parameters can then be incorporated into ongoing 

databases and data collection efforts such as TechPort, One NASA Cost Engineering 

(ONCE) and Nasa Cost Instrument Model (NICM). 

9. Data should be collected for all technology projects centrally and maintained in a manner 

that provides a complete audit trail with dates, so that cost can be adjusted for inflation. 

10. The Technology Estimating research has determined that the estimating process presented 

herein is initially conceived as a predictive estimate performed at the inception of the 

technology, and based on the level of confidence indicated with the number of similar 

projects included and with reference to their respective analogies of projects. Ultimately, it 

was desired that the process will lead to development of Cost Estimating Relationships and 

Parametric analysis to be developed later.

11. It was recognized in this research that development of a technology estimating process and 

tool will evolve to a higher levels of credibility once the commitment in this research 

continues and the level and number of data required on which the technology estimates are 

based becomes increasingly established. The high level of credibility anticipated to be 

required for users will continue as a function of the long term and perpetual commitment 

to data acquisition and its maintenance. 

12. The Research Team identified gaps during the conduct of the research. These included: 

a. Cost and schedule data for some Technology Projects were not readily available in an 

accessible open source or not available at all. 

b. Research papers for the Technology project were attempted to be used to help satisfy 

Technology Estimating data collection effort. Technology Projects did not always have 

a research paper. 

c. Verifiable means to track a technical paper back to the funded research project was not 

found. 

d. Where multiple technical papers were produced, no means existed to verifiably track to 

the funded research project. 

e. Since this research was conducted primarily for completed projects, not all of the 

completed projects were assigned to a NASA Technology Roadmap Area. 

f. Tracking low TRL projects through the system to a higher level was not possible. 
Page 36 of 88 

g. Although TechPort was released during the course of the research, TechPort data 

could not be used exclusively as it understandably had not been backfilled with 

completed projects. 

h. The SAP system was initiated on or about 2003. Cost data for Technology Projects 

before 2003 is not readily available. 

i. TechPort could be optimized further to include additional parameters data, requiring a 

log in to view the specific parameter. 

13. Improvements to the TCASE tool may include: 

a. An investigation should be made to determine if it would be useful to have the option of 

selecting one analogy and pulling up all of its available data into one tab/sheet rather 

than using its ID to search the database. The Research Team will investigate to 

determine if this data base query capability for one analogy can be provided in future 

version of the tool. 

b. Annotating the percentage as a score of parameter “hits” as relevance, and/or 

c. Listing the parameters with a ranking for each or say using a top to bottom score for 

each parameter in the analogous project. 

d. System Hierarchy for the project as the system level down approach to the 

Hardware/Material level was addressed however, some of the data in the TCASE model 

appears inconsistent with this breakout. Some projects seem to include much more 

cost beyond just an integrated system/spacecraft level. The TCASE should be refined 

further with respect to the data according to the system Hierarchy.

e. The Research Team should identify if there is a way to identify the technology data 

that’s been validated for cost, schedule, TRL, and other parameters. 

f. The Research Team will verify that the tool data base does not have duplicates. 

g. The Research Team will verify that TRLs listed in the data base appear correct and 

that lower TRL at the end of the project are consistent with the actual project results, 

or the project will be eliminated from the data base. 

Suggestions 

The following next steps are suggested and include follow-on research: 

(Year 2013 to 2014)

– Continue push to collect any remaining retrospective data from historical NASA technology projects. 

– Data Analysis to consolidate parameters selected for use. 

– Further TCASE Model development in the four TAs. 

– Investigate use of TechPort for data repository. 

– Begin analysis of Data for the remaining TAs. 

(Near Term)

– Refine Data /Quality Requirements. 

– Continue TCASE Model use within NASA Cost Community for Beta Testing. 

– Develop CERs for use in Parametric Analysis. 

– Initiate Technology Estimating data input requirements /management requirements to TechPort. 
Page 37 of 88 

(Long Term)

– Include balance of Technology Areas (1, 2, 5, 6, 7, 9, 10, 11, 13, and 14). 

– Configure data pull from data sources such as TechPort. 

– Maintain and Update Data and TCASE Model. 

– Refine tool for NASA-wide deployment and use by Technologist, Management, and Projects. 

Conclusion 

The Technology Estimating process has been evaluated to determine its feasibility for providing a 

means to estimate technology. Based on the preliminary research, the process was determined to be a 

viable means to help determine the cost and schedule of technology for four of the 14 NASA Roadmap 

Technology Areas if the data was centralized, verifiable, and maintained. The research for Technology 

Estimating should continue to help refine and expedite the initial efforts initiated in this research. 

The following conclusions have been derived from the research: 

1. This Technology Estimating Research project addresses a gap in the NASA cost estimating 

capabilities for technologies in the range from TRL 2 to 6. 

2. The data collection work in the research has yielded valuable information from available sources, 

although only for a small subset of targeted parameters. These initial results have afforded the 

development of a useful analogy based estimation tool. However, retrospective data collection 

has been challenging and efforts are continuing, to enable rigorous analysis of parameters. 

3. A preliminary recommendation has been identified for data parameters to be used in the 

development of CERs to be applied to cost estimating using parametric analysis. 

4. The future goal, pending further analysis of all parameters, will lead to CER development and 

evolving capabilities from an analogy based cost estimating tool now to a parametric analysis tool 

for the initial four technology areas investigated, i.e. TA3, TA4, TA8 and TA12. 
Page 38 of 88 

Acknowledgements 

This work was performed as collaboration between the NASA HQ Office of Evaluation, Cost Analysis 

Division, the Game Changing Development Program Office in the Space Technology Mission 

Directorate, LaRC Systems Analysis & Concepts Directorate, and the LaRC Research Directorate to 

fulfill the need for meeting NASA’s goals, furthering technological advances and research, and promoting 

opportunities for inter-departmental working relationships. The work was also performed in cooperation 

with the NASA HQ Office of the Chief Technologist and Center Chief Technologists who helped with 

support to data acquisition. The Cost Analysis Division, the Game Changing Development Program 

Office funded a portion of this research under LaRC contract number NNL11AA05B, Order # 

NNL12AC80T, Req. # 4200449610 to SpaceWorks Enterprises Inc. (SEI), http://sei.aero. The LaRC 

Systems Analysis & Concepts Directorate is acknowledged for research support by providing one quarter 

full time employee, and the LaRC Research Directorate supported the research by a providing a full time 

employee, for the duration of this research. Special acknowledgement is given for the inter-agency and 

non-governmental participation and contributions made by DARPA SBIR-STTR Programs Office, U.S. 

Air Force Cost Analysis Agency-Space Division, Office of the Director of National Intelligence 

Affordability and Cost Research, National Institute for Aerospace- Georgia Tech., the Foundation for 

Enterprise Development, California Institute of Technology Jet Propulsion Laboratory, Artemis 

Innovation Management Solutions LLC, Aerospace Corp., JB Consulting International, The Tauri Group, 

and Old Dominion University. 
Page 39 of 88 

References 

1. NASA Strategic Space Technology Investment Plan, NASA, December 5, 2012. 

2. NASA Space Technology Roadmaps and Priorities: Restoring NASA's Technological Edge and 

Paving the Way for a New Era in Space, Steering Committee for NASA Technology Roadmaps; 

National Research Council of the National Academies, The National Academies Press, 500 Fifth 

Street, NW Washington, DC, ISBN 978-0-309-25362-8, 2012. 

3. Bob Sefcik, Glenn Research Center, Chris Blake, Booz Allen Hamilton, Technology 

Advancement: Estimating the “Soft” Side 2011 NASA Cost Analysis Symposium August 17, 

2011. 

4. What is Technology Estimating, a proposal description, by the NASA Office of Evaluation, Cost 

Analysis Division, November 2012. 

5. Systems Engineering Handbook, NASA/SP-2007-6105 Rev1, National Aeronautics and Space 

Administration, NASA Headquarters, Washington, D.C. 20546, December 2007. 

6. Boer, F. P., The Valuation of Technology, Business and Financial Issues in R&D, ISBN 0-471-

31638-5, John Wiley & Sons, New York, 1999. 

7. Howey, R. A., Valuing Technology, Knowledge, Technology & Policy, Fall/Winter, 2004, Vol. 

17 Issue 3/4, p44-64., 21p, 2005. 

8. Beltramo, M. N., Beyond Parametrics: The Role of Subjectivity in Cost Models, Engineering 

Costs and Production Economics, 14 (1988) 131-136 131, Elsevier Science Publishers B.Y., 

Amsterdam, Netherlands, 1988. 

9. Oberlender, G. D., and Trost, S. M., Predicting the Accuracy of Early Cost Estimates Based on 

Estimate Quality, Journal of Construction Engineering and Management, Vol. 127. No.3. 

May/June. 2001. ASCE. ISSN 0733 9634/0110003-0173-0182, Paper No. 22225, June 2001. 

10. Hazelrigg, G. H., and Greenberg, J. S., Cost Estimating for Technology Programs, National 

Science Foundation, Washington, D.CIAA-91-638, 42nd Congress of the International 

Astronautical Federation, October 5-11, 1991, Montreal, Canada, 1991. 

11. Roy, R., Colmerb, S, and Griggs, T., Estimating the Cost of a New Technology Intensive 

Automotive Product: A Case Study Approach, Int. J. Production Economics, 97 (2005) 210–226, 

2005. 

12. Ingalls, R .G., Rossetti, M. D., Smith, J. S., Peters, B. A., eds. Watson, R. J., Kwak, Y. H., 

Development of a Parametric Estimating Model for Technology-Driven Deployment Projects, 

Proceedings of the 2004 Winter Simulation Conference School of Business, The George 

Washington University, Washington, DC 20052, U.S.A WSC '04 Proceedings of the 36th 

conference on Winter simulation, pg. 1290-1295, Winter Simulation Conference, ISBN:0-7803-

8786-4, 2004. 

13. Cyr, K. J., Cost Estimating Methods for Advanced Space Systems, SAWE Paper No. 1856, 

Society of Allied Weight Engineers 47th Annual Conference, Plymouth, Michigan, May 23-24, 

1988. 

14. Hess R., Nelson, J.R., Killingsworth, P., and Resetar, S.A., Estimating the Cost of Advanced 

Technology, 30th Annual Joint National Meeting of the Operations Research Society of America 

and the Institute of Management Sciences, October 29-31, 1990, in Philadelphia, PA, ISBN 0-

8330-2044-7, Rand Doc #P-7732, 93 pp, 1990. 
Page 40 of 88 

15. Shishko, R., Ebbeler, D. H., Fox, G., NASA Technology Assessment Using Real Options 

Valuation, Jet Propulsion Laboratory, California Institute of Technology, 4800 Oak Grove Drive, 

Pasadena, CA 91109, July 7, 2003 Published online Wiley InterScience 

(www.interscience.wiley.com), DOl I O.lOO2lsys.1 0052, Systems Engineering, Vol. 6, No.4, 

2003. 

16. Dumas, L. J., Parametric Costing and Institutionalized Inefficiency, AII Transactions, Vol. II, No. 

2, AIIE Institute Conference May 1978, pp 147-154, 1979. 

17. Cavalieria, S., Maccarroneb, P., Pintoa, R., Parametric vs. Neural Network Models for the 

Estimation of Production Costs: A case study in the automotive industry, Int. J. Production 

Economics, 91 (2004) 165–177, 2003. 

18. Parametric Estimating Handbook, Fourth Ed., April 2008, International Cost Estimating and 

Analysis Association, ISBN # 0-9720204-7-0ISPA/SCEA, Vienna, VA, 2008. 

19. Thibault, M., Cost-Estimating Relationships: A DCAA Perspective, Progress in Astronautics and 

Aeronautics 144 (1992): 89-95, 1992. 

20. Fad, B. E., and Summers, R. M., Parametric Estimating for New Business Ventures, Engineering 

Costs and Production Economics, 14 (1988) 165-176 165, Elsevier Science Publishers B.Y., 

Amsterdam , Netherlands, 1988. 

21. Federal Acquisition Regulation 15.404-1, pkisupport@gpo.gov, U.S. Government Printing 

Officehttp://www.gpo.gov/fdsys/pkg/CFR-2002-title48-vol1/pdf/CFR-2002-title48-vol1-sec15-

404-1.pdf 

22. Space Systems-Programme Management-Breakdown of Project Management Structures, 

International Standard, ISO 27026, First edition, 2011-02-15, 2011. 

23. Statement of Dr. Mason Peck Chief Technologist National Aeronautics and Space 

Administration before the Subcommittee on Space and Aeronautics, Committee on Science, 

Space and Technology, United States House of Representatives, July 12, 2012. 

24. Smith, J. H., and Weisbin, C., Beyond Global Project Management: Technology R&D Planning 

for Interplanetary Projects, Pan Pacific Conference XXI International Business and Global 

Project Management Session on Global Project Management Anchorage, Alaska May 26-28, 

2004 Jet Propulsion Laboratory California Institute of Technology power point and paper, 2004. 

25. Weisbin, C. R., Rodriguez, G., Elfes, A., and Smith, J. H., Toward a Systematic Approach for 

Selection of NASA Technology Portfolios, Jet Propulsion Laboratory, California Institute of 

Technology, 4800 Oak Grove Drive, Pasadena, CA 91109, 24 May 2004, Published online in 

Wiley InterScience, (www.interscience.wiley.com) DOI 10.1002/sys.20010, 2004. 

26. Wilhite, A., Odom, P., Lovell, N., Lord, R., Estimating the Risk of Technology Development, 

Document ID (CASI) 20040000607, NASA Aeronautics and Space Database, 2003. 

27. Bilbro, J.W., A White Paper - Risk vs. Uncertainty - Cost Risk vs. Program Risk, JB Consulting 

International, jbci@knology.net, 2012. 

28. Meisel, C. J., Techniques for Cost estimating in Early Program Phases, Engineering Costs and 

Production Economics, 14 (1988) 95-106, Elsevier Pub., B.V. Amsterdam, Printed in the 

Netherlands, 1988. 

29. Using Structured Interviewing Techniques, Program Evaluation and Methodology Division, 

USGAO, PEMD-10.1.5, June 1991. 

30. Cannell, C. F., A Summary of Studies of Interviewing Methodology, Data Evaluation and 

Methods Research, Series 2, Number 69, U.S. Department of Health, Education & Welfare, 
Page 41 of 88 

Public Health Service, DHEW Publication No. (HRA) 77-1343, Rockville, Md., ISBN 0-8406-

0062-3, March 1977. 

31. SpaceWorks Enterprises, Inc., Technology Cost & Schedule Estimation, Quarterly Report, 

Independent Assessments for Systems Analysis and Concepts Development Task 05: Technology 

Cost & Schedule Estimation (TCASE) Tool Development, Subtask #2 Report, Milestone 2, 

February 2013. 

32. Colmer, G., Dunkley, M., Cray, K., Pugh, P., and Williamson, A., Estimating the cost of new 

technology products, Int. J. Technology management, Vol. 17, No.s 7/8, pp 840-846, 1999. 

33. SpaceWorks Enterprises, Inc., Technology Cost & Schedule Estimation, Quarterly Report, 

Independent Assessments for Systems Analysis and Concepts Development, Task 05: 

Technology Cost & Schedule Estimation (TCASE) Tool Development, Subtask #2 Report, 

Milestone 3, May 2013. 

34. Sargent, R. G., Verification, Validation, and Accreditation of Simulation Models, Proceedings of 

the 2000 Winter Simulation Conference Simulation Conference, 2000. Proceedings, Volume: 1

02/2000; DOI:10.1109/WSC.2000.899697, ISBN: 0-7803-6579-8, 2000. 

35. Opricovic, S., Tzeng, G., Extended VIKOR method in comparison with outranking methods, 

European Journal of Operational Research, 178 (2007), 514-529, 2006. 
Page 42 of 88 

APPENDIX A 

Definitions 
Page 43 of 88 

TECHNOLOGY ESTIMATING RESEARCH 

PROJECT 

Introduction and Definitions 

Date: March 18, 2013; Revised June 21, 2013 

The NASA HQ, Cost Analysis Division is performing research to improve the capabilities for estimating 

the cost and schedule of TRL 2 to 6 technology projects. For this research, the Research Team has helped 

to define the following 20 parameters. 

PARAMETERS 

1. Start Date and End Date 

 The beginning and end date of the project. These dates will be used to determine the duration in years 

and months of the technology development project. 

2. Technology Readiness Level (TRL) 

A systematic measurement system that supports assessment of the maturity of a particular technology and 

the consistent comparison of maturity between different types of technologies.(1) The Technology 

Readiness Level (TRL) describes the stage of maturity in the development process from observation of 

basic principles through final product operation. The exit criteria for each level documents that principles, 

concepts, applications or performance have been satisfactorily demonstrated in the appropriate 

environment required for that level. TRL is measured on a scale of increasing maturity from 1 to 9 as 

shown in the following Table 1. (2)

Technology Readiness Level 

Table 1 

TR

L Definition Hardware Description Software Description Exit Criteria 

1 

Basic 

principles 

observed and 

reported 

Scientific knowledge 

generated underpinning 

hardware technology 

concepts/applications. 

Scientific knowledge 

generated underpinning basic 

properties of software 

architecture and 

mathematical formulation. 

Peer reviewed 

publication of 

research underlying 

the proposed 

concept/application. 

2 

Technology 

concept and/or 

application 

formulated 

Invention begins, 

practical application is 

identified but is 

speculative, no 

experimental proof or 

detailed analysis is 

Practical application is 

identified but is speculative, 

no experimental proof or 

detailed analysis is available 

to support the conjecture. 

Basic properties of 

Documented 

description of the 

application/concept 

that addresses 

feasibility and 

benefit. 
Page 44 of 88 

available to support the 

conjecture. 

algorithms, representations 

and concepts defined. Basic 

principles coded. 

Experiments performed with 

synthetic data. 

3 

Analytical and 

experimental 

critical 

function 

and/or 

characteristic 

proof-of 

concept 

Analytical studies place 

the technology in an 

appropriate context and 

laboratory 

demonstrations, 

modeling and simulation 

validate analytical 

prediction. 

Development of limited 

functionality to validate 

critical properties and 

predictions using nonintegrated software 

components. 

Documented 

analytical/experiment

al results validating 

predictions of key 

parameters. 

4 

Component 

and/or 

breadboard 

validation in 

laboratory 

environment 

A low fidelity 

system/component 

breadboard is built and 

operated to demonstrate 

basic functionality and 

critical test 

environments, and 

associated performance 

predictions are defined 

relative to the final 

operating environment. 

Key, functionally critical, 

software components are 

integrated, and functionally 

validated, to establish 

interoperability and begin 

architecture development. 

Relevant Environments 

defined and performance in 

this environment predicted. 

Documented test 

performance 

demonstrating 

agreement with 

analytical predictions. 

Documented 

definition of relevant 

environment. 

5 

Component 

and/or 

breadboard 

validation in 

relevant 

environment 

A medium fidelity 

system/component 

brassboard is built and 

operated to demonstrate 

overall performance in a 

simulated operational 

environment with 

realistic support 

elements that 

demonstrates overall 

performance in critical 

areas. Performance 

predictions are made for 

subsequent development 

phases. 

End-to-end software 

elements implemented and 

interfaced with existing 

systems/simulations 

conforming to target 

environment. End-to-end 

software system, tested in 

relevant environment, 

meeting predicted 

performance. Operational 

environment performance 

predicted. Prototype 

implementations developed. 

Documented test 

performance 

demonstrating 

agreement with 

analytical predictions. 

Documented 

definition of scaling 

requirements. 

6 

System/subsys

tem model or 

prototype 

demonstration 

in a relevant 

environment 

(ground or 

space) 

A high fidelity 

system/component 

prototype that 

adequately addresses all 

critical scaling issues is 

built and operated in a 

relevant environment to 

demonstrate operations 

under critical 

Prototype implementations 

of the software demonstrated 

on full-scale realistic 

problems. Partially integrate 

with existing 

hardware/software systems. 

Limited documentation 

available. Engineering 

feasibility fully 

Documented test 

performance 

demonstrating 

agreement with 

analytical predictions. 
Page 45 of 88 

environmental 

conditions. 

demonstrated. 

7 

System 

prototype 

demonstration 

in a space 

environment 

A high fidelity 

engineering unit that 

adequately addresses all 

critical scaling issues is 

built and operated in a 

relevant environment to 

demonstrate 

performance in the 

actual operational 

environment and 

platform (ground, 

airborne, or space). 

Prototype software exists 

having all key functionality 

available for demonstration 

and test. Well integrated 

with operational 

hardware/software systems 

demonstrating operational 

feasibility. Most software 

bugs removed. Limited 

documentation available. 

Documented test 

performance 

demonstrating 

agreement with 

analytical predictions. 

8 

Actual system 

completed and 

“flight 

qualified” 

through test 

and 

demonstration 

(ground or 

space) 

The final product in its 

final configuration is 

successfully 

demonstrated through 

test and analysis for its 

intended operational 

environment and 

platform (ground, 

airborne, or space). 

All software has been 

thoroughly debugged and 

fully integrated with all 

operational hardware and 

software systems. All user 

documentation, training 

documentation, and 

maintenance documentation 

completed. All functionality 

successfully demonstrated in 

simulated operational 

scenarios. Verification and 

Validation (V&V) 

completed. 

Documented test 

performance 

verifying analytical 

predictions. 

9 

Actual system 

“flight 

proven” 

through 

successful 

mission 

operations 

The final product is 

successfully operated in 

an actual mission. 

All software has been 

thoroughly debugged and 

fully integrated with all 

operational 

hardware/software systems. 

All documentation has been 

completed. Sustaining 

software engineering support 

is in place. System has been 

successfully operated in the 

operational environment. 

Documented mission 

operational results. 

Note: The Technology Readiness Assessment Tool (TRAT, can be found at https://trl.msfc.nasa.gov/), 

developed by NASA Marshall Space Flight Center is available. If desired, the tool can assist in 

conducting an assessment of the beginning and ending TRL. 
Page 46 of 88 

3. Total Cost 

Total dollars required to complete the technology development project. This data will be provided by 

year. Cost represents total cost of labor, materials, travel, testing and equipment, etc. and should also 

include (and separately identify) facilities and infrastructure capital investments made as part of the 

research project (if any). 

4. Estimate or Actuals 

A categorization of the data. 

 Actuals: Data collected from realized, historical technology development projects 

 Estimate: Data from studies, projections, and other sources that have not yet been realized 

5. Level of Effort 

A measure (in total work-years) of personnel workload required in terms of Full Time Equivalents (FTE) 

for civil servants, or Work Year Equivalent (WYE) for contractors. One work-year represents a single 

project staff member (either civil servant or contractor) working full time on the project for one year. For 

example, if a project has two individuals working full time during a three month period of performance, 

then the project has 0.5 work-years. 

6. Key Performance Parameter (KPP) 

Those capabilities or characteristics associated with the technology that is considered to be the primary 

objective of the technology development effort. Technology development programs seek to improve the 

current state-of-the-art, so the advancement of a program can be measured by the percentage 

improvement in one or more KPPs.(3) 

KPP information can include: 

 Description of KPP and units 

 Current value for state-of-the-art at the start of the project 

 Demonstrated achievable value at the end of the project 

For example, a jet engine propulsion technology development activity might have the goal to improve 

Specific Fuel Consumption (SFC). Thus SFC would be a KPP for this technology, the units would be 

defined (for example, g/[N*s]), along with values for state-of-the-art at the beginning of the project, and 

the demonstrated achievable improved value at completion of the project would be entered. The data 

would then be used to determine the percent improvement in the KPP as a result of the technology 

development project. 
Page 47 of 88 

7. Performing Organization 

 A list of the names of the agencies, organizations, or companies conducting the technology development 

project. The number of organizations involved will be determined from this list and used as a data 

parameter. 

8. Team Experience 

Experience of the team conducting the research may be a big factor in how long the project takes and 

what it will cost. 

Four categories of research team experience are identified, as defined below. Research team experience is 

to be assessed at the beginning of the technology project. The categories and definitions are: 

1. The team has little or no experience conducting research. 

2. The team has experience conducting research, but little or no experience with this specific 

technology. 

3. The team has experience conducting research on this specific technology 

4. The team has experience conducting research on this specific technology and ALSO has 

incorporated members from outside the field(s) of research who can bring new ideas and sources of 

innovation to the team. 

9. Technology Area (TA) 

The fourteen areas of space technology research being pursued within NASA, plus the fundamental 

research area of aeronautics. Each technology area has an associated development roadmap. The list of 

space technology areas and their supporting roadmaps was developed by NASA and reviewed and 

validated by the National Research Council (NRC). This process began in 2010 and was completed in a 

final report in early 2012. (4) The space Technology Areas have 3 levels, (5) which will be available to the 

Technology Manager in pull-down menus for entry in the data sheet. Aeronautics is normally not 

addressed within the Space Technology Roadmap TA taxonomy, but is included here for information as 

an additional ("+1") area to fully represent the technology research pursued at NASA. 
Page 48 of 88 

14+1 TECHNOLOGY AREAS 

Table 2

TA # Description 

TA01 Launch Propulsion Systems 

TA02 In-Space Propulsion Technologies 

TA03 Space Power and Energy Storage 

TA04 Robotics, Telerobotics, Autonomous Systems 

TA05 Communication and Navigation 

TA06 Human Health, Life Support, Habitation Systems 

TA07 Human Exploration Destination Systems 

TA08 Science Instruments, Observatories, Sensor Systems 

TA09 Entry, Descent, and Landing Systems 

TA10 Nanotechnology 

TA11 Modeling, Simulation, Information Tech 

TA12 Materials, Structures, Mechanical Systems, Manufacturing 

TA13 Ground and Launch Systems Processing 

TA14 Thermal Management Systems 

(+) 1 Aeronautics 

 

10. System Hierarchy 

The scope of the technology project in terms of its application of a vehicle or system. The project can be 

classified in one of the following five tiers (6) as shown on Table 3. If the technology is a process or 

software, the tier indicates at which level of the system hierarchy the process or software is applied. The 

tiers are: 
Page 49 of 88 

System Hierarchy 

Table 3 

No. Tier Definition Example 

1 System 

An integrated set of constituent 

elements that are combined in an 

operational or support environment to 

accomplish a defined objective 

A spacecraft or launch 

vehicle stage 

2 Subsystem A portion of a system 

A satellite’s propulsion 

system or launch vehicle’s 

propulsion system 

3 Assembly A set of components (as a unit) before 

they are installed to make a final 

product 

A satellite’s thruster or 

launch vehicle’s engine 

turbo-machinery 

4 Component / Part A portion of an assembly 

A satellite’s propellant valve 

or a launch vehicle’s engine 

injector 

5 Hardware / Material An item or substance used to form a 

component 

Alloy, polymer, screws, 

bolts, pipes, semiconductor 

chips 

11. Hardware vs. Software 

A top-level assessment of whether the end application of the technology is: 

 Hardware: The physical items necessary for conducting an activity, as distinguished from the 

theory and design that make the activity possible. Examples include mechanical equipment, tools, 

implements, instruments, devices, sets, fittings, trimmings, and assembled collections of those 

items. 

 Software: Computer programs, procedures, scripts, rules, and associated documentation and data 

pertaining to the development and operation of a computer system. Software includes programs 

and data. This also includes COTS, GOTS, MOTS, reused software, auto generated code, 

embedded software, firmware, and open source software components.(7)

12. Research and Development Degree of Difficulty (R&D3) 

A top-down assessment of the anticipated difficulty likely to be encountered over the course of a 

technology maturation project (e.g. from the beginning TRL to end TRL). (9) R&D3

 is a qualitative 

evaluation of the probability of success of the development project. R&D3

 is assessed on a scale of 

increasing difficulty from 1 to 5 as shown on Table 4: 
Page 50 of 88 

Research and Development Degree of Difficulty 

Table 4 

R&D3 Definition Probability 

of Success 

1 A very low degree of difficulty is anticipated in achieving research and 

development objectives for this technology. ≥ 95%-99% 

2 A moderate degree of difficulty should be anticipated in achieving R&D 

objectives for this technology. ≥ 90% 

3 A high degree of difficulty anticipated in achieving R&D objectives for this 

technology. ≥ 80% 

4 A very high degree of difficulty anticipated in achieving R&D objectives for 

this technology. ~ 50-60% 

5 The degree of difficulty anticipated in achieving R&D objectives for this 

technology is so high that a fundamental breakthrough is required. ≤ 30% 

13. Advancement Degree of Difficulty (AD2) 

A bottoms-up assessment of the anticipated difficulty over the course of a technology maturation project 

(e.g. from the beginning TRL to end TRL). (9) AD2

 is determined through consideration of cost, schedule, 

and risk across several dimensions, including: 

 Design and Analysis 

 Manufacturing 

 Software and Development 

 Test and Evaluation 

 Operations 

 AD2

 is intended to be determined using the AD2 calculator found at https://trl.msfc.nasa.gov/, or 

with a resulting value on a scale of increasing difficulty from 1 to 9 as shown on Table 5: 

Advancement Degree of Difficulty 
Page 51 of 88 

Table 5 

AD2 Definition Risk Category Success 

Chance 

1 Exists with no or only minor modifications being 

required. A single development approach is adequate. 0% Guaranteed 

Success 

2 Exists but requires major modifications. A single 

development approach is adequate. 10% 

3 Requires new development well within the experience 

base. A single development approach is adequate. 20% 

4 

Requires new development but similarity to existing 

experience is sufficient to warrant comparison across 

the board. A single development approach can be taken 

with a high degree of confidence for success. 

30% 

Well 

Understood 

(Variation) 

Almost Certain 

Success 

5 

Requires new development but similarity to existing 

experience is sufficient to warrant comparison in all 

critical areas. Dual development approaches should be 

pursued to provide a high degree of confidence for 

success. 

40% Known 

Unknowns 

Probably Will 

Succeed 

6 

Requires new development but similarity to existing 

experience is sufficient to warrant comparison on only a 

subset of critical areas. Dual development approaches 

should be pursued in order to achieve a moderate degree 

of confidence for success. (Desired performance can be 

achieved in subsequent block upgrades with high 

confidence. 

50% 

7 

Requires new development but similarity to existing 

experience is sufficient to warrant comparison in only a 

subset of critical areas. Multiple development routes 

must be pursued. 

70% 

8 

Requires new development where similarity to existing 

experience base can be defined only in the broadest 

sense. Multiple development routes must be pursued. 

80% Unknown 

Unknowns 

High 

Likelihood of 

Failure (High 

Reward) 

9 

Requires new development outside of any existing 

experience base. No viable approaches exist that can be 

pursued with any degree of confidence. Basic research 

in key areas needed before feasible approaches can be 

defined. 

100% Chaos 

Almost Certain 

Failure (Very 

High Reward) 

Note: If needed, the AD2

 Calculator is a tool that can assist the Technology Manager in conducting an 

objective assessment. The AD2 calculator is co-located with TRAT. 
Page 52 of 88 

14. Technology Push or Pull 

 Technology Push: Technologies that seek to meet long term goals by taking advantage of 

emerging or radically different ideas or approaches. Push technologies inspire new and different 

Missions and Mission Architectures to accomplish strategic long-term goals. (10)

 Technology Pull: Technologies that support or enable specific functions, capabilities or 

performance levels, for a defined mission, that is not achievable by proven, existent approaches. 

Pull technologies are motivated by the specific needs for missions, where the missions are 

developed by enterprise organizations to satisfy long-term strategic goals. (10)

15. Evolutionary or Revolutionary 

 Evolutionary: An incremental or stepwise improvement from a more mature, less advanced 

technology. 

 Revolutionary: A fundamentally new concept without a direct analogy, or a disruptive new 

innovation that is a significant advancement from an existing analogy. 

16. Funding Sources 

A list of the names of the organization(s), program(s), or contract(s) providing funding to the technology 

development project. 

17. Facilities and Infrastructure 

A list of the facilities and major infrastructure items required in the support of the technology 

development project. These could include: commercial or government laboratories, buildings, test stands, 

wind tunnels, etc. The number of facilities and infrastructure items will be determined from this list and 

used as a data parameter. 

18. Budget Constraints and Disruptions 

A list of the external constraints and disruptions to the resources required for the technology development 

project that may have negatively influenced the required project cost or schedule. For disruptions that 

influenced schedule, such as a break in funding between phases, the duration of these disruptions should 

be included. The number of total budget constraints and disruptions will be determined from this list and 

used as a data parameter. 

19. Unplanned Events 

A list of events outside of the control of the technology development project that had a negative effect on 

the required project cost or schedule. Examples include natural disasters, realignment of government or 

organization priorities, retirement or reassignment of key personnel, strikes, embargos, or testing 
Page 53 of 88 

incidents or accidents, etc. The number of total unplanned events will be determined from this list and 

used as a data parameter. 

20. Defining System Characteristics 

Technical parameters or characteristics that broadly define the scope of the technology project, but are not 

objectives of the technology development effort. For example, consider an energy storage technology 

development project with the objective of increasing Li-ion battery subsystem reliability to a goal value 

of 99.9%. In this case, parameters such as the energy density and the efficiency of the test battery are not 

direct objectives for improvement, but they may still be listed as Defining System Characteristics. (The 

battery subsystem reliability, on the other hand, should be listed as a Key Performance Parameter.) 

References 

1. Mankins, J., "Technology Readiness Levels: A White Paper," April 1995, NASA Office of Space 

Access and Technology, Advanced Concepts Office. 

2. “Technology Readiness Level (TRL) Descriptions”, Appendix B, Small Business Innovative 

Research (SBIR) Solicitation, referenced December 11, 2012, 

http://sbir.gsfc.nasa.gov/SBIR/sbirsttr2012/solicitation/forms/appendix_B.pdf

3. “NASA Systems Engineering Processes and Requirements”, NPR 7123.1A, Nov 2009 

4. Talbert, T., “Space Technology Roadmaps: The Future Brought To You By NASA”, Referenced 

12/4/2012, from http://www.nasa.gov/offices/oct/home/roadmaps/index.html. 

5. “Space Technology Roadmaps: Technology Area Breakdown Structure”, NASA, referenced 

December 11, 2012, http://www.nasa.gov/pdf/501627main_STR-Int-Foldout_rev11-

NRCupdated.pdf

6. U.S. Congress, Office of Technology Assessment, The National Space Transportation Policy: 

Issues for Congress, OTA-ISS-620 , Washington, DC, U.S. Government Printing Office, May 

1995 

7. “ NASA Software Engineering Requirements”, NASA Procedural Requirements NPR 7150.2A, 

Office of the Chief Engineer, Effective November 19, 2009, 

http://nodis3.gsfc.nasa.gov/displayDir.cfm?Internal_ID=N_PR_7150_002A_&page_name=Appe

ndixA&search_term=software

8. Mankins, J., "Research and Development Degree of Difficulty: A White Paper," March 1998, 

NASA Office of Space Access and Technology, Advanced Concepts Office. 

9. Bilbro, J., "Using the Advancement Degree of Difficulty (AD2

) as an input to Risk Management, 

Technology Maturity Conference, Virginia Beach, VA, Sept 8-12 2008. URL For TRAT 

including TRL and AD2

 found at https://trl.msfc.nasa.gov/. 

10. Office of the Chief Technologist (OCT) Definitions, NASA, January 2011 

  
Page 54 of 88 

APPENDIX B 

Survey Form 
Page 55 of 88 

Data Source Name

Respondent Data Source POC

Name

Organization

Email Address

Work Phone

Description

Tech Area(s)

Type of Data (Respond as applicable):

Historical Actual Historical Estimates Number of Projects Ea. Active Projects

Technology Project Size, Each (Mark all that apply)

Less than $1M $1M to $250M Greater than $250M

Tracked Technology Parameters (Mark all that apply):

Cost TRL Technology Area Facilities & Infrastructure Reqr'mnts Complexity FTEs

Schedule R&D 3 Tech. Proj. Tier /Scope Funding Source Push vs. Pull Unplanned Events

Programmatic Risk A&D 2 ∆ or TRL Progression Agencies/ORG/Companies Involved Funding Source Actual vs. Estimates

Perfomance Characteristics / KPPs, status and goal, per TA                    Evolutionary Vs. Revolutionary

  Capture progression/Change of Parameters over time               Budget Constraints & Disruptions

Projected Metric & Theoretical Limit SOP & SOA at start of Tech. Development

KPPs (list or describe)

Other (describe)

Platform (e.g. MS Excel)

Date Created

Update Frequency

Last Update

Data Sensitivity (Mark all that apply)

SBU Classified or Restricted Contractor Proprietary

Other (describe)

NOTES:

1. You may print this sheet, hand write the answers, and then electronically scan it for email.

2. Please complete form and email to Stuart.Cole@nasa.gov. Call at 757‐951‐7403 if there are any questions.

3. Multiple sheets are acceptable.

4. Complete the information to the best of your knowledge, cite sources for future reference.

5. The purpose of the "Survey"  is to obtain the best characterization of the data that is availiable.

Technology Estimating  ‐ Data Source Survey

NASA HQ Office of Evaluation, Cost Analysis Division, Technology Cost and Schedule Estimating Research Project

(Version: Rev A, Issue Date 11‐15‐2012)
Page 56 of 88 

APPENDIX C 

List of Data Sources 
Page 57 of 88 

Tier Data Source Name Full Name

Primary Sources

1 I NTID NASA Tech Inventory Database

2 I Cx ETDP Portfolio    

3 I ARC/Mathias Lunar Surface Data    

4 I External Gov Tech Maturation Data Tauri Group research into External Gov Technology Tech

Maturation data

5 I Sefcik Analysis GRC Analysis of NTID

6 I ESMD R&T Tech Data Sheets (2006)    

7 I RLV Tech DB (1993) NASA RLV Technology Database 1993

8 I HEOMD TechDev Team     

9 I ATLAS/TITAN Advanced Technology Lifecycle Analysis System  

10 I High Speed Research Program (80s‐90s) Wilhite led High Speed Research Program

11    National Aerospace Plane    

12    NASA SBIR/STTR    

13 I HEOMD NSPIRES     

14 I SMD NSPIRES    

15 I TBCC Tech Assessment (2010) Turbine Based Combined Cycle Tech Assessment Technology

Data Sheets 2010

16 I HLS ACENT DARPA/NASA Horizontal Launch Study ACENT Follow‐On

17 I Gryphon Phase 2A Technologies Gryphon Study Phase IIA High Speed Flight Experiments Concept

Study Technologies

18 I PIT Program Integration Tool

19 I GCD Project Snapshot    

20 I TechPort    

21 I START Strategic Assessment of Risk and Technology

22    DLR Data set    

23    IDA Dataset    

24    DoD Technology Mall    

Tools and Templates

25 I TCE Technology Cost Estimator

26 I AFRL Technology Calculator    

27 I Bilbro Technology Calculator Technology Assessment Calculator

29    Bilbro AD2 Calculator    

30 I DHS SS&T Readiness Level Calculator

31    NAVAIR spreadsheet    

32    NICM Template    

33    DARPA Business Acumen Assessment

34    RI3 Guidebook from Airforce    

35 II AFRL Projects    

36 II DARPA Projects    

37 II DOD/Army SBIR/STTR    
Page 58 of 88 

38 II DOD/MDA SBIR/STTR    

39 II DOD/Navy SBIR/STTR    

40 II DOD/USAF SBIR/STTR    

41 II DOE Projects    

42 II NASA NIAC I and II    

43 II NASA OCT (Early Stage, Game Changing, Cross Cutting)

44 II NASA SBIR/STTR    

45 II NRL Projects    

46 II NSF Grants    

47 II NSF SBIR/STTR    

48 II SBIR "Enhancement Awards"    

49 I CADRe Cost Analysis Data Requirement

50 III Advanced Product Design Team (Team X) Cost

Model

Advanced Product Design Team (Team X) Cost Model

51 III ALS/ADP Cost Model   ALS/ADP Cost Model

52 III A‐PICOMO Aerospace Picosatellites Cost Model

53 III BOE Tool Basis of Estimate Tool

54 III CEV TRL Summary/Details    

55 III ChiCoMo Chicago Cost Model

56 III ICSAT Integrated Cost and Schedule Analysis Tool

57 III MATCH Database Mapping Applicable Technology To Exploration Challenges

58 III MICM Multivariable Instrument Cost. Model

59 III NAFCOM 2011 NASA / Air Force Cost Model

60 III NAFCOM/REDSTAR    

61 III NICM NASA Instrument Cost Model

62 III Parametric Cost Modeling for Space Telescopes Parametric Cost Modeling for Space Telescopes

63 III PMCM Planetary Mission Cost Model

64 III PRICE TruePlanning PRICE TruePlanning

65 III PRICE‐H PRICE for Hardware

66 III PSCM Passive Sensor Cost Model  

67 III QuickCost 5.0 QuickCost 5.0

68 III SEER‐EOS (formerly SEER‐Spyglass) SEER for Electro Optical Sensors

69 III SEER‐H SEER for Hardware, Electronics, & Systems

70 III SEER‐IC SEER for Integrated Circuits

71 III SEER‐MFG SEER for Manufacturing

72 III SEER‐SEM SEER for Software

73 III Sensat   Sensor Satellite Expert System

74 III SICM Scientific Instrument Cost Model

75 III SSCM10 Small Satellite Cost Model 2010

76 III VIEW VIEW

APPENDIX D 
Page 59 of 88 

Data Collection Sheets 

Data Collection Sheets are available from the Cost Analysis Division.
Page 60 of 88 

APPENDIX E 

Technology Estimating Data Quality and Screening Methodology 
Page 61 of 88 

To: Doug Comstock, CAD 

From: Research Team 

Re: Technology Estimating Data Quality and Screening Methodology 

Date: January 30, 2013; Revised June 10, 2013 

The purpose of this paper is to help elicit discussion leading to formulation of a method for screening data 

for this research. The purpose for screening data is to obtain quality Technology data in the Technology 

Estimating research project. The screening methodology is also intended to provide guidance for selection 

of data to be used in the effort, and which data should be applied, or discarded. 

Currently, based on the Phase I meeting, technology project data is to be obtained for low TRL (2 to 6) 

technology projects and for NASA Technology Roadmap areas defined by TA 03, 04, 08, and 12. There 

are 20 parameters to be investigated for each of the technologies and to help determine Cost Estimating 

Relationships. The effort to determine CERs from these 20 parameters generally requires generation of 

the information from persons intimately knowledgeable of the project. 

Generally, the research effort is to help define which of the 20 parameters are beneficial to use for 

development of CERs. In addition, development of a model requires use of data that has been carefully 

reviewed by the Research Team for any inconsistencies or are poorly developed which might lead to 

misleading conclusions as to which of the parameters are of benefit towards the preparation of a point 

cost and schedule estimate with a reported accuracy range. 

Initially, for analysis of the 20 parameters, it is perceived that the data used in the model will help lead 

development and reporting of the cost and schedule accuracy range based on the quality data used. It is 

proposed that this accuracy range should perceptively change with the quality of data used. 

 Upon completion of the data collection process with the Technology Managers expected in the 1st

Quarter 2013, it should become apparent which of the 20 parameters will be eliminated. The elimination 

of any parameter for use in the model will need to be documented. The determination should include a 

narrative for why the parameter was eliminated, discuss any misconceptions, the “feel” as it would apply, 

or list other shortcomings/notes/comments in definition or interpretation by any of the Technology 

Research Team or research participants. 
Page 62 of 88 

SCREENING CRITERIA 

1. Technology Projects would be categorized into high, medium, low, or no/questionable value. 

2. High value: Technology projects could be identified as high value would be completed work 

showing TRL progression from TRL 2 to TRL 6 and/or beyond. These projects most likely would 

be represented by: 

 SBIR/STTR projects Phase III 

 ESTO, 

 EDTP, 

 Game Changing Development Program Office projects, 

Data would be used to satisfy or resolve of the 20 parameters would be generated by Technology 

Managers and/or those persons with direct project knowledge. 

3. Medium value: These Technology Project data includes some or satisfy part of the 20 parameters 

without confirmation by a Technology Manager or project knowledgeable person. These data 

would be obtained from currently published or documented resources and include description of 

the technology, TA identification, beginning and end costs, dates, and TRL. The medium value 

data would include a citation for the source from where it was derived, i.e. traceable. 

4. Low value: The Low value Technology Projects data include any one of the following: 

 Un-traceable with no citation for source of data, 

 FY dollars are not identified, 

 Project data does not indicate the total project time, 

 Project is active and a Technology Manager can or cannot be found to verify the data, 

 Technology Manager can verify project future end point costs, TRL progression, 

 Only one cost is provided. 

5. No/Questionable value: These criteria include technology data where anyone of the following 

applies: 

 Traceability does not exist, 

 The project is active, and no Technology Manager cannot be found to verify the data, 

 The origin of the data is in question, 

 Beginning or end cost data has not been provided, 

 No TRL data exists, 

 Project descriptions are not well defined, 

 No TA data, or 

 Basic technology project data is in question. 

The No/Questionable project data should not be used in the research. 
Page 63 of 88 

APPENDIX F 

Parameter Evaluation Narrative 
Page 64 of 88 
Page 65 of 88 
Page 66 of 88 
Page 67 of 88 
Page 68 of 88 
Page 69 of 88 
Page 70 of 88 
Page 71 of 88 
Page 72 of 88 
Page 73 of 88 
Page 74 of 88 

APPENDIX G 

Technology Cost and Schedule Estimating (TCASE) Tool 

The tool is available from the Cost Analysis Division. 
Page 75 of 88 

APPENDIX H 

User Manual 

Technology Cost and Schedule Estimating (TCASE) Tool 
User Manual 

TCASE 

Technology Cost and Schedule Estimating 

Beta Version

Document Revision 0.3 

30 August 2013 

Developed by:

SpaceWorks Software 

A Division of SpaceWorks Enterprises, Inc. (SEI) 

Funded by:

NASA Cost Analysis Division (CAD) and Game 

Changing Development (GCD) Program Office 
Page 2 of 13

Technology Cost and Schedule Estimation (TCASE) Tool

User Manual | Rev0.3 (2013-08-30)

User Manual

Contents

Introduction............................................................................................................................................................................................................... 3

List of TCASE Worksheets ................................................................................................................................................................................... 3

Front End Worksheet............................................................................................................................................................................................. 4

Search Criteria ..................................................................................................................................................................................................... 4

Statistical Results................................................................................................................................................................................................ 4

Top Analogies....................................................................................................................................................................................................... 5

Analogy Manager Worksheet ............................................................................................................................................................................. 6

Data Viewer Worksheet........................................................................................................................................................................................ 6

Data Importer Worksheet.................................................................................................................................................................................... 7

Data Exporter Worksheet .................................................................................................................................................................................... 8

Database Worksheet.............................................................................................................................................................................................. 9

Similarity Matrices Worksheet.........................................................................................................................................................................10

Frequently Asked Questions.............................................................................................................................................................................10

Acknowledgements ..............................................................................................................................................................................................11

Appendix A - Running a Simple Search........................................................................................................................................................12

Appendix B – Sources of Data Used in TCASE ...........................................................................................................................................13
Page 3 of 13

Technology Cost and Schedule Estimation (TCASE) Tool

User Manual | Rev0.3 (2013-08-30)

TCASE: Technology Cost and Schedule Estimation

Introduction

NASA and SpaceWorks have identified a need within the cost and schedule estimating community to rapidly assess the 

cost and schedule associated with maturing technologies along the Technology Readiness Level (TRL) scale. In response 

to this need, SpaceWorks has developed a Microsoft Excel® based software application called the Technology Cost and 

Schedule Estimation (TCASE) tool.

TCASE generates anticipated ranges of cost and schedule duration for a technology development project by drawing 

analogies to historical and current project. Data for historical and current projects is stored in an accompanying 

database. The tool is specifically designed to examine technologies in the range of TRL 2 through TRL 6.

At the end of this first year of research and development (September 2013), a prototype version of TCASE will be 

delivered to our research sponsors for testing and evaluation. The long-term goal of this research project is position

TCASE as an everyday tool used by the NASA cost estimating and technology assessment communities.

List of TCASE Worksheets

The TCASE Excel® workbook contains several worksheets that the user should become familiar with. The content and 

function of each is explained here.

Worksheet Name Description

Front End The primary point of interaction with the TCASE tool. Enter search criteria, perform 

project analogy searches, view statistical results, and examine the top analogies.

Analogy Manager View all analogy results returned by a search. Turn individual analogies ON or OFF as 

desired. View additional detailed information about each analogy result.

Data Viewer

View detailed information for a single analogy result / technology project record. 

Display is optimized for human viewing, as opposed to Database sheet (see below) 

which is optimized for software performance.

Data Importer Import Technology Project Data Collection Sheets into the TCASE database.

Data Exporter Export data records from the TCASE database to the Technology Project Data Collection 

Sheet format.

Database Collection of over 1,700 historical technology development project data records. 

Analogous projects are drawn from this database when user executes a search.

Similarity Matrices

User-defined matrices define relative similarity between different TRLs and between 

different System Hierarchy levels. Required for proper implementation of analogy 

matching algorithm.
Page 4 of 13

Technology Cost and Schedule Estimation (TCASE) Tool

User Manual | Rev0.3 (2013-08-30)

Front End Worksheet

The Front End worksheet serves as the basic interface with the TCASE tool. The sheet is neatly divided into 3 

sections: Search Criteria, Statistical Results, and Top Analogies. A screenshot can be found in Figure 2 on the 

following page.

SEARCH CRITERIA

TCASE offers tremendous flexibility in the use of exact match filters and/or weighted search terms to enable you to 

obtain the most desirable set of analogy results. There are a few basic guidelines to keep in mind when inputting 

search criteria:

 It is not necessary to fill out all of the Search Criteria search terms – you may use as many or as few as you 

like. As an extreme example, executing a search with no search terms will simply return the entire analogy 

database!

 To enter multiple search terms in a single field, simply separate each with a space. Multiple terms will be 

processed as a Boolean OR search (e.g. a Description Keywords search for HYDROGEN ROCKET will identify 

projects that contain HYDROGEN and/or ROCKET in the description field).

 The Exact Match parameter can be set to either yes or no. If set to yes, then that particular search term will 

be treated as a filter, meaning that any project in the database that does not contain that search term in the 

specified field will not be returned as a viable result.

 The Weighting parameter can be used to establish the relative importance of each search term (e.g. 

assigning a weighting = 1.0 for Title Keywords and a weighting of 2.0 for Primary TA would indicate that 

matching the Primary TA search term is twice as important as matching the Title Keywords).

 The 2 parameters listed under Search Settings (Minimum Score for Analogy Matching and Maximum 

Number of Analogies in Results) are optional parameters that can be used to limit the number of results 

returned by a particular search. Lowering the minimum score or raising the maximum number of analogies 

will increase the number of results returned, and vice versa.

STATISTICAL RESULTS

The Statistical Results section will display summary statistics and graphical outputs based on the data returned by 

your most recent search. These results constitute some of the most significant outputs of the TCASE tool, so we’ll 

take a moment to review them in detail here:

 Summary statistics can be found at the top of the Statistical Results section. These statistics are presented 

in 2 columns: 1 for cost and 1 for schedule. The count statistic indicates the number of analogies returned 

by your search that contain data for that metric (data for cost and/or data for schedule duration). Some 

records in the database may include cost data but not schedule data, or vice versa – thus, the 2 counts may 

be different numbers! Below the count statistic are some familiar measures of center and spread, including 

25th/75th percentile values, median value, mean value, maximum, and minimum. Finally, a box plot (aka 

“box-and-whiskers”) is provided for each column. If you are unfamiliar with the box plot format, please 

refer to Figure 1 on the following page.
Page 5 of 13

Technology Cost and Schedule Estimation (TCASE) Tool

User Manual | Rev0.3 (2013-08-30)

Figure 1. Definition of box plot as used in the TCASE tool.

 Below the summary statistics, you will find several plots in the Analogy Breakdown section. These plots are 

intended to provide some contextual information about the analogy results returned by your search. At a 

glance you can see the prevalence of various Lead NASA Centers, System Hierarchy Levels, and Project Start 

Years within your results set.

TOP ANALOGIES

After a database search has been executed, the resulting analogous project results will be displayed in 2 locations 

in TCASE: (1) the Top Analogies section of the Front End, and (2) the Analogy Manager worksheet. The purpose of 

the Top Analogies sections is simply to give the user an overview, in list format, of the highest ranking analogous 

projects from the most recent search. Up to 50 of the highest ranking results can be displayed here. A 

comprehensive and more detailed list of the analogy results can be found on the Analogy Manager worksheet.

Figure 2. TCASE Front End Worksheet.

Match search term 

exactly [optional]

Apply a weighting factor 

to each search term

Number of records 

returned with cost data

Number of records returned 

with schedule data

Box plots indicate center 

and spread of results data

View all analogy results in 

the Analogy Manager

General breakdown of 

analogy characteristics

Maximum 

Value

75th

Percentile

Mean 

Value

Median 

Value

25th

Percentile

Minimum 

Value
Page 6 of 13

Technology Cost and Schedule Estimation (TCASE) Tool

User Manual | Rev0.3 (2013-08-30)

Analogy Manager Worksheet

The Analogy Manager worksheet is a dynamic sheet containing a list of all analogous projects returned by the most 

recent user search. In some ways, the Analogy Manager is similar in function to a results webpage on Google or 

Bing. Whereas the Front End worksheet displays summary information such as statistics and up to 50 of the top 

analogy results, the Analogy Manager displays a simple list of all technology project analogies that met the user’s 

search criteria. A screenshot of this worksheet is shown below in Figure 3.

Figure 3. TCASE Analogy Manger Worksheet.

In addition to containing a comprehensive list of results, the Analogy Manager worksheet includes a feature to 

include or exclude individual analogy results from the statistical calculations on the Front End sheet. In other 

words, if the user discovers one or more analogy results that he or she finds undesirable or inappropriate, these 

offending results may be omitted by selecting “N” (i.e. “No”) in the column labeled “Include”. The Summary 

Statistics section will instantly and automatically reflect changes due to any such data exclusions.

Data Viewer Worksheet

The Data Viewer worksheet offers a user friendly way to view the contents of a particular technology project data 

record from the database. The sheet contains a single input cell labeled “Enter Database ID”. Simply enter the 

Database ID corresponding to the project record in question, and all data from that record will be displayed in a 

clearly labeled and organized table. A screenshot of the Data Viewer is provided in Figure 4 on the following page.

Note: The Data Viewer is a read-only utility and is not the appropriate place to attempt to edit a database record. 

The Database worksheet is the master location for all project data records.
Page 7 of 13

Technology Cost and Schedule Estimation (TCASE) Tool

User Manual | Rev0.3 (2013-08-30)

Figure 4. TCASE Data Viewer Worksheet.

Data Importer Worksheet

From time to time it will be necessary to add new technology project data to the TCASE database. The Data 

Importer worksheet is designed to make this task much more efficient than manual data entry. The current version 

of TCASE is only capable of automatically importing data from specially formatted Technology Project Data 

Collection Sheets. Future versions of this application may be capable of interfacing with other input file formats. A 

screenshot of the Data Importer is provided below in Figure 5.

Figure 5. TCASE Data Importer Worksheet.
Page 8 of 13

Technology Cost and Schedule Estimation (TCASE) Tool

User Manual | Rev0.3 (2013-08-30)

To use the Data Importer, start by clicking the button. You should see a standard file 

browser interface. Navigate to the location of the Technology Project Data Collection Sheet or Sheets that you wish 

to import (you may select multiple files in the browser by holding the CTRL key while clicking) and then click 

Open. You will now see one or more rows of data appear in the Data Importer – 1 row corresponding with each 

imported project.

TCASE will display a warning message if the name of any imported project appears to match the name of a project 

already in the database. Examine the column titled “Existing Entry in Database” to identify which project(s) may 

already exist. For each affected project, you may choose to either overwrite the existing data record in the database 

or create a new entry.

Once you are satisfied with the project data to be imported, click the button to make the specified 

changes and/or additions to the TCASE database.

Data Exporter Worksheet

The Data Exporter worksheet (see Figure 6) is designed to export one or more project data records from the 

TCASE database to a new Technology Project Data Collection Sheet. To begin using the Data Exporter, first open the 

Database worksheet in TCASE. To the left of each record in the database you will see a column entitled “Export 

Y/N”. Simply identify the project record(s) you wish to export and change the value in the “Export” column to Y for 

each.

Figure 6. TCASE Data Exporter Worksheet.

Once all desired projects have been identified and marked with a Y, open the Data Exporter worksheet and click 

the button. Doing so will copy each marked database record into the Data 

Exporter as a new row in the list of projects. Take a moment to review the “Export File Name” for each project. 
Page 9 of 13

Technology Cost and Schedule Estimation (TCASE) Tool

User Manual | Rev0.3 (2013-08-30)

These have been created automatically, but you are free to change the file name at your discretion. Once you are 

satisfied with the list of projects to export and the export file names, simply click the button and 

a set of new Technology Project Data Collection Sheets will be created. Newly created Technology Project Data 

Collection Sheets will be saved in a subfolder within the directory where the TCASE Excel workbook resides. One 

sheet will be created for each technology project that you have exported.

Database Worksheet

As the name suggests, the Database worksheet is the core database that supports all other TCASE functionality. Out 

of the box, so to speak, the TCASE database contains more than 1,700 rows of data (aka “records” or “entries”) each 

of which includes up to 184 different data fields. The number of data records can be expanded – TCASE includes a 

Data Importer utility that allows a user to add new data records from Technology Project Data Collection Sheets. If 

you take a moment to browse the Database worksheet, you will notice that many data fields contain no data –

perhaps half of all data fields in the database are empty. In some cases the empty cells indicate that that data field 

was not required for that particular project, while in other cases it represents incomplete information available 

about the project. As the TCASE research project continues, one objective is to improve the overall data quality in 

the database. A screenshot of a small portion of the TCASE database is shown below in Figure 7.

Figure 7. TCASE Database Worksheet.

Practically speaking, a typical user will not have much need to access the Database worksheet directly. All analysis 

functions take place on the Front End and Analogy Manager worksheets. A more advanced user may wish to export 

data from the TCASE database using the Data Exporter described in the previous section. However, as we have 

seen even the Data Exporter requires only minimal user interaction with the database.

It is possible to edit the TCASE database directly on the Database worksheet, although such edits must be 

conducted with great care and should only be attempted by an experienced user with a backup file. For 

example, it is possible to delete an unwanted row from the TCASE database by clicking on the row number and 

using Excel’s “Delete” function. It is also possible, although somewhat riskier, to insert a new row and manually fill 

it with valid data for a new project record.
Page 10 of 13

Technology Cost and Schedule Estimation (TCASE) Tool

User Manual | Rev0.3 (2013-08-30)

Similarity Matrices Worksheet

The Similarity Matrices worksheet contains 2 matrices used during the TCASE analogy matching process. These 

matrices are used to determine a degree of similarity between levels in the Technology Readiness Level (TRL) and 

System Hierarchy parameters.

The default similarity matrix for TRL is show in Figure 8 below. Interpreting the matrix is fairly straightforward. If 

for example we were interested in determining the similarity score between a TRL of 4 and a TRL of 7, we would 

look for the intersection of row 4 and column 7 and find a value of 0.2. Values along the diagonal of the matrix are 

equal to 1.0 since they occur at the intersection of identical values. The similarity matrices are user-editable, if 

desired.

Figure 8. TCASE Similarity Matrix for TRL.

Frequently Asked Questions

 What is the origin of the technology development cost and schedule data in the TCASE database?

The TCASE Database includes over 1,700 records from more than 60 sources. These sources include historical 

databases such as NTIS and MATCH, project records such as GCD and ETDP, SBIR records, and data solicited from 

technology managers during this current research period (2012-2013). More information can be found in 

Appendix B of this manual.

 Has the cost data in the TCASE database been normalized?

Yes. All costs in the TCASE database have been normalized to FY2013 dollars.

 Does the TCASE tool utilize cost estimating relationships (CERs)?

No, not at this time. TCASE uses an analogy-based approach to provide a likely range of cost and schedule for a 

particular type of technology development project. Future work plans call for a transition to CER-based 

parametric estimation methods if and when sufficient data is available to support that approach.
Page 11 of 13

Technology Cost and Schedule Estimation (TCASE) Tool

User Manual | Rev0.3 (2013-08-30)

 My search returned more than 50 analogous projects, but only the top 50 are shown in the Top Analogies 

section. Where can I view the full results?

Select the Analogy Manager worksheet to view the full list of analogous projects.

 My search returned several projects that I believe are poor analogies for my estimating scenario. How can I 

improve the results?

You have several options to refine your search. First, you might revise your search criteria, setting one or more 

of your search terms to “Exact Match” in order to impose a hard filter. Alternatively, you may view the Analogy 

Manager worksheet where you have the option to manually include or exclude each analogous project returned 

by TCASE. With a few clicks you can toss out one or more analogous projects that you feel are improperly

skewing your results. The effects of any changes made on the Analogy Manager tab will be reflected in the 

results on the Front End as soon as you return to that worksheet. 

Acknowledgements

This research and development was sponsored by NASA’s Cost Analysis Division (CAD) and Game Changing Development 

(GCD) program office. The research team is grateful for feedback and guidance offered by the estimating community at 

2 workshops held in the fall of 2012 and summer of 2013.
Page 12 of 13

Technology Cost and Schedule Estimation (TCASE) Tool

User Manual | Rev0.3 (2013-08-30)

Appendix A - Running a Simple Search

1. Open the MS Excel® workbook entitled TCASE Analogy Tool Beta v0.50.xlsm and access the 

Front End worksheet.

2. Adjust parameters in the Search Criteria section as desired. Some parameters accept text 

inputs, others accept numerical inputs, and some are set via a pull-down menu. Relative 

weightings may be applied to your parameters to reflect their importance in your search.

3. Click the button to run the TCASE search algorithm and generate analogous 

project results. *

4. Review the outcome of your search in the Statistical Outputs section and the Top 

Analogies section of the Front End worksheet.

5.

To refine your search, simply edit the parameters in the Search Criteria section and click 

the button again. If you would like to start a completely new search, click the 

button and return to step 2 above.

* Important Notes: The TCASE tool utilizes VBA macros during normal operation. Macros must be enabled in the 

user’s local copy of Excel. Excel’s Automatic Calculation setting must also be turned on.
Page 13 of 13

Technology Cost and Schedule Estimation (TCASE) Tool

User Manual | Rev0.3 (2013-08-30)

Appendix B – Sources of Data Used in TCASE

Early in the TCASE research project, the team identified more than 70 sources of historical technology project data. 

These sources included broad databases such as NTIS and MATCH, and also numerous individual project records 

from programs such as Game Changing Development (GCD) and the Exploration Technology Development 

Program (ETDP), to name a few. The table below lists some of the largest sources of technology project data.

Short Name Description Year Number 

of Entries

ETDP NASA Exploration Technology Development Program 2012 42

ESTO NASA Earth Science Technology Office 2012 138

SBIR Current NASA SBIR Technologies 2012 51

GCD Game Changing Development Technology Projects 2012 35

ESMD NASA Exploration Systems Mission Directorate ESAS 

Technologies 2005 304

Tech Tool Box ATLAS (Advanced Technology Lifecycle Analysis System) 2002 6

Ext Tech Data Tauri Group research into External Gov Technology Tech 

Maturation data 2012 28

Cx ETDP Constellation Program Technology Portfolio 2007 19

NTID NASA Tech Inventory Database 2004 991

Hist SBIR STTR Historical SBIR and STTR data 2012 1191

RLV Tech DB NASA RLV Technology Database 1993 64

HEOMD TechDev 

Team Estimates of HEOMD needed technology developments 2012 78

ETDP from MATCH Mapping Applicable Technology To Exploration Challenges 2007 21

External from MATCH Mapping Applicable Technology To Exploration Challenges 2007 192

JSS TBCC NASA/Air Force Joint Systems Study TBCC Technologies 2010 18

Not every technology project data record contained the critical cost and schedule duration values needed for use in 

TCASE. As indicated in the figure below, about half of the total records collected were suitable for use in TCASE (1,723 

out of 3,178). Thus, the database embedded in TCASE contains records for 1,723 historical and current technology 

development projects.
REPORT DOCUMENTATION PAGE Form Approved

OMB No. 0704-0188

2. REPORT TYPE 

Technical Publication

 4. TITLE AND SUBTITLE

Technology Estimating: A Process to Determine the Cost and Schedule of 

Space Technology Research and Development 

5a. CONTRACT NUMBER

NNL11AA05B

 6. AUTHOR(S)

Cole, Stuart K.; Reeves, John D.; Williams-Byrd, Julie A.; Greenberg, 

Marc W.; Comstock, Douglas A.; Olds, John R.; Wallace, Jon; 

DePasquale, Dominic; Shaffer, Mark

 7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES)

NASA Langley Research Center

Hampton, VA 23681-2199

 9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES)

National Aeronautics and Space Administration

Washington, DC 20546-0001

 8. PERFORMING ORGANIZATION

 REPORT NUMBER

L-20337

10. SPONSOR/MONITOR'S ACRONYM(S)

NASA

13. SUPPLEMENTARY NOTES

Questions concerning the content of this paper should be directed to NASA Headquarters, Office of Evaluation, Cost Analysis Division.

12. DISTRIBUTION/AVAILABILITY STATEMENT

Unclassified - Unlimited

Subject Category 83

Availability: NASA CASI (443) 757-5802

19a. NAME OF RESPONSIBLE PERSON

STI Help Desk (email: help@sti.nasa.gov)

14. ABSTRACT

NASA is investing in new technologies that include 14 primary technology roadmap areas, and aeronautics. Understanding the cost for research and 

development of these technologies and the time it takes to increase the maturity of the technology is important to the support of the ongoing and future NASA 

missions. Overall, technology estimating may help provide guidance to technology investment strategies to help improve evaluation of technology 

affordability, and aid in decision support. The research provides a summary of the framework development of a Technology Estimating process where four 

technology roadmap areas were selected to be studied. The framework includes definition of terms, discussion for narrowing the focus from 14 NASA 

Technology Roadmap areas to four, and further refinement to include technologies, TRL range of 2 to 6. Included in this paper is a discussion to address the 

evaluation of 20 unique technology parameters that were initially identified, evaluated and then subsequently reduced for use in characterizing these 

technologies. A discussion of data acquisition effort and criteria established for data quality are provided. The findings obtained during the research included 

gaps identified, and a description of a spreadsheet-based estimating tool initiated as a part of the Technology Estimating process. 

15. SUBJECT TERMS

Affordability; Analysis; Cost; Decision support; Estimating; Schedules; TCASE; Technologies; Technology readiness level

18. NUMBER

 OF 

 PAGES

93

19b. TELEPHONE NUMBER (Include area code)

(443) 757-5802

a. REPORT

U

c. THIS PAGE

U

b. ABSTRACT

U

17. LIMITATION OF 

 ABSTRACT

UU

Prescribed by ANSI Std. Z39.18

Standard Form 298 (Rev. 8-98)

3. DATES COVERED (From - To)

August 2012 - August 2013

5b. GRANT NUMBER

5c. PROGRAM ELEMENT NUMBER

5d. PROJECT NUMBER

5e. TASK NUMBER

NNL12AC80T. Req# 4200449610

5f. WORK UNIT NUMBER

 122711.03.07.07.17.80

11. SPONSOR/MONITOR'S REPORT

 NUMBER(S)

NASA/TP-2013-218145

16. SECURITY CLASSIFICATION OF:

The public reporting burden for this collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, 

gathering and maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this 

collection of information, including suggestions for reducing this burden, to Department of Defense, Washington Headquarters Services, Directorate for Information Operations and 

Reports (0704-0188), 1215 Jefferson Davis Highway, Suite 1204, Arlington, VA 22202-4302. Respondents should be aware that notwithstanding any other provision of law, no person 

shall be subject to any penalty for failing to comply with a collection of information if it does not display a currently valid OMB control number.

PLEASE DO NOT RETURN YOUR FORM TO THE ABOVE ADDRESS.

1. REPORT DATE (DD-MM-YYYY)

01- 12 - 2013