April 2014 

NASA/TM–2014-218251 

Technology Estimating 2: A Process to 

Determine the Cost and Schedule of Space 

Technology Research and Development 

Stuart K. Cole 

NASA Headquarters, Washington, D.C. 

 

Jon Wallace, and Mark Schaffer 

SpaceWorks Enterprises, Inc., Atlanta, Georgia 

M. Scott May and Marc Greenberg 

NASA Headquarters, Washington, D.C. 
NASA STI Program . . . in Profile

Since its founding, NASA has been dedicated to the 

advancement of aeronautics and space science. The 

NASA scientific and technical information (STI) 

program plays a key part in helping NASA maintain 

this important role. 

The NASA STI program operates under the 

auspices of the Agency Chief Information Officer. 

It collects, organizes, provides for archiving, and 

disseminates NASA’s STI. The NASA STI 

program provides access to the NASA Aeronautics 

and Space Database and its public interface, the 

NASA Technical Report Server, thus providing one 

of the largest collections of aeronautical and space 

science STI in the world. Results are published in 

both non-NASA channels and by NASA in the 

NASA STI Report Series, which includes the 

following report types: 

• TECHNICAL PUBLICATION. Reports of 

completed research or a major significant phase 

of research that present the results of NASA 

Programs and include extensive data or 

theoretical analysis. Includes compilations of 

significant scientific and technical data and 

information deemed to be of continuing 

reference value. NASA counterpart of peerreviewed formal professional papers, but 

having less stringent limitations on manuscript 

length and extent of graphic presentations. 

• TECHNICAL MEMORANDUM. Scientific 

and technical findings that are preliminary or of 

specialized interest, e.g., quick release reports, 

working papers, and bibliographies that contain 

minimal annotation. Does not contain extensive 

analysis. 

• CONTRACTOR REPORT. Scientific and 

technical findings by NASA-sponsored 

contractors and grantees. 

• CONFERENCE PUBLICATION. 

Collected papers from scientific and 

technical conferences, symposia, seminars, 

or other meetings sponsored or cosponsored by NASA. 

• SPECIAL PUBLICATION. Scientific, 

technical, or historical information from 

NASA programs, projects, and missions, 

often concerned with subjects having 

substantial public interest. 

• TECHNICAL TRANSLATION. 

English-language translations of foreign 

scientific and technical material pertinent to 

NASA’s mission. 

Specialized services also include organizing 

and publishing research results, distributing 

specialized research announcements and feeds, 

providing information desk and personal search 

support, and enabling data exchange services. 

For more information about the NASA STI 

program, see the following: 

• Access the NASA STI program home page 

at http://www.sti.nasa.gov

• E-mail your question to help@sti.nasa.gov

• Fax your question to the NASA STI 

Information Desk at 443-757-5803 

• Phone the NASA STI Information Desk at 

443-757-5802 

• Write to: 

 STI Information Desk 

 NASA Center for AeroSpace Information 

 7115 Standard Drive 

 Hanover, MD 21076-1320
National Aeronautics and 

Space Administration 

Langley Research Center 

Hampton, Virginia 23681-2199 

 

April 2014 

NASA/TM–2014-218251 

Technology Estimating 2: A Process to 

Determine the Cost and Schedule of Space 

Technology Research and Development 

Stuart K. Cole 

NASA Headquarters, Washington, D.C. 

 

Jon Wallace, and Mark Schaffer 

SpaceWorks Enterprises, Inc., Atlanta, Georgia 

M. Scott May and Marc Greenberg 

NASA Headquarters, Washington, D.C. 

.
Available from: 

NASA Center for AeroSpace Information 

7115 Standard Drive 

Hanover, MD 21076-1320 

443-757-5802 

Acknowledgments 

This work was performed in succession to a STI paper NASA/TP–2013-218145, and as 

collaboration between the NASA HQ Office of Evaluation, Cost Analysis Division, the 

Game Changing Development Program Office in the Space Technology Mission 

Directorate, LaRC Systems Analysis & Concepts Directorate, the LaRC Research 

Directorate providing one FTE, and MSFC Strategic Analysis & Communication 

providing one quarter FTE to fulfill the need for meeting NASA’s goals, furthering 

technological advances and research, and promoting opportunities for multiple centers 

and interdepartmental working relationships. The work was also performed in 

cooperation with the NASA HQ Office of the Chief Technologist and Center Chief 

Technologists who helped with support to data acquisition. The CAD, the Game 

Changing Development Program Office funded a portion of this research under LaRC 

contract number NNL11AA05B, Order # NNL12AC80T, Req. # 4200449610 to 

SpaceWorks Enterprises Inc. The LaRC Systems Analysis & Concepts Directorate is 

acknowledged for research contributions. The Old Dominion University, Norfolk, 

Virginia is acknowledged for contributions to this research. 

The use of trademarks or names of manufacturers in this report is for accurate reporting and does not constitute an 

official endorsement, either expressed or implied, of such products or manufacturers by the National Aeronautics 

and Space Administration.
Abstract 

As a leader in space technology research and development, NASA is continuing in the development 

of the Technology Estimating process, initiated in 2012, for estimating the cost and schedule of low 

maturity technology research and development, where the Technology Readiness Level is less than TRL 

6. NASA’s Technology Roadmap areas consist of 14 technology areas. The focus of this continuing 

Technology Estimating effort included four Technology Areas (TA): TA3 Space Power and Energy 

Storage, TA4 Robotics, TA8 Instruments, and TA12 Materials, to confine the research to the most 

abundant data pool. This research report continues the development of technology estimating efforts 

completed during 2013-2014, and addresses the refinement of parameters selected and recommended for 

use in the estimating process, where the parameters developed are applicable to Cost Estimating 

Relationships (CERs) used in the parametric cost estimating analysis. This research addresses the 

architecture for administration of the Technology Cost and Scheduling Estimating tool, the parameters 

suggested for computer software adjunct to any technology area, and the identification of gaps in the 

Technology Estimating process. 

 
2 

Introduction and Background 

This research covers the period from July 2013 to February 2014 and was a continuation of the 

work to implement a capability for providing a technology estimating process, initiated in 2012, and 

sponsored by the NASA Office of Evaluation, Cost Analysis Division (CAD) in collaboration with the 

Space Technology Mission Directorate, and Game Changing Development Program Office. 

The research goal was to improve NASA’s ability to estimate the cost of technology research and 

development for low maturity technologies. This technology estimating research evolved by selection of 

critical parameters that were expected to influence the cost of a technology research and development, a 

continuation of collecting actual data for completed technology efforts, and then conducting mathematical 

and statistical analyses to generate cost estimating relationships between the parameters and use of these 

data in an analysis to determine the cost and schedule of technology development for Technology 

Readiness Level less than 6.(1) 

Recognizing that NASA currently has 14 Technology Roadmap Areas, the effort for constructing 

a database of completed projects for the full 14 Technology Areas was beyond the scope of the research. 

Strategically, only a few select technologies were to be investigated and these included Technology Areas 

(TAs): Space Power (TA03) and Energy Storage, Robotics (TA04), Instruments (TA08), and Materials 

(TA12), as these provided the highest number of potential projects for use in evaluating the parameters 

considered, and were determined to be good candidates for fulfillment of data acquisition. 

A diagram, Figure 1, was prepared to map the continuation of the procedural process followed 

during the research project and includes a series of activities, parallel analysis, workshops, an interim 

review, and internal reviews within the NASA Cost Community to continue with the identification of 

issues and the development of the Technology Estimating Research project. This structure facilitated the 

necessary NASA Cost community input and acceptance of the process, and thereby allowing a high 

profile platform for the internal review. More specifically, the process was adapted to include dual 

processes in the Technology Cost and Schedule Estimating (TCASE) tool. 

Figure 1. Technology Estimating Process Diagram, 2013-2014 
3 

Data is critical for the Technology Estimating process. Obtaining this data in the form of cost 

estimating parameters for each of NASAs R&D projects was considered fundamental to the Technology 

Estimating process and continued development of the Technology Cost and Schedule Estimating 

(TCASE) tool. NASA’s TechPort was identified in this research as the most reliable resource and 

repository for this technology estimating data. TechPort is a portal developed by NASA, as a software 

system to capture, track, and manage NASA's portfolio of technology investments. 

Evaluation of Technology Estimating Parameters 

The Technology Estimating research conducted during 2012-2013 determined that the best 

method suited for use as an estimating method for estimating technology research and development 

(R&D) was the parametric analysis. The research then focused on defining those parameters, which best 

represent the drivers for cost and schedule in technology R&D and ultimately incorporating those 

parameters in the parametric analysis as Cost Estimating Relationships (CERs). 

Parametric estimating is a statistical relationship method developed between historical costs and 

program, physical, and performance characteristics. The parametric method relates cost to one or more 

technical, performance, cost, or program parameters, using a statistical relationship. The goal of 

parametric estimating is to create a statistically valid cost estimating relationship using historical data.(2)

The foundation on which parametric estimating is based is the use of CERs by which the analysis is 

performed. 

An analysis was conducted for approximately 600 project records in only four of the 14 

Technology Areas (i.e., TA03, TA04, TA08, and TA12) as a test bed, and to reduce the set of technology 

project data parameters to those key parameters determined to be most informative in terms of predicting 

target variables for both total project cost and total project duration. This reduction in the parameter set 

number makes the data collection more practical. In addition, the reduced number of parameters provides 

for increased efficiency and affords a practical step in the process for creating a predictive model for cost 

& duration. 

The analysis of the historical technology project data in the TCASE database was also performed 

to help determine which parameters were believed to be most useful for cost and duration prediction. As a 

predictor, the parameter must be a schedule and cost driver. As a driver, a parameter that tends to increase 

or decrease schedule/cost, or has been identified to be highly correlated with schedule/cost. To qualify as 

a cost driver, a parameter must be a significant predictor of schedule/cost at least in the statistical sense. 

The methodology used in the analysis included an examination of each technology project data 

parameters on a one-at-a-time basis. First, each parameter was taken and then the set of 600 plus data 

records were subdivided by all levels of that parameter. Next, a statistical test was performed to determine 

whether there was a significant difference, i.e. beyond simple randomness in the data, between the 

distributions of cost and/or duration for projects in these subdivided groups. The tendency of a parameter 

to subdivide the full dataset into groups with significantly different costs and/or durations was evaluated 

to indicate if a parameter was a good choice for use in predictive modeling. The statistical testing 

described here was just one of several factors that may ultimately determine which technology project 

data parameters were to remain in the final set. 

The statistical analysis used for the examination of the parameters was the Kruskal-Wallis oneway analysis of variance. This statistical method of analysis is a non-parametric test that is applied much 

like the Analysis of Variance (ANOVA) test.(3) Non-parametric means that this test does not assume the 

data to be tested conform to any particular distribution shape or characteristics, e.g. does not require input 

data to adhere to a normal distribution. The ANOVA test is best suited for test data that has been found to 
4 

be approximately normally distributed (although the test is robust to violations of this assumption as long 

as the sample size is large). The TCASE data tested was not always normally distributed. Kruskal-Wallis 

has no such assumption of normally distributed test data and is non-parametric. 

Where the Kruskal-Wallis test did not apply, the Two Sample t-Test was used for the parameter 

analysis. In cases where the parameter were dichotomous, e.g. Yes/No, Push/Pull, etc., the t-test was 

determined to be an appropriate method and will return identical or nearly identical results when 

compared with the more involved tests such as ANOVA. The analysis was performed to reveal those 

parameters when considered individually, partition the full sample set into groups with statistically 

significant distributions of project cost and/or duration. In statistical terms, the null hypothesis is a 

condition where the distributions are the same across all levels. The alternative hypothesis is that the 

distributions are not the same. 

In summary, the statistical test results presented in Table 1, although interesting, were not 

predisposed to fulfill the notion that a useful regression model could be identified or a parametric model 

could be created using these data. The analysis indicates that the parameters identified as “rejected null 

hypothesis” could potentially contribute to a future parametric model that is statistically valid and wellconceived. 

Table 1 

Parameter Data Analysis Results

Parameter 

Name 

Parameter 

Type 

Sample Size 

(cost; 

duration) 

Test Type 

(p-value = 

0.05) 

Test Outcome 

Target = 

COST 

Test Outcome 

Target = 

DURATION 

 

Reject Null 

Hypothesis 

Failed to 

Reject 

Reject Null 

Hypothesis 

Failed to 

Reject 

Technology Area Categorical (4 levels) n = 620; 583 Kruskal-Wallis  

Technology Readiness Level Categorical (27 levels) n = 541; 535 Kruskal-Wallis  

System Hierarchy Level Categorical (5 levels) n = 609; 573 Kruskal-Wallis  

Lead NASA Center Categorical (8 levels) n = 166; 139 Kruskal-Wallis  (a) 

RD3 Categorical (5 levels) n = 101; 99 Kruskal-Wallis  

Hardware vs. Software Categorical (5 levels) n = 55; 54 Kruskal-Wallis  

Team Experience Categorical (4 levels) n = 53; 52 Kruskal-Wallis  

AD2 Categorical (9 levels) n = 53; 52 Kruskal-Wallis  

Unplanned Events Dichotomous n = 44; 43 Two Sample t-Test  

Technology Push vs. Pull Dichotomous n = 63; 62 Two Sample t-Test  

Evolutionary vs. Revolutionary Dichotomous n = 52; 51 Two Sample t-Test (a) 

Facilities and Infrastructure Dichotomous n = 49; 49 N/A (could not test) 

Budget Constraints/Disruptions Dichotomous n = 46; 55 Two Sample t-Test (a) 

Note: N/A = Not applicable, (a) = Small relaxation in p-value (significance level) would have led to rejection of the null 

hypothesis. 

The results of the analysis conducted using the Kruskal-Wallis and t-Test contributed to the 

elimination or reconfiguration of the following parameters: push versus pull, evolutionary versus 

revolutionary; single project versus portfolio; budget constraints/disruptions, Advancement Degree of 

Difficulty (AD2

), and hardware versus software. The parameter for hardware was eliminated and software 

was expanded to include additional parameters based on information to be acquired by submission of 

Software Management Plans in accordance with NPR 7150.2A. 
5 

A qualitative analysis was performed to establish the preferred parameters to be used in 

Technology Estimating. The qualitative analysis is included in Appendix A. 

Based on the statistical analysis conducted, Table 1 and the qualitative analysis, Appendix A, the 

cost estimating parameters recommended for use in estimating the cost and schedule of technology R&D 

with low maturity, i.e. TRL less than 6, are included in Table 2. Each parameter was classified as a 

qualitative or quantitative data type. Seven of the 15 Technology Estimating parameters are included in 

TechPort. 

Table 2 

Technology Estimating Parameters

Category Parameter Data Type Included in TechPort 

 Qualitative Quantitative 

Project Characterization Technology Area  

 System Hierarchy 

Defining system characteristics  

 Computer Software 

 Project Type 

 Software Class* 

 Primary Language 

Number of Requirements 

 Software Size 

Projects Results Key Performance Parameter(s)  

Level of Effort 

Actual Cost  

Schedule (Start and End Date)  

Development Metrics TRL  

 RD3 

Project Execution Funding source(s)  

Performing Organizations  

Team Experience 

Facilities and Infrastructure 

Programmatic Factors Unplanned Event(s)  

Note: TRL = Technology Readiness Level; RD3 = Research and Development Degree of Difficulty; 

* = Class A, B, C, D, or E in accordance with NPR 7150.2A

Cost Estimating Relationships 

The TCASE cost and schedule tool is currently an analogy based cost and schedule estimating 

tool. Its future development includes the application development and use of parametric analysis once 

sufficient project data is available. The continued increase of data will allow the analysis to be conducted 

for subsequent establishment of Cost Estimating Relationships (CERs). Obtaining an adequate number of 

data in each of the Technology Areas and sub-TAs present has presented a challenge for derivation of 

CERs that span the four Technology Areas that were initially investigated, i.e. TA03 Robotics, TA04 

Power, TA08 Instrumentation, and TA12 Materials. It was determined that CER development for the 

balance of the remaining Technology Areas will be accomplished after cost and schedule data can be fully 

acquired from sources such as TechPort. 

The validity of a CER will usually be judged by its regression statistics, which measure the 

accuracy of the fit of the CER to the sample data points used in developing the CER. The most commonly 

used regression statistic is the coefficient of determination (R2

), sometimes described as “goodness-offit”. A CER that perfectly predicts each sample point in the database would have an R2

 of 1.0. For CERs, 
6 

an R2

 value of 0.9 or better is considered desirable, although in practice, CERs with R2 values of 0.8 or 

better may be accepted for use in a cost estimate.(4)

The format of the CERs to be developed for Technology Estimating should follow The Project 

Cost Estimating Capability (PCEC). In 2013, NASA began investigating the development of changes to 

the existing cost estimating data platforms. The Project Cost Estimating Capability (PCEC) was identified 

as a new framework that will replace NASA Air Force Cost Model (NAFCOM) as the standard NASA 

capability for estimating the cost of new spaceflight hardware systems during concept exploration and 

refinement. When the development is complete, the PCEC will be an ExcelTM based architecture that 

combines a user interface running Visual Basic Algorithm (VBA) with Work Breakdown Structure 

(WBS) and CER libraries. 

Technology Estimating Process 

The Technology estimating process includes the acquisition of data for use the Technology 

Estimating Cost and Schedule Estimating tool. Multiple sources of data have been investigated and the 

most robust source of data determined viable for low maturity technologies, i.e. TRL less than 6. 

TechPort is NASA’s premier information portal for research and development technology. It has 

been identified as a significant data source for Technology Estimating. Of the 15 technology estimating 

parameters identified for support of Technology Estimating (Table 3), seven parameters are already 

included in TechPort. Eight additional parameters have been identified that support the Technology 

Estimating process should be included in TechPort. To facilitate the addition of eight additional 

Technology Estimating parameters in future changes to TechPort, Table 3 indicates a suggested order 

sequence for their inclusion. A sequencing of these additional parameters into TechPort would also help 

to integrate changes systematically, adjust users to these new parameters methodically, and help reduce or 

minimize potential errors in their application. 

The priority for order of sequence was a ranking from “1” the highest priority to be implemented 

first, to “3” the lowest priority to be implemented last. The rank and priority was based on: 

• The metric associated with the parameter, 

• The parameters’ perceived ease of application as observed during the Technology Estimating 

research project, 

• Experience for the parameters’ use and application by Technology Managers as observed during 

the research, and 

• The parameters’ level of definition. 
7 

Table 3 

Technology Estimating Parameters to Be Added in TechPort

Category Parameter Data Type Priority and 

Sequence for 

Inclusion in TechPort 

 Qualitative Quantitative 

Projects Results Level of Effort  1 

Development Metrics RD3  1 

Project Execution Team Experience  1 

Project Characterization System Hierarchy  1 

Project Characterization Defining system characteristics   2 

Project Characterization Computer Software 

 Project Type  2 

 Software Class*  2 

 Primary Language  2 

 Number of Requirements  2 

 Software Size  2 

Project Execution Facilities and Infrastructure  3 

Programmatic Factors Unplanned Event(s)   3 

Note: TRL = Technology Readiness Level; RD3 = Research and Development Degree of Difficulty. * = Class A, B, C, D, or E in accordance 

with NPR 7150.2A

The TCASE tool should include projects from TechPort for use in Technology Estimating. Once 

cost and schedule data are obtained from TechPort, for all of the NASA Technology Roadmap areas 1 

through 14, and a level of confidence is obtained for importing data and Technology Estimating, then 

additional Technology Estimating parameters, as identified in Table 3 should be incorporated sequentially 

according to its indicated priority. 

Administration of Technology Estimating Process 

The process for technology estimating includes the use of data for completed research and 

development projects with low maturity. A chart, Figure 2, indicating proposed administration activities 

was prepared to reveal the path of project data from initiation of the project by the Technology Owner to 

the project documentation data sources such as TechPort, Cost Analysis Data Requirements (CADRe) by 

NPR 7120.5E, and Software Management Plans submitted in accordance with NPR 7150.2A. Other 

project data have been migrated to TCASE such as those data sources identified in Table 4. 
8 

Table 4 

Data Sources Used in this Research (5)

Short Name Description Year Entries 

ETDP NASA Exploration Technology Development Program 2012 42 

ESTO NASA Earth Science Technology Office 2012 138 

SBIR Current NASA SBIR Technologies 2012 51 

GCD Game Changing Development Program Office Technology Projects 2012 35 

ESMD NASA Exploration Systems Mission Directorate ESAS Technologies 2005 304 

Tech Tool Box ATLAS (Advanced Technology Lifecycle Analysis System) 2002 6 

Ext Tech Data Tauri Group research into External Government Technologies (EGT within 

the MATCH relational database) 2012 28 

Cx ETDP Constellation Exploration Technology Development Program 2007 19 

NTID NASA Tech Inventory Database 2004 991 

Hist SBIR STTR Historical SBIR and STTR data 2012 1191 

RLV Tech DB NASA RLV Technology Database 1993 64 

HEOMD TechDev 

Team Estimates of HEOMD needed technology developments 2012 78 

ETDP from 

MATCH Mapping Applicable Technology To Exploration Challenges 2007 21 

External from 

MATCH Mapping Applicable Technology To Exploration Challenges 2007 192 

JSS TBCC NASA/Air Force Joint Systems Study TBCC Technologies 2010 18 

Individual project data from the data sources are then migrated to One NASA Cost Engineering 

(ONCE) for use in the TCASE tool. The data for TCASE can be reviewed by an Administrator who will 

review data quality, assess parameters and perform any needed data scrubbing; tool updates, distribution 

and version control; and CER development. Any data that requires retirement due to age or applicability 

will be evaluated by the TCASE Administrator and notify the Technology Owner of any errors, omissions 

or issues as may be determined helpful. 
9 

Figure 2. Technology Estimating process 

Data Migration from TechPort to the TCASE Tool 

TechPort is NASA’s repository of information on each individual project within NASA R&D 

programs. The TechPort R&D projects are searched by several parameters including seven of those 

identified for Technology Estimating. Representative parameters included are the project name, start and 

end date, NASA technology roadmap area, and key performance parameter, etc. After the TechPort user 

completes a selection process, the multiple attributes for each of the selected projects are then 

downloaded to an EXCELTM spreadsheet. 

Currently, due to gaps observed in the project records for a project in TechPort, the completeness 

of Technology Estimating parameters may not always be fulfilled or included for the respective project 

data file and therefore a parameter field may be blank or omitted. A February 2014 survey of project 

records for “completed” projects indicated that a critical parameter such as “cost” data may not have been 

imported to TechPort by the Technology Owner. The lack of cost data in TechPort impedes the 

Technology Estimating capability. Currently, the TCASE (version 5.0) tool was designed to be 

configured to accept a download of the projects EXCELTM attribute file directly from TechPort. 

Future anticipated refinement of ONCE will be expected to be configured to allow the project 

data exported or downloaded directly from TechPort to ONCE. As the data will be “mirrored” from 

TechPort to ONCE, no modifications to TechPort will be necessary with respect to Technology 

Estimating. As ONCE will be the data repository for Technology Estimating, the project data will be 

normalized and evaluated for use in Technology Estimating. If errors are found in the data, which are 

located in ONCE, questions arise, or the data needs to be retired, then the TCASE Administrator will 

make the changes in ONCE or notify the respective Technology Owner directly as may be required. 
10 

Data Migration from CADRe to the TCASE Tool 

CADRe is NASA’s primary technology data file for projects meeting requirements of NPR 

7120.5E. The CADRe is prepared by the CAD using existing project data prepared during the life-cycle 

review process. By capturing key information, the CADRe tracks and explains changes that occurred 

from one milestone to the next, and it helps the project manager record in an Agency document all the 

events that occurred during the project both internal and external. The CADRe is a project owned 

document and is signed by the project manager; therefore, it does not include any independent 

assessments or evaluations, or opinions about the project. It simply records the known configuration at the 

specific milestone. The CAD provides the necessary funding and contractor support to prepare the 

document on behalf of the project using existing project documentation prepared during the milestone 

review process. Composed of three parts, the CADRe captures detailed programmatic, technical, and cost 

data in a standardized format. The document is prepared six times during the life cycle of a project at 

major milestones (SRR, PDR, CDR, SIR, Launch, End of Mission). Currently, CADRe and the data being 

included in the database are potentially migrating to the PCEC, and ONCE. Future refinement of ONCE 

should allow the data for technologies (projects) with TRL less than 6 to be included in TCASE. 

Data Migration from NPR 7150.2A Software Management Plans to the TCASE Tool 

Observations made during the research indicated that a significant portion of the technology 

effort might be attributed to the development of computer software. Software applications may be 

required for support other Technologies, or required as an integral part of the technology R&D. 

Segregating this software effort could be accomplished via use of the Software Management Plans 

(SMPs) required by NPR 7150.2A. The SMPs contain pertinent data about the software and includes the 

parameters that drive the cost and schedule. As identified in Table 3, the Technology Estimating 

parameters identified for computer software include project type, software class, primary language, 

number of requirements, and software size. 

Data Tracking 

Technology R&D projects may be initiated within any number of potentially 42 different NASA 

R&D programs. Technology owners may initiate a project in one NASA Technology Roadmap area and 

the project may take on follow on work that could progress into later phases of work, “morph” into other 

technology areas, or there may be successive parent-daughter technology projects. Tracking the initial 

R&D technology projects and its follow-on work is paramount to obtaining the total cost and schedule 

of a particular technology as would be accomplished in Technology Estimating. 

Based on the findings from this research, tracking a technology projects cost, schedule, and other 

pertinent project data through the succession of phases has been identified to be complicated by a number 

of items. These include: 

• The Principal Investigator left the project where there is no information related to their 

succession, 

• The R&D project work did not result in a formal technical paper, 

• Contract or research grant identifying numbers were not identified in the research technical 

paper, 

• The project administration of the research did capture the R&D project administration 

information in a centralized NASA location similar to TechPort, 

• The project records were archived and are difficult to retrieve, 

• Funding made by two separate NASA organizations on the same research project could not be 

connected without identification of the original contract number, or WBS numbers were 

unavailable, 

• Project funding was postponed, 
11 

• The research scope changed, or was transferred to another program, or 

• Information in the acknowledgment section of the technical paper did not indicate sufficient 

information to connect with previous or succession work. 

Tracking technology is best accomplished from the beginning of the R&D work where the 

Technology Owner’s researcher is notified of the requirement to complete and submit all the 

applicable information to TechPort at the conclusion of the work. This new requirement would be 

identified in two primary documents. The first is NASA’s Guidebook for Proposers Responding to a 

NASA Research Announcement (NRA) or Cooperative Agreement Notice (CAN), January 2013 Edition, 

and the second is NPR 5800.1 Grant and Cooperative Agreement Handbook, Section A, Part 1260, Grants 

ad Cooperative Agreements, § 1260.77, Closeout procedures, located at 

https://prod.nais.nasa.gov/pub/pub_library/grcover.htm.

During this research, it was identified that some technical papers reviewed did not always reveal a 

clear line of technology succession even though multiple papers were identified by author and successive 

dates. A possible solution to help reveal technology and their successive technologies is a new 

requirement to be incorporated in NASA’s Science and Technical Information (STI) paper submission 

process. The NASA documents to be modified to help in the tracking effort would include NPD 2200.1B, 

Subject: Management of NASA Scientific and Technical Information, Effective Date: November 19, 

2009, Expiration Date: November 19, 2014; NPR 2200.2C, Requirements for Documentation, Approval, 

and Dissemination of NASA Scientific and Technical Information, Effective Date: April 19, 2011, 

Expiration Date: April 19, 2016. Additional research should be conducted to determine the best method 

for tracking technologies and identified in the “Acknowledgement” section of the technical paper. 

Tracking technology R&D by the use of Work Breakdown Structure (WBS) as a means for 

classification of technology was not considered in this research. It was found that the majority of 

NASA‘s individual R&D low maturity projects with TRL less than 6 typically did not appear to have a 

WBS. Portfolios of low maturity R&D technology projects were thought to follow the portfolio 

requirements of NPR 7120.8, Chapter 5 by use of a Portfolio Project Plan as required by NPR 7120.8, 

NASA Research and Technology Program and Project Management Requirements (w/change 3 dated 04/18/13)

Section 5.2.5 Project Implementation, 5.2.5.1. In the Implementation phase, the R&T Portfolio project lead executes 

the R&T Portfolio Project Plan, which usually consists of one or more portfolio cycles (see section 5.2.5.7). 

These Portfolio Project Plans were not reviewed in this Technology Estimating research as the effort 

was outside the research scope. Further research should be conducted to determine the extent of 

tracking requirements as could be identified in the administration of NASA’s low maturity R&D 

projects by NPR 7120.8. This tracking of individual projects, as managed in the Portfolio Project Plan 

could prove useful and whether WBS for individual projects over portfolios would prove valuable to 

Technology Estimating. 

For reference, the WBS Template worksheets in the PCEC Library v0 have been formatted with a 

blue tab colors and each WBS worksheet contains a WBS. The following three WBS types are included 

in the PCEC: 1) NASA NPR 7120.5E WBS which is the NASA WBS. At this time the NAFCOM/PCEC 

CERs do not estimate to the WBS structure and is included in the PCEC for reference; 2) the CAD 

Requirements (CADRe) WBS which is used to collect data for NASA CADRe documents and is included 

in the PCEC for reference.; and 3) NFC12 WBS where it was derived from NAFCOM. The current 

PCEC/NAFCOM CERs align to these WBSs. Each template represents one of several major categories of 

systems produced by NASA. 
12 

Conclusion 

Technology Estimating and the TCASE tool will provide a significant capability to Technology 

Managers and Technology Owners. The use of TCASE as an analogy based estimating tool has been 

demonstrated and the tool is continuing to be improved. Once additional project data has been included 

from sources such as NASA’s TechPort, the value of the tool as an aid in estimating the cost and schedule 

of NASA R&D technology will increase greatly and its popularity for use in the cost community should 

expand. 

A critical gap in the Technology Estimating process was identified. The significant gap includes 

tracking initial and successive R&D technology projects. Closing this gap for tracking an initial project 

and a project’s follow-on work was determined to be paramount for obtaining the total cost and schedule 

of a particular technology. It was found during this work that additional research should be conducted in 

three primary areas to help support R&D technology project tracking, thus directly supporting the 

Technology Estimating process: 

• The first includes study to determine the best method for tracking technologies in technical 

papers. For example, these may include new requirements for specific notations to be identified in 

the “Acknowledgement” section of a technical paper. Further, where there is noted to be a 

succession of papers under the same contract, an annotation could be strategically identified. The 

implementation of these tracking requirements could be applied in NPD 2200.1B and NPR 

2200.2C. 

• The second area of additional study would be to determine the best methodology for tracking 

technology as it pertains to including project data in TechPort. It is suggested that tracking 

technology may best accomplished from the beginning of the R&D work, where the Technology 

Owner’s researcher is notified of the requirement to complete and submit all the applicable 

information to TechPort at the conclusion of the work, i.e. during closeout. Consideration of the 

appropriate changes to NPR 5800.1 could improve tracking of technology projects and support 

Technology estimating as Technology Owners could be required at the beginning of their work 

to complete data input to TechPort at the conclusion of the project. 

• The third area of additional study to be conducted would be to determine what technology 

tracking requirements that could be applied to the Portfolio Project Plan as required by NPR 

7120.8, and if the NASA WBS can be successfully applied to low maturity technology projects. 

The technology estimating parameters used in the estimating process were analyzed and based on 

both a statistical and a qualitative analysis. A total of 15 final Technology Estimating parameters have 

been recommended. Seven of these 15 parameters were identified to be already included in TechPort. A 

sequence and priority for the inclusion of the eight additional parameters into TechPort was identified. 

The eight additional parameters include System Hierarchy, Defining system characteristics, Computer 

Software, Level of Effort Research and Development Degree of Difficulty, Team Experience, Facilities 

and Infrastructure, and Unplanned Event(s). The addition of these parameters into TechPort should 

progress as a phased sequence where the benefit for including these eight additional parameters into 

TechPort will significantly outweigh the cost for their implementation. 

 
13 

References 

1. What is Technology Estimating, a proposal description, by the NASA Office of Evaluation, Cost 

Analysis Division, November 2012. 

2. GOA Cost Estimating and Assessment Guide, Best Practices for Developing and Managing Capital 

Program Costs, GAO-09-3SP, March 2009. 

3. Kruskal, W. H., Wallis, W. A., Use of Ranks in One-Criterion Variance Analysis, Jnl` American 

Statistical Association, Vol. 47, Issue 260, pages 583-621, DOI: 10.1080/01621459.1952.10483441, 

1952. 

4. DAU, Parametric Cost Estimate Method, accessed on January 2014 

https://dap.dau.mil/acquipedia/Pages/ArticleDetails.aspx?aid=36157b0b-69b4-4a0c-b16d2e978b4c425c.

5. Cole, S. K., Reeves, J. D., Williams-Byrd, J. A., Greenberg, M., Comstock, D., Olds, J. R., Wallace, 

J., DePasquale, D., Schaffer, M., Technology Estimating: A Process to Determine the Cost and 

Schedule of Space Technology Research and Development, NASA Technical Publication, TP-2013-

218145, December 2013. 

 
14 

Appendix A 

Qualitative Analysis of Technology Estimating Parameters 

 
15 

The following is a list of the parameters recommended for use in Technology Estimating process. The 

attribute of the parameter was identified as either qualitative or quantifiable and the use of both type 

would help describe the technology. Further, a rank for ease of administration of the parameter was 

identified. This ease of administration was based on observations made by the Research Team throughout 

the project. Generally, the guiding tenets for development of Technology Estimating parameters include 

that an item is a schedule/cost driver if a parameter tends to increase or decrease schedule/cost or is highly 

correlated with schedule/cost. 

Following each parameter is a narrative discussion. The discussion was provided to indicate any 

observations made during the research and help address the parameters suitability for use. 

1. Parameter related to Project Characterization: Technology Area (TA) 

a. Metric Attribute: Qualitative 

b. Rank Ease of Administration: High 

c. Schedule Driver: Yes 

d. Cost Driver: Yes 

DISCUSSION 

e. Technology areas in secondary and tertiary breakdown were sometimes difficult to apply by 

Technology Managers in all cases due to level of familiarity in application of the parameter. 

There is substantial documentation used to define these TAs and overtime, the use of the 

parameter will become easier for application and use. 

f. This parameter only applied to recent projects as the 14 TA Roadmap was promulgated within the 

last two years. 

g. Technology Area (TA) significantly helps to refine the Technology, however branching TAs can 

or may be questioned with regard to the accuracy of their application versus definition. 

h. Different individual TAs are naturally expected to have generally different schedule and costs. 

i. This parameter is included in TechPort. 

2. Parameter related to Project Characterization: System Hierarchy 

a. Metric Attribute: Qualitative 

b. Rank Ease of Administration: High 

c. Schedule Driver: Yes 

d. Cost Driver: Yes 

DISCUSSION 

e. This parameter was relatively easy to be applied by Technology Managers. The distinction 

between the various level of system hierarchy can be close, depending on the system. 

 EXAMPLE: If a satellite (mission i.e. “platform” or “system of systems”) had multiple subsystems comprising the total science package, it could not readily be determined that a 

technology was itself a complete system or was it to be included in the larger science system or 

platform. 

f. Will benefit from a standard reference and definition to help clarify appropriate level in the 

hierarchy. 

3. Parameter related to Project Characterization: Defining System Characteristics 

a. Metric Attribute: Qualitative and Quantifiable 

b. Rank Ease of Administration: High 
16 

c. Schedule Driver: Yes 

d. Cost Driver: Yes 

DISCUSSION 

e. Provides the detailed characterization of TAs. 

f. Helps to characterize between technologies in same TA and for branching technologies from a 

single TA. 

g. This parameter includes metrics such as mass, volume, power, etc. 

h. With future data, the characterization could reveal that for some TAs i.e. 08 Instrumentation, that 

they follow Moore’s Law where “number of transistors on integrated circuits doubles 

approximately every two years” or possess smaller volumes, and less mass due to increased 

computing capability. 

4. Parameter related to Project Characterization: Software 

a. Metric Attribute: Qualitative and Quantifiable 

b. Rank Ease of Administration: High 

c. Schedule Driver: Yes 

d. Cost Driver: Yes 

DISCUSSION 

e. Recommended that “Hardware” be dropped from original parameter (hardware versus software) 

as it did not provide significant correlation to amount or extent of software in all technologies. 

f. Suggested typical software cost drivers include: Project type, Software class, (i.e. A, B, C, D or 

E), Primary language, Number of requirements, and software size. 

g. Software development and its cost and schedule increases can be expected to be significant in the 

range of TRL 5 to 6. 

h. Technology papers for technologies with TRL < 6 indicate sometimes one or two persons 

exclusively dedicated to software when required. Extent of software used in R&D technologies 

was observed to be major influence to the cost and success of the technology. 

i. Data for software cost drivers are being collected as part of NPR 7150.2A Software Management 

Plan and can be used in Technology Estimating. 

5. Parameter related to Project Characterization: Key Performance Parameter (KPP) 

a. Metric Attribute: Qualitative and Quantifiable 

b. Rank Ease of Administration: High 

c. Schedule Driver: Yes 

d. Cost Driver: Yes 

DISCUSSION 

e. This parameter normally includes any and all technology project objectives. 

f. The KPPs provide significant and relevant data about the project and its intended functional 

objectives. 

g. This parameter provides the description in sufficient detail to compare it against similar 

technology projects. 

h. This parameter also provides technical details such that branching applications may be 

understood. 

i. When adequate data is provided, technologies with similar objectives may be compared by their 

respective schedule and cost. 

j. The parameter is included in TechPort. 
17 

6. Parameter related to Project Results: Level of Effort 

a. Metric Attribute: Quantifiable 

b. Rank Ease of Administration: High 

c. Schedule Driver: No 

d. Cost Driver: Yes 

DISCUSSION 

e. This parameter includes manpower estimate for NASA civil servant but does not capture the cost 

of universities, faculty, students, adjunct research investigators, etc. 

f. Currently, at the resolution of the project data obtained to date, the analysis indicates that the data 

cannot support that Level of Effort drives the schedule. 

g. Includes the capture the contracting costs in WYEs. 

7. Parameter related to Project Results: Cost 

a. Metric Attribute: Quantifiable 

b. Rank Ease of Administration: High 

c. Schedule Driver: Yes 

d. Cost Driver: Yes 

DISCUSSION 

e. This parameter includes the total cost (actual). 

f. It is suggested that the cost per year be obtained and identified with the respective year. 

g. The cost that should be identified as the cost in the management information system. 

h. The parameter is included in TechPort. 

8. Parameter related to Project Results: Schedule 

a. Metric Attribute: Quantifiable 

b. Rank Ease of Administration: High 

c. Schedule Driver: Yes 

d. Cost Driver: Yes 

DISCUSSION 

e. This parameter includes the total duration (actual). 

f. It is suggested that the total duration be identified by beginning and end dates. 

g. The duration that should be identified is the time in the management information system. 

h. The parameter is included in TechPort. 

9. Parameter related to Development Metrics: Technology Readiness Level 

a. Metric Attribute: Qualitative 

b. Rank Ease of Administration: High 

c. Schedule Driver: Yes 

d. Cost Driver: Yes 

 DISCUSSION 

e. This parameter includes the Technology Readiness Level as it is submitted by the Technology 

Manager, PI, PM or Technical Paper. 

f. It is suggested that the TRL for a technology be assessed by an unbiased referee. 

g. It has been reported that the TRL may undergo changes in definition in the future. 

h. The parameter is included in TechPort. 
18 

10. Parameter related to Development Metrics: Research and Development Degree of Difficulty (R&D3

) 

a. Metric Attribute: Qualitative 

b. Rank Ease of Administration: High 

c. Schedule Driver: Yes 

d. Cost Driver: Yes 

DISCUSSION 

e. This parameter includes five categories and can be difficult to administer. 

f. The use of this parameter in this research was impeded somewhat by subjectivity for its 

application. 

g. The statistical analysis reveal that the parameter may be a cost and schedule driver. 

11. Parameter related to Project Execution: Funding Sources 

a. Metric Attribute: Qualitative 

b. Rank Ease of Administration: High 

c. Schedule Driver: Yes 

d. Cost Driver: Yes 

DISCUSSION 

e. Comments from Technology Managers revealed that certain funding sources such IRAD 

programs do not provide enough time in the funding program to fully accomplish the technology 

project efforts. 

f. NASA undergoes a mix of funding programs, i.e. ETDP to GCPDO, etc. and the funding sources 

help to identify tracking of the technology and for branching of the technology. 

g. A gap exists for Tracking technologies from lower TRL to higher TRL, or from incubation to 

mission level. There is no way to identify if the work has been ceased due to funding, lack of 

capability or other reasons. The parameter was understood to be an imperative as it pertains to 

understanding technology infusion. 

h. The parameter is included in TechPort. 

12. Parameter related to Project Execution: Performing Organizations 

a. Metric Attribute: Qualitative 

b. Rank Ease of Administration: High 

c. Schedule Driver: No 

d. Cost Driver: No 

DISCUSSION 

e. Identification of organizations involved helps to identify tracking of the technology and for 

branching of the technology. 

f. The identification of organizations and points of contact is an imperative for tracking technology. 

g. There was a perception that the parameter could be a schedule or driver if a certain organization 

is identified with the Technology. 

h. The parameter is included in TechPort. 

13. Parameter related to Project Execution: Team Experience 

a. Metric Attribute: Quantifiable 

b. Rank Ease of Administration: High 

c. Schedule Driver: Yes 

d. Cost Driver: Yes 
19 

DISCUSSION 

e. Identification of the team structure goes directly to the schedule and cost driver of a technology. 

f. The team structure was not well documented in any of the data sources. 

g. The team makeup would be noted in project administration records and should be identified. 

Categories in this parameter include: Little or no experience conducting research, Experience 

conducting research but little or no experience with specific technology, Experience with research 

of this specific technology, Experience with this specific technology and innovative team. 

14. Parameter related to Project Execution: Facilities and Infrastructure 

a. Metric Attribute: Quantifiable 

b. Rank Ease of Administration: High 

c. Schedule Driver: Yes 

d. Cost Driver: Yes 

DISCUSSION 

e. Identification of the facilities or infrastructure goes directly to the schedule and cost driver of a 

technology. 

f. The use of aircraft for testing was expected to be a large expenditure, however the cost was not 

normally documented. 

g. Some laboratories were noted in the literature to be specialized and there should have been a cost 

identified for these labs. 

h. Certain University test facilities by were noted in discussion with Technology Managers to 

severely impact the delivery date of a technology project. These types of changed conditions 

should be noted in project administration records and their impacts noted for use in Technology 

Estimating. 

15. Parameter related to Programmatic Factors: Unplanned Events 

a. Metric Attribute: Quantifiable 

b. Rank Ease of Administration: High 

c. Schedule Driver: Yes 

d. Cost Driver: Yes 

DISCUSSION 

e. This parameter includes budget constraints and disruptions including any and all significant 

events to the capture of any or all impacts to the project such as change orders, requirements 

change, or unforeseen conditions. 

f. This parameter should include any singular event(s) that that contribute to or impact the cost or 

schedule of the project. 

g. This parameter also includes any budget constraints and disruptions during the term of the work. 
REPORT DOCUMENTATION PAGE Form Approved

OMB No. 0704-0188

2. REPORT TYPE 

Technical Memorandum

 4. TITLE AND SUBTITLE

Technology Estimating 2: A Process to Determine the Cost and Schedule 

of Space Technology Research and Development 

5a. CONTRACT NUMBER

 6. AUTHOR(S)

Cole, Stuart K.; Wallace, Jon; Schaffer, Mark; May, M. Scott; Greenberg, 

Marc W.

 7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES)

NASA Langley Research Center

Hampton, VA 23681-2199

 9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES)

National Aeronautics and Space Administration

Washington, DC 20546-0001

 8. PERFORMING ORGANIZATION

 REPORT NUMBER

L-20394

10. SPONSOR/MONITOR'S ACRONYM(S)

NASA

13. SUPPLEMENTARY NOTES

12. DISTRIBUTION/AVAILABILITY STATEMENT

Unclassified - Unlimited

Subject Category 83

Availability: NASA CASI (443) 757-5802

19a. NAME OF RESPONSIBLE PERSON

STI Help Desk (email: help@sti.nasa.gov)

14. ABSTRACT

As a leader in space technology research and development, NASA is continuing in the development of the Technology Estimating process, initiated in 2012, 

for estimating the cost and schedule of low maturity technology research and development, where the Technology Readiness Level is less than TRL 6. NASA’

s Technology Roadmap areas consist of 14 technology areas. The focus of this continuing Technology Estimating effort included four Technology Areas (TA): 

TA3 Space Power and Energy Storage, TA4 Robotics, TA8 Instruments, and TA12 Materials, to confine the research to the most abundant data pool. This 

research report continues the development of technology estimating efforts completed during 2013-2014, and addresses the refinement of parameters selected 

and recommended for use in the estimating process, where the parameters developed are applicable to Cost Estimating Relationships (CERs) used in the 

parametric cost estimating analysis. This research addresses the architecture for administration of the Technology Cost and Scheduling Estimating tool, the 

parameters suggested for computer software adjunct to any technology area, and the identification of gaps in the Technology Estimating process. 

15. SUBJECT TERMS

Cost; Estimating; Estimating Tool; Schedule; TCASE; Technology

18. NUMBER

 OF 

 PAGES

24

19b. TELEPHONE NUMBER (Include area code)

(443) 757-5802

a. REPORT

U

c. THIS PAGE

U

b. ABSTRACT

U

17. LIMITATION OF 

 ABSTRACT

UU

Prescribed by ANSI Std. Z39.18

Standard Form 298 (Rev. 8-98)

3. DATES COVERED (From - To)

5b. GRANT NUMBER

5c. PROGRAM ELEMENT NUMBER

5d. PROJECT NUMBER

5e. TASK NUMBER

5f. WORK UNIT NUMBER

122711.03.07.07.17.80 

11. SPONSOR/MONITOR'S REPORT

 NUMBER(S)

NASA/TM-2014-218251

16. SECURITY CLASSIFICATION OF:

The public reporting burden for this collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, 

gathering and maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this 

collection of information, including suggestions for reducing this burden, to Department of Defense, Washington Headquarters Services, Directorate for Information Operations and 

Reports (0704-0188), 1215 Jefferson Davis Highway, Suite 1204, Arlington, VA 22202-4302. Respondents should be aware that notwithstanding any other provision of law, no person 

shall be subject to any penalty for failing to comply with a collection of information if it does not display a currently valid OMB control number.

PLEASE DO NOT RETURN YOUR FORM TO THE ABOVE ADDRESS.

1. REPORT DATE (DD-MM-YYYY)

01- 04 - 2014