National Aeronautics and

Space Administration

Presentation Abstracts 

of the 

2023 NASA

Cost and Schedule

Symposium

May 2- 4, 2023

Jet Propulsion Laboratory

Pasadena, CA
2023 Abstract Collection

The Strategic Investments Division would like to welcome you to the 2023 NASA Cost and Schedule

Symposium. This document contains the names of the authors and abstracts for the presentations that 

will be given this year. In the Symposium Agenda you will notice that there is a unique ID number

mapped to each presentation. These same ID numbers can be used, within this document, to find the 

presentation abstract that you are interested in.

This year, with a cadre of excellent presentations and an awards banquet full of worthy nominations, 

the NASA Cost and Schedule Symposium will be a full and eventful three days!
Table of Contents

01_ Challenges Aggregating Multiple, Interdependent Projects into a Program Quantitative Schedule 

Risk Analysis Model.......................................................................................................................................5

03_Propulsion Cost Model Update...............................................................................................................6

04_Complexity the Right Way ......................................................................................................................7

05_ MBSE Cost Study ....................................................................................................................................7

06_ An Independent Schedule Risk Assessment (iSRA) – Utilizing Existing Objective Data and Subjective 

Applied Expertise for a Quantitative, Comprehensive and Credible Schedule Risk Assessment.................7

07_ Enhanced Tornado Sensitivity Analysis Process (ETSAP): Prioritizing & Taking Action on the Critical 

Program Inputs with the Highest Impact on Program Completion ..............................................................8

09_ Getting More out of the MS Project IMS .............................................................................................10

10_ The Changing Role of NASA Cost Analysis ...........................................................................................10

11_EVM Year in Review ..............................................................................................................................11

12_ CADRe: Year in Review........................................................................................................................11

15_ Minding your P’s and Q’s: Escalation in the NNSA ..............................................................................11

16_ How Effective are NASA’s Collaborations with the Industry?..............................................................12

17_ What's a metric got to do to be useful 'round here?: A SCaN Schedule Case Study...........................12

18_ The Schedule Deep Dive – An IBR Head Start? ....................................................................................13

19_ PCEC Robotic Missions – Challenges Getting Data and Statistics to Cooperate..................................13

20_ HLS Joint Confidence Level Analysis Approach....................................................................................14

23_ Preventing Increased Cost with Sound IGCEs......................................................................................15

25_ You Get What You Pay For: The New Frontiers Operations Cost Cap .................................................15

26_ NICM 10: The 2023 Release of the NASA Instrument Cost Mode .......................................................15

28_ Assessing Program Level Objectives of Human Mars Missions Using Portfolio Optimization Methods

....................................................................................................................................................................16

29_ Test Facility Request System (TFRS) Augmentations: Integration and Automation of Facilities 

Scheduling and Workforce Planning for All of Glenn Research Center’s Aero and Space Testing 

Capabilities..................................................................................................................................................17
30_ Wait, What? How Competed Missions Might Be Experiencing More Cost Growth than Directed 

Missions......................................................................................................................................................18

31_Math is EZIE (aka Math is Hard 2.0): How Contracts Help Control Cost...............................................18

32_ Aerospace Viewer of NASA Project Staffing Data (aView): A Practical Tool for Analyzing Staffing 

Levels and Cost Across Missions.................................................................................................................19

34_ Shaping Workforce with Business Intelligence....................................................................................19

36_ Thresher and TIDBIT: Tools on Automating Schedule Risk Assessments ............................................20

37_ Breaking Down Data Silos....................................................................................................................21

38_ Successful EVM Implementation Depends on Surveillance.................................................................21

39_ The Smart Projects and Reviews with Transformative Analytics (SPARTA) Project ............................22

40_ Mission Operations Cost Estimation Tool (MOCET) ............................................................................23

41_ The Mystery of the Metrics: Contractor Schedule Management........................................................23

42_ Agile Methodology Applied to Systems Engineering in the Domain of Cost Analysis of a 6U CUBESAT

....................................................................................................................................................................24

45_ Through the Looking Glass- Why EVM Is An Essential Risk Mitigation Measure for Decision Makers 

and Program Managers at NASA ................................................................................................................24

47_ A Significant Other: Hypothesis Testing on Qualitative Predictor Coefficients in CER Regression .....25

48_ Aleatoric and Epistemic Uncertainty Quantification in Bayesian Dirichlet Cost Rules of Thumb .......25

49_ ASCoT 3: Nonlinear Principal Components Analysis and Uncertainty Quantification in Early Concept 

Spacecraft Flight Software Cost Estimation................................................................................................26

50_ Analogy Cost Estimation for CubeSats using COMPACT......................................................................26

52_ Joint Confidence Level Analysis for Projects with Multiple Objectives...............................................27

53_ Quantified Benefits of Earned Value...and Their Benefits...................................................................29

54_ Programmatic Cost Tool.......................................................................................................................29

56_ Yes, NASA, a project can be cancelled .................................................................................................29

57_ EVM Analysis Tool for Complex Projects with Varied Data Sources....................................................30

58_ Examining the Effects of Implementing Data-Driven Uncertainty in Cost Estimating Models............31
59_ Bayesian Quantitative Risk Analysis.....................................................................................................32

60_ Distribution Free Uncertainty for CERs ...............................................................................................32

61_ONCE Database .....................................................................................................................................32

62_ Diving Deep in the Domains, Running with AzTech.............................................................................33

64_Risk Management .................................................................................................................................33

65_ Using Excel to Facilitate JSC Analyses..................................................................................................34
01_ Challenges Aggregating

Multiple, Interdependent 

Projects into a Program 

Quantitative Schedule Risk 

Analysis Model

Authors: David Hulett, Ian Bailey, Lorrie Tietze

Presenters: David Hulett, Ian Bailey, Lorrie

Tietze

Abstract: Some complex programs are 

structured so that several projects must interact 

with each other to produce the final integrated

deliverable. The constituent projects have their 

own project schedules, that may be large and 

detailed, and are typically assigned to different 

entities or sites that may be in the same 

umbrella company or government agency.

• The constituent projects are specialized 

entities, focusing on design, fabrication of 

various components, integration and 

production of multiple units for the ultimate 

customer.

• They have their own deliverables that are 

required by other projects in the program.

Ultimate delivery of the first product, a 

major program milestone, is built from the 

efforts of these interdependent entities.

• Each project within the integrated program

has its own level of uncertainty and set of 

risk events. Due to the interdependencies 

between projects, a delay experienced by

one entity, the “giver” of a product is 

transmitted to another entity, the “receiver”

of that same product. If the receiver does 

not know about the delay early in their own 

planning, and that delay is probabilistic, the 

ultimate first-product delay could be weeks 

to months.

• There are risk events that affect only one of 

the specialized entities and others that 

affect several of the entities. The 

cumulative effect of a risk event impacting 

multiple entities could be much larger to the 

overall program than any individual risk 

event.

• A risk analysis for any giver entity can 

forecast a delay in giving the receiver 

needed input. Often the receiver does not 

know about the delay early in its own 

planning, so the entities’ conduct of risk 

analysis need to be coordinated and, ideally, 

analyzed in the order in which the 

interdependent activities will be conducted.

In addition, that delay is probabilistic and 

may be a few weeks or several months.

• The receiver entity needs to incorporate the

delay in its own planning. The knock-on 

effects of the giver’s delay on the receiver’s

plans need to be represented somehow so 

the receiver’s output (may be to another 

receiver such as the fabricating entity) will 

correctly reflect the delay. Alternative ways 

to communicate the delayed delivery would 

be to select a date determined by the giver’s 

risk analysis, say the P-70 date, or a duration 

of the predecessor’s work on the product, or 

a complete probability distribution of the 

giver entity’s possible dates / durations.

• Broader risks impacting multiple entities 

need to be incorporated into the overall 

programs’ commitments and funding 

requirements.

The consequence of such an integrated program 

on the final delivery to the customer needs to be 

understood and probably mitigated in some way 

for a better result. Prioritization of the risks 

must account for risks that affect one project, 

multiple projects or the entire program. Risk 

mitigation should affect risks where it provides 

the most benefit to the final program’s delivery 

to the client.

The risk analysts need to understand which risks 

have serious effects on the giver-entity’s 

delivery, but also what is occurring with other 

entities working in parallel in another sequence.

The program-level review deals with parallel vs. 

sequential entity work resulting in the final 

product’s delivery date. Some risks affect more 

than one entity so the analysis has to reflect the 
distribution of the effects of risks at the program 

level, which is made up of its constituent parts.

Strategies for developing a program-level 

schedule must take into account the size of the 

projects’ schedule, which have thousands of 

activities. It may not be feasible to build a single 

interrelated program-level schedule. Other 

strategies might include developing top-level 

projects and represent the projects’ deliverables 

to other projects in a summary fashion. The

chosen strategy must take into account the 

maintenance required to ensure that the 

program schedule is truly and consistently 

representative of the work in the individual 

project schedules.

This paper will explore these questions using 

small, case-study schedules with giver-receiver 

logic between project in the program:

• Building individual program entities’ 

schedule risk model

• Linking the output of some projects that are 

inputs to the work of other projects

• Evaluate the integrated program –

alternative strategies

• Prioritization of risks for focused risk 

mitigation across all entities

03_Propulsion Cost Model

Update

Authors: Richard Webb

Presenters: Richard Webb

Abstract: This presentation will provide an 

update on the status of the Propulsion Cost 

Model (PCM). PCM is a model for estimating 

liquid rocket engines, nuclear thermal 

propulsion systems, and solid rocket motors. It 

can be used as either an add-on to PCEC or as a 

standalone model. The model, which is being 

developed and released in phases, greatly

expands the existing CASTS estimating

capabilities in these areas. We will present an 

overview of the entire model, with greater 

depth provided regarding the newly completed

solid motor portion in particular. The

development of the CERs for the solid portion 

have been completed, thus completing CER 

development for the entire model. Integrating 

the solid CERs into the PCM spreadsheet models

and completion of the solid motor Virtual Black 

Book (VBB) documentation set are the only 

tasks remaining prior to full release of the 

model.

The breadth and depth of the solid motor 

historical cost database have been greatly 

expanded beyond that included in the original 

CASTS data set. The depth has been expanded 

from 18 to 55 historical data points for Flight 

Unit costs, including additional data points 

received through a data sharing agreement with 

the Missile Defense Agency and other REDSTAR 

sources. With the addition of the new data 

points, the breadth has been expanded such 

that the database, formerly comprised of largediameter solid motor Booster data only in 

CASTS, has been segregated into 5 categories 

depending upon the SRM usage and size, 

including: Boosters, Stages, Upper Stages, Kick 

Motors, and Sounding Rockets. In addition, the 

nature of the expanded data set necessitated a 

change in the primary independent variable 

from Total Impulse to Gross Weight to better 

accommodate the wider range of sizes and uses 

of the motors in the database.

The unrestricted version of the CERs developed 

for each of the 5 categories will be presented as 

well as examples of the VBB information to be 

made available both to the general public

(unrestricted) and redacted examples of 

restricted cost data available to authorized 

users.
04_Complexity the Right Way

Authors: Andy Prince

Presenters: Andy Prince

Abstract: Over the past several years I have 

engaged in a (hopefully) good natured 

discussion with several of my peers over the 

proper use of complexity factors in cost 

estimating relationships (CERs). While I still 

stand by contention that complexity factors 

should not be used as a parameter in a CER, I

theorize that using complexity factors to adjust

the output of a CER is a reasonable and sound 

approach to better cost estimates.

(By the way, this is how cost estimating was 

often done back in the days of weight-based and 

other single variable CERs, this method is not 

new, but possibly has been forgotten)

In my presentation I will share my results from

developing and testing complexity factors based 

on a simple CER and the underlying data. The 

CER is developed using log-transformed least 

squares. The CER forms the basis for both 

developing and testing the complexity factors.

Statistics such as r-squared, root mean squared 

deviation, and mean absolute error are 

calculated. Use of simulation and crossvalidation are explored as ways to test the 

efficacy of the complexity factors.

05_ MBSE Cost Study

Authors: Brook Cavell, Kirsten Lam

Presenters: Brook Cavell, Kirsten Lam

Abstract: In order to understand the costing 

and timing nature of Model Based System 

Engineering (MBSE), The Aerospace Corporation 

executed a study to cover the following facets of 

MBSE impact on an acquisition:

• What is the initial cost for setting up 

MBSE themed support for a 

project/program?

• What are the over time costs for MBSE 

for configuration management and 

curation of the models?

• What key MBSE factors and quantifiers 

should the government program office 

consider when reviewing a proposal?

• What are possible +/-/ugly features of 

MBSE versus traditional system 

engineering methods?

• What is the LCCE for implementing

MBSE?

The objective of the effort was to study the cost 

estimation of MBSE by gathering information to 

support key qualifiers/quantifiers to assess 

costs, ask the questions, and investigate cost 

drivers derived from the literature review and 

root cause analysis of the problems including 

inputs from MBSE experts.    This presentation 

highlights the results of the study and provides 

preliminary answers to these questions.

06_ An Independent Schedule 

Risk Assessment (iSRA) –

Utilizing Existing Objective 

Data and Subjective Applied 

Expertise for a Quantitative, 

Comprehensive and Credible

Schedule Risk Assessment

Authors: Charles Delio, Daniel Donaldson, 

Matthew Gonzales, Vladimir Karavodin, James 

Quilliam

Presenters: Charles Delio, Daniel Donaldson, 

Matthew Gonzales, Vladimir Karavodin, James 

Quilliam

Abstract: This case study is an effort to 

document and share with practitioners the 

results of an independent schedule risk 

assessment process. This process utilizes both 
objective and subjective assessment areas for 

comprehensive, quantitative, and credible SRA 

results for decision makers.

This case study will provide an in-depth 

explanation of the principal objective and 

subjective assessment criteria of the process.

The elements of this process will focus first on 

three (3) primary objective assessment criteria 

of performance metrics, program risk/s, and 

historical (analogous) program data. The second 

part of the comprehensive process will focus on 

three (3) primary subjective assessment criteria 

of subject matter expert (SME) risk ratings, 

known risk issues (not yet formulated) and 

possible what-if excursions (What-if scenarios). 

The results of implementing a comprehensive 

objective and subjective based process utilizing 

these six (6) data criteria should greatly enhance 

the understanding and confidence that 

leadership and project teams have in the results 

of this independent comprehensive schedule 

risk assessment. It will also assure that sound 

decisions are being made based on the reliance 

of these crucial schedule risk assessment 

criteria. This study also establishes a foundation 

for future research into schedule risk 

assessment tool results, accuracy and 

capabilities. 

When completed, the attendee will be able to:

1. Comprehend the differing aspects of an 

independent schedule risk assessment 

process and the application of this tool 

as lessons learned for practitioners. 

2. Analyze the value of applying the six (6) 

principal objective and subjective 

assessment criteria of the independent 

schedule risk assessment analysis tool.

3. Synthesize the benefits of 

understanding and utilizing the 

comprehensive objective and subjective 

based process versus traditional 

schedule risk assessment practices and 

procedures for sound decision making 

and accurate and reliable schedule risk 

assessment results.

07_ Enhanced Tornado 

Sensitivity Analysis Process 

(ETSAP): Prioritizing & 

Taking Action on the Critical 

Program Inputs with the 

Highest Impact on Program 

Completion

Authors: James D. Quilliam

Presenters: James D. Quilliam

Abstract: This case study is an effort to 

document and share with practitioners the 

enhanced tornado sensitivity analysis (ETSAP) 

process. This process utilizes a comprehensive 

five (5) step process during a schedule risk 

assessment to prioritize and provide the critical 
program inputs for decision makers to take 

management action on tasks with the highest 

impact on program completion.

This will provide an in-depth explanation of the 

principal assessment criteria of this novel 

process for practitioners. In order to fully 

comprehend this process, the case study will 

focus first on the output charts of a traditional 

schedule risk assessment. Next, tornado 

sensitivity attributes with be reviewed along 

with the importance of utilizing the population

of all available tornado sensitivity options. A 

comparison will be provided on the traditional 

vs. the enhanced tornado sensitivity process 

culminating with the ability to extract primary 

and secondary risks. Once this is accomplished, 

the enhanced tornado sensitivity analysis 

process flow will be covered in detail. This will 

allow for a complete understanding of the 

process steps. The enhanced tornado sensitivity 

analysis approach for prioritizing the top risk 

inputs will then be shared. A detailed 

representation of the five process steps will be 

outlined culminating with a summarized 

overview of the comprehensive process. From 

this one-page summary comes the listing of the 

harvested top primary and secondary risk inputs 

culled from the assessed tornado results as part 

of the schedule risk assessment simulations.

The results of implementing this comprehensive 

tornado assessment process utilizing the five (5) 

step criteria should greatly enhance the 

understanding and confidence of leadership and 

project teams. This will allow them to focus on 

the critical risk inputs that are crucial to the 

successful completion of their programs. It will 

also assure that sound decisions are being made 

based on the reliance of these crucial schedule 

risk assessment drivers. These primary and 

secondary level tasks from the tornado 

sensitivity report are significant since they are 

most likely to cause a delay to project 

completion and/or provide an opportunity to 

reduce the remaining duration of the project. 

The benefits of this enhanced analysis to 

uncover these important risk drivers far 

outweigh the added minimal processing time at 

the end of a simulation. The enhanced tornado 

sensitivity analysis provides a listing of the tasks 

requiring increased management attention and 

for providing meaningful information when 

making decisions about which tasks to mitigate 

in an effort to improve project completion 

dates. This study also establishes a foundation 

for future research into schedule risk 

assessment tool results and insuring data use 

and credibility.

When completed, the attendee will be able to:

• Comprehend the differing aspects of the 

enhanced tornado sensitivity analysis 

(ETSAP) process and the application of this 

tool as lessons learned for practitioners.

• Analyze the value of applying the five (5) 

step process during a schedule risk 

assessment to prioritize the critical program 

inputs for decision makers in order to take 

management action on tasks with the 

highest impact on program completion.

• Synthesize the benefits that the enhanced 

tornado sensitivity analysis provides in 

listing the tasks requiring increased 

management attention and for providing 

meaningful information when making 

decisions about improving project 

completion dates while establishing a 

foundation for future research into schedule 

risk assessment tool results.
09_ Getting More out of the 

MS Project IMS

Authors: William Paradis

Presenters: William Paradis

Abstract: Have you ever considered the price 

paid to construct and maintain an Integrated 

Master Schedule or IMS? On any given flight 

project here at NASA, there is one or more 

Planner/Schedulers who work full time 

maintaining the IMS over the course of the 

project? A lot of work goes into the IMS and 

there is a lot that comes out and providing an 

early warning system to the project of future 

project schedule impacts is probably the biggest 

thing. It is a great tool, it’s like an anti-acid tablet 

helping to relieve heartburn caused by the 

alternative of not planning. What else can we 

get out of the MS Project file or how do we get 

more? The earned value folks frequently come 

up with new ways to get the schedulers to get 

more for them making life easier for the earned 

value analyst but what can schedulers do to get 

more out of the IMS to make life easier for 

themselves and the projects they support? I see 

schedule products get developed as standalone 

products, hand drawn charts. I see project 

personnel combing through MS Excel snapshots 

of the IMS to create one time use charts for 

upcoming reviews; and many times, I do not 

think they have to. So, what products or data 

can be extracted right out of the IMS that would 

help save time for others on the project to help 

simplify things? Folks who mine data for 

schedule, hand draw charts, or develop 

standalone schedules can benefit from things 

extracted from the IMS and do not even know 

it? How can schedulers get more out of the MS 

Project and be of more help to the project? So 

many things get tracked and analyzed in parallel 

that if we put our minds to it; we can become a

more value-added resources to the project and 

to NASA by more fully using the MS Project file. 

I’ve prepared a presentation to share some 

ideas and examples to enlighten schedulers of 

how to get more out of the MS Project file then 

they have before and would like to share it with 

the schedule community; the things I will share 

in this presentation has proved beneficial to me 

and my projects and I hope it does the same for 

you; I hope to see you there.

10_ The Changing Role of 

NASA Cost Analysis

Authors: John Dotson

Presenters: John Dotson

Abstract: Kennedy Space Center (KSC) has 

undergone significant changes with the advent 

of commercial space providers. With the rise of 

private companies vying for a share of the space 

market, KSC has been forced to adapt and find 

ways to remain competitive. One of the key 

challenges facing KSC today is a reduced budget 

and a corresponding need to do more with less 

money.

One approach KSC has taken to address these 

challenges is to embrace a more cost-effective 

and efficient business model. This has involved 

exploring new technologies and processes 

known as Affordability, such as automation and 

digitalization, to reduce the time and resources 

required for operations. Additionally, KSC has 

sought to form partnerships with private 

companies and other organizations to share 

resources, knowledge, and expertise.

Overall, the changes that have come with the 

addition of commercial space providers have 
been significant and challenging for KSC. 

However, the center has been proactive in 

addressing these challenges and remains 

committed to exploring new technologies and 

processes to maintain its position as a leader in 

space exploration and technology. By trying 

embracing a more cost-effective and efficient 

business model and developing new spacecraft 

and launch vehicles, KSC is working to ensure its 

continued success in an increasingly competitive 

marketplace but may prove to be too little too 

late.

11_EVM Year in Review

Authors: Jon Fleming, Kristen Kehrer

Presenters: Jon Fleming, Kristen Kehrer

Abstract: The EVM Year in Review highlights 

some of the great accomplishments and 

progress by the Agency Team and EVM 

practitioners across NASA’s diverse portfolio of 

programs and projects. An overview of the 

support and products provided by OCFO-SID, the 

stewards of the EVM Discipline, will be discussed 

along with several efforts such as: EVMS 

Surveillance, NASA-led EVMS Compliance 

Review, Integrated Baseline Review 

Improvement Initiative, Agency’s EVM Software, 

Integrated Program Management Data Analysis 

and Report (IPMDAR) Rollout, and more!  Please 

visit nasa.gov/evm for more information.

12_ CADRe: Year in Review

Authors: Eric Plumer

Presenters: Eric Plumer

Abstract: The Cost Analysis Data Requirement 

(CADRe) is a formal project document that 

describes the programmatic, technical, and lifecycle cost, schedule, and risk information of a 

project. CADRe is NASA’s unique response to 

improve cost and schedule estimates during the 

formulation process, providing a common 

description of a project at a given point in time. 

By capturing key technical and programmatic 

information, the CADRe tracks and explains 

changes that occur from one milestone to the 

next. Completed CADRes are available on the 

One NASA Cost Engineering Database (ONCE) 

database, a secure, web-based application 

which allows for easy retrieval and fast analysis 

of CADRe data across multiple projects and 

milestone events. Utilization of CADRe data 

supports analysis of important project attributes 

and enables project managers to develop 

improved cost and schedule estimates. 

Additionally, CADRe documents provide a 

wealth of project data that enables countless 

types of analysis to support data driven decision 

processes. This presentation will provide an 

overview of CADRe and describe recent CADRe 

accomplishments and planned future 

enhancements. 

15_ Minding your P’s and Q’s: 

Escalation in the NNSA

Authors: Mike Metcalf, Alan Karickhoff, Brian 

Flynn, Omar Akbik, Ray Vera

Presenters: Mike Metcalf, Alan Karickhoff, Brian 

Flynn, Omar Akbik, Ray Vera

Abstract: Inflation ran at its highest levels in 

decades through 2021 and 2022 in the U.S. and 

abroad. The U.S. dollar in 2022 was worth about 

73 cents compared to ten years earlier, or 73 
cents of buying power in the marketplace. The 

same situation of diminished value holds for a 

government budget, a salary, a pension, a 

dividend, or a company’s cash balances. Many 

firms and government organizations are 

attempting to develop custom escalation indices 

for labor, material, and construction in the face 

of high inflation. But questions remain as to the 

consistency of any such indices with local labormarket conditions, their technical quality and 

accuracy, the scope of their coverage, and the 

degree to which self-fulfilling prophecies are at 

play. 

The authors ventured to understand the 

National Nuclear Security Administration’s 

(NNSA’s) unique labor and material environment 

and its relationship to the broader inflationary 

market. We established a view of escalation 

history for the labor and materials markets in 

which the NNSA participates, focusing on 

nuclear construction, non-nuclear construction, 

and weapons programs. We then used the 

broader financial market to build probability 

distributions of inflation rate forecasts, using 

financial instruments actively traded on Wall 

Street, such as Treasurys and inflation 

derivatives. As economists note, focus sharpens 

and credibility rises when prices are set by 

market agents that bear financial risk, such as 

pension, insurance, and hedge-fund managers. 

By combining market projections with NNSA’s 

recent history, we built unique composite 

escalation indices and probability distributions 

for each NNSA Site and each category of 

program, suitable for use in cost estimates, 

AoAs, budget estimates and other estimates 

where contractual rates have not been 

established.

16_ How Effective are NASA’s 

Collaborations with the 

Industry?

Authors: Moon Kim

Presenters: Moon Kim

Abstract: In line with the national policies for 

fostering a robust commercial space industry, 

NASA’s modern day acquisition strategy involves 

more collaborations with the industry. In forms 

of public-private partnerships and commercial 

developments, industry collaborations have 

gained both internal and external support and 

will continue to be a major part of the Agency’s 

strategy. As an effort to assess the effectiveness 

of NASA’s collaborations, this study developed a 

set of measures of effectiveness (MOEs) that 

encompass cost-savings, access to capability, 

and market development. Using the MOEs, the 

study analyzed 16 space flight programs of 

various collaboration types using a mixedmethods approach. Based on the MOEs, the 

study found a varied collection of effective, 

partially effective, and ineffective outcomes. 

Furthermore, in the assessment process, the 

study found several topics of importance the 

Agency could consider for future collaborations. 

The study initiates a critical discussion of 

performance metrics in the absence of a formal 

Agency process to assess the effectiveness of 

the emerging procurement arrangement types.

17_ What's a metric got to do 

to be useful 'round here?: A 

SCaN Schedule Case Study

Authors: Kailey Melton, David Payne
Presenters: Kailey Melton, David Payne

Abstract: To be worth their salt, metrics need 

to provide actionable insights to decisionmakers. Metrics also are like lunch; they are 

never free - they impose burdens, and the 

benefit needs to be worth the cost. And for 

better and worse, Heisenberg's lessons on 

observers applies to schedules as the choice of 

measurements alters the thing measured. Over 

the past year, the loosely-coupled SCaN

Program has revamped its schedule metrics 

approach with lessons learned and a 

smorgasbord of analogies and metaphors to 

share.

18_ The Schedule Deep Dive –

An IBR Head Start? 

Authors: Christopher Sadler, Melissa Lee

Presenters: Christopher Sadler, Melissa Lee

Abstract: Miriam-Websters dictionary defines 

the Deep Dive as “an exhaustive investigation, 

study, or analysis of a question or topic.” 

Industry defines the Schedule Deep Dive (SDD) 

as “a schedule analysis that identifies anomalies, 

facts and issues that can happen throughout the 

process of the project due to both intentional or

unintentional actions. This is used to verify 

compliance with the client's contract 

specification, overall schedule coherence and 

logical sequence between activities.” SDDs are 

used across the NASA agency at various times; 

however, many projects across the agency are 

not using this process as a pre-cursor to the 

Integrated Baseline Review (IBR).

The IBR is a risk-based review conducted to 

ensure a mutual understanding between the 

customer and supplier of the inherent risks in 

the supplier’s Performance Measurement 

Baseline (PMB). A secondary, but equally 

important purpose, is to ensure the PMB is 

realistic for accomplishing all the authorized 

work within the authorized schedule and budget 

across the five risk areas of technical, cost, 

schedule, resources, and management 

processes. The SDD process can give projects a 

head start on the IBR by revealing valuable 

points of focus to assist in gaining the mutual 

understanding between customer and supplier.

This session will provide background 

information on SDDs, the process of conducting 

of an SDD, and uncover the benefits of 

conducting a deep dive before the IBR. Our goal 

is to introduce Schedule Deep Dives as a best 

practice prior to IBRs.

19_ PCEC Robotic Missions –

Challenges Getting Data and 

Statistics to Cooperate

Authors: Shawn Hayes, Mark Jacobs, Brian 

Alford

Presenters: Shawn Hayes, Mark Jacobs

Abstract: Since the release of PCEC v2.3, many 

new projects have been added to the PCEC 

Robotic Mission Database. During the 

application of the PCEC normalization process to 

the new project data, several challenges arose, 

such as; capturing of COVID impacts, 

appropriate splitting of the different flight 

elements of Mars landed missions, and 

limitations associated with Firm Fixed Price (FFP) 

contracts. While deriving updated CERs, it was 

observed that CER performance is not as good 

for flagship missions and some of the high-end 

outliers which had complex payloads that were 
not well-represented by CER input candidates. 

Multiple alternative approaches have been 

explored including Classification and Regression 

Tree (CART) analysis, payload accommodations 

Figures of Merit (FoM), and category specific 

CERs.

New missions added to the database include; 

SMAP/JPSS-1/GOES-T (Earth Sciences), Solar 

Probe (Helio-Physics), TESS/IXPE/JWST 

(Astrophysics), and Mars 2020/DART/Lucy 

(Planetary). Collectively, these new missions 

cover a broad range of mission types. All

missions in development after March 2020 

experienced some impact from COVID. 

Normalization efforts are wrestling with the 

question - Is this the new normal?

The Mars landed missions (MER/MSL/Mars 

2020) have multiple flight elements (Cruise 

Stage, Entry/Descent/Landing, and 

Lander/Rover) that were not clearly separated 

during development. Methods to allocate costs 

to each element have been over-simplistic 

(mass-based). Analysis of the data indicates a 

mass-based approach over-estimates the Cruise 

Stage and EDL and under-estimates the 

Lander/Rover.

Multiple alternative methods are being 

explored. These include CART analysis, FoMs for 

Payload Accommodations, and Category Specific 

CERs. CART analysis may make it possible to 

include larger directed missions, which were 

outliers in the PCEC v2.3 CERs, providing more 

inclusive CERs. The lower-cost end of the 

mission spectrum may also see improvements.

PCEC inputs characterizing the payload are 

limited. Payload mass and power are input 

candidates for System I&T, but mass and power 

do not seem to accurately affect payload 

complexities that affect System I&T. Other 

options to characterize the payload complexity 

are being explored. Metrics under consideration 

for a Payload accommodation FoM include 

mass, power, physical volume, thermal 

requirements, data rates, and pointing 

requirements. FoMs can be included as input 

candidates for the PCEC CER process to test 

their merit.

Multiple category specific CERs are being 

explored. Categories include Directed or AO 

Mission, Flagship-specific, or by SMD directorate 

(Planetary/Helio-Physics/Earth 

Sciences/Astrophysics).

As the normalized data is studied while 

developing PCEC CERs, weaknesses from data 

limitations can arise. PCEC data collection and 

normalization efforts have been continually 

refined to incorporate lessons-learned from CER 

development. These challenges that have been 

encountered as work progresses towards the 

next release of PCEC will be discussed.

20_ HLS Joint Confidence 

Level Analysis Approach

Authors: Brian Alford

Presenters: Brian Alford

Abstract: The Human Landing System Program 

acquisition approach uses Firm Fixed Price (FFP) 

contracts for the development and 

demonstration of an initial HLS system for the 

first Artemis crewed flight demonstration to the 

lunar surface. A Joint Confidence Level (JCL) 

analysis is typically one of the key inputs to the 

Key Decision Point -C (KDP-C) Agency Baseline 

Commitment (ABC) decision, however, 

traditional JCL analyses at NASA have not 

involved commercial programs with Firm Fixed 
Price contracts. This presentation will look at 

the HLS FFP services approach and how the 

major differences from a traditional JCL are 

factored into modeling cost and schedule 

growth. Future work to improve modeling will 

also be discussed.

23_ Preventing Increased 

Cost with Sound IGCEs

Authors: John Moore, Erin Roberts

Presenters: John Moore, Erin Roberts

Abstract: To inform the Cost Estimating 

Community on current policy as to when IGCEs 

are formally required. This session will identify 

and discuss the background areas of Federal 

Acquisition Regulations (FAR) driving the need 

for development and use of sound IGCEs as well 

as lessons learned from weak IGCE development 

and/or documentation. Sound IGCEs serve the 

procurement process from planning to award. 

Best practices for determining and developing 

adequate level of detail, documentation style, as 

well as when and how interactions between cost 

estimator and procurement personnel is best 

timed will be discussed. There can be 

significant, complex differences in estimating 

techniques between Program hardware buys 

and Institutional services. But if the different 

estimating techniques are applied and 

documented effectively, either method can 

result in the development of sound IGCEs. 

Benefits of a clearly stated and documented 

IGCE result in successful procurement reviews, 

timelines, and ultimately set the program up for 

effective performance.

25_ You Get What You Pay 

For: The New Frontiers 

Operations Cost Cap

Authors: Ben Clare, Kathy Kha, Rachel Sholder

Presenters: Ben Clare, Kathy Kha

Abstract: As budgets get tighter, NASA is 

increasingly looking at different ways to control 

costs. In the latest draft Announcement of 

Opportunity for New Frontiers 5, a cost cap is 

being implemented for Phase E and F. Capped at 

$300M FY22$, this seemingly limits both the 

destination and type of mission that will be 

possible. But exactly how limiting is it? In this 

presentation, we will go back and examine the 

Phase E costs of previous Discovery and New 

Frontiers missions and how well they would 

have fared with a $300M FY22$ cost cap to see 

what seems feasible for $300M FY22$.

26_ NICM 10: The 2023 

Release of the NASA 

Instrument Cost Mode

Authors: Joe Mrozinski, Luther Beegle, Kyle 

Brown, Robert Cesarone, Michael DiNicola, 

Samuel Fleischer, Michael Fong, Melissa Hooke, 

Alfred Nash, Sherry Stukes, Marc Walch 

Presenters: Joe Mrozinski

Abstract: The newly released NASA Instrument 

Cost Model (NICM 10) will be introduced and 

demonstrated. NICM 10 includes 36 instrument 

cost and schedule estimating relationships, 

including 6 new models. The demonstration will 

focus on many of the upgraded capabilities 

including: 1) the new repeatable, analytic cost 

estimating solutions in both the System and 
Subsystem Tools, 2) the new isoquants displays 

in the JCL plots, 3) the improved Bayesian 

imputation method with boundary conditions, 

4) K Nearest Neighbors weighted average 

estimating capability added to the NICM Search 

Engine Outputs along with new summary data 5) 

Expanded Search Engine capabilities, and 

more!

28_ Assessing Program Level 

Objectives of Human Mars 

Missions Using Portfolio 

Optimization Methods

Authors: Bill O’Neill

Presenters: Bill O’Neill

Abstract: The large number and significant 

variety of systems available for space 

exploration missions produce countless 

potential architecture combinations. 

Compounding this are the scheduling intricacies 

of system lifecycle phases, time dependent 

operational dependencies, as well as the 

uncertainty associated with each technology in 

terms of cost, schedule, and performance. 

Traditional space mission architecting 

emphasizes the individual design of component 

systems over the wide-ranging and robust 

assessment of architecture options early in 

mission design. A top-down method that can 

assess the capabilities, requirements, and risks 

associated with the diversity of available space 

systems and select optimal portfolios of 

interdependent systems based on stakeholder 

objectives is necessary. Our work describes and 

demonstrates a portfolio optimization technique 

that can design and assess Mars exploration 

architectures by optimizing on programmatic 

objectives such as cost, performance, schedule, 

and robustness while simultaneously accounting 

for system operational interdependencies and 

schedule dependencies of the selected systems. 

This adaptation of portfolio optimization is 

further differentiated by including system sizing 

relationships within the architecture as well as 

accounting for technological dependencies of 

undeveloped systems.

The horizon goal of NASA's human space 

exploration initiative is a landed crew mission to 

the Martian surface. Further, private companies 

are developing their own vehicles and mission 

objectives for cis-lunar activities. While a 

bespoke architecture could be designed to 

accomplish mission objectives, it would be 

advantageous to integrate a portfolio of existing 

and future systems that satisfies overarching 

objectives, minimizes both development cost 

and schedule, reduces risk through flight tested 

hardware, and encourages commercial and 

international involvement. An abundance of 

potential systems exist or are in near term 

developmental stages that could compose an 

optimal architecture. Capitalizing on these 

systems and technologies while balancing the 

complexity of integration will be challenging but 

could be rewarding to stakeholders.

The questions that remain are which systems to 

select, when to develop, produce and operate 

them, and how they interact. Given the large 

number of potential choices for various systems 

and the differences in capabilities, cost, 

schedule and robustness of each, the resulting 

combinatorial problem becomes difficult to 

evaluate in terms of overall architecture cost, 

performance, schedule and robustness. 

Compounding this is the scheduling 

dependencies that exist between systems, the 

cost of Design Development Test and Evaluation

(DDT&E) of new technologies, and the impact of 

an annual budget on system selection. The 
enhanced version of Robust Portfolio 

Optimization demonstrated in this presentation 

and paper offers a potential solution to these 

problems. This optimization forms a framework 

of combining various tools in the form of a 

modified version of NASA’s Advanced Missions 

Cost Model (Cost and Schedule), various domain 

specific estimators, technology sizing tools, and 

existing published data.

29_ Test Facility Request 

System (TFRS) 

Augmentations: Integration 

and Automation of Facilities 

Scheduling and Workforce 

Planning for All of Glenn 

Research Center’s Aero and 

Space Testing Capabilities

Authors: Dennis Bowers, Alan Sikon, Joseph 

Panek, Michael Zernic

Presenters: Dennis Bowers, Alan Sikon, Joseph 

Panek, Michael Zernic

Abstract: The Test Facility Request System was 

originally developed to provide annual test 

services forecasting for the utilization of all of 

GRC’s aero and space test capabilities which are 

grouped into 10 different Capability Asset 

Groups.  Each Capability Asset Group covers a 

distinct test area. (e.g. Aero Acoustics, In Space 

Propulsion and Power, Fuel Cells, Aero Sciences, 

etc.). Currently, test requestors provide basic 

parameters to allow facility managers, in 

conjunction with engineering and technical staff 

managers, to develop high-level schedules and 

estimates of staffing levels. Once all parties 

approve the labor levels and schedule for 

hundreds of tests per year, the data remains 

static for the remainder of the fiscal year.

Since the current system is static in nature, it 

limits the managers’ capability to keep up with 

changing testing requirements. Integrating 

monthly labor estimates into the schedule 

would provide a higher level of fidelity to 

manage staff. This would allow managers to 

forecast future labor utilization across the 

testing facilities and provide data for staff 

leveling. If the staff could be tracked by skill, the 

proper discipline could be assigned when 

required. These capabilities would deliver higher 

efficiency.

Managers had previously created Excel planning 

tools that were designed for their own unique 

purposes. Leveraging on the concepts of those 

tools, we are creating a standardized yet 

tailorable solution that will automate and 

streamline the process of integrating staffing 

and schedule. Tailoring will allow each manager 

to enter the appropriate amount of information 

based on the complexity of the test. Automation 

will reduce labor, improve schedule consistency, 

and eliminate errors in transferring data. A labor 

and schedule dashboard provides a simple way 

to monitor labor and schedule trends. The 

captured data can be utilized to enhance future 

test planning for all of Glenn Research Center’s 

aero and space testing capabilities, grouped into 

10 different Capability Asset Groups (CAGs).

Using standard tools, MS Project, and Excel with 

their capability to run macros, standard 

templates are being created to implement the 

solution. Initial estimates for labor and 

schedules are created in Excel. Macros create a 

standard MS Project file with estimated labor, 

per project subtask. MS Project schedules will be 

updated monthly by the respective facility 

managers, while labor will be updated monthly 
by the engineering and technical managers in 

Excel. Macros will be run to update the labor 

data in MS Project. This information will then be 

collected within an online database. The 

database will source information to the labor 

and schedule dashboard.

This presentation will discuss the current state 

of the project and snapshots of the various 

tools.

30_ Wait, What? How 

Competed Missions Might Be 

Experiencing More Cost 

Growth than Directed 

Missions

Authors: Rachel Sholder

Presenters: Rachel Sholder

Abstract: In a follow-up to last year’s runaway 

hit presentation Math is Hard, in this analysis, 

we dive deeper into one of the most surprising 

findings from that analysis. At face value, a 

competitive selection process is supposed to 

translate into lower cost missions. However, 

Math is Hard showed that is actually not the 

case. In fact, from Math is Hard, we learned that 

the opposite is true and that, when missions are 

selected via a competitive AO process, they 

experience on average 13% more cost growth 

than directed missions. In Wait, What?, we 

examine competed mission cost growth by 

program class (i.e., SMEX, MIDEX, Discovery), 

evaluate potential reasons for cost growth 

variability, and discuss ideas on how to control 

mission level cost growth when a mission is 

selected via a competitive AO process.

31_Math is EZIE (aka Math is 

Hard 2.0): How Contracts 

Help Control Cost

Authors: Rachel Sholder

Presenters: Rachel Sholder

Abstract: The NASA cost estimating community 

relies on risk analyses to estimate confidence in 

a project’s budget. At a NASA mission’s 

preliminary design review (PDR), convention 

requires that baseline cost confidence plus 

project-held reserves should be around the 50th

percentile and cost plus project-held reserves 

and unallocated future expenses (UFE) should 

be around the 70th percentile of the joint 

distribution of total cost and schedule. But how 

can we test whether our approach to 

determining 50th and 70th percentiles for 

missions going into PDR is reliable? An analysis 

of historical costs from past NASA missions 

shows that there is an 84% chance that a 

mission will experience cost growth from PDR to 

Launch. At the empirical 50th and 70th

percentiles, NASA missions are spending their 

full budgets plus 16% and 27%, respectively. But 

does this change when we compare in-house 

spacecraft builds to contracted builds? As more 

commercial hardware options become available, 

NASA missions have relied on various cost 

contracts, particularly cost-plus (CP) contracts 

and firm-fixed-price contracts (FFP). In May 

2022, the Electrojet Zeeman Imaging Explorer 

(EZIE), passed its PDR. EZIE is just one of many 

recent examples of NASA using commercially 

available hardware and FFP contract vehicles to 

control costs. With carefully designed science 

techniques that take advantage of commercial 

off the shelf (COTS) hardware from Blue Canyon 
Technologies, EZIE will be able to collect 

groundbreaking science for a lifecycle cost of 

$57.5M (FY22). To ensure the success of the 

EZIE mission, NASA must allocate and budget 

adequate funding. In this analysis, actual costs 

of NASA missions with contracted spacecraft are 

compared with actual costs of NASA missions 

with in-house builds. This paper will allow us to 

examine the NASA cost community’s approach 

to reserve postures, particularly when contracts 

are employed. Using the empirical dataset as 

our guide, how can projects approaching PDR 

provide cost and schedule analysis that supports 

the goal of achieving 70% confidence in the 

budget at the portfolio level?

32_ Aerospace Viewer of 

NASA Project Staffing Data 

(aView): A Practical Tool for 

Analyzing Staffing Levels and 

Cost Across Missions

Authors: Sarah Lang, Justin McNeill Jr, Tommy 

Tran, Alexander Zarate Garcia, C Jason Zhang

Presenters: Sarah Lang, Justin McNeill Jr, 

Tommy Tran, Alexander Zarate Garcia, C Jason 

Zhang

Abstract: A team from The Aerospace 

Corporation will discuss the capabilities and 

recent updates that have been released in 

aView 2022, the Aerospace Viewer of NASA 

science missions staffing profiles. aView is 

currently in the final test and compliance review 

phases with the One NASA Cost Engineering 

(ONCE) team and will soon be available via the 

ONCE portal to the NASA cost/schedule analyst 

community. Built upon the FTE Tool first 

released in 2011, aView is a database and 

browser-based charting tool for historical 

programmatic data of NASA science missions, 

specifically of Full-time Equivalents (FTE), Work 

Year Equivalents (WYE), and cost. It can be used 

for comparative analysis of staffing profiles of 

science missions. aView provides high-level 

views of the historical data of primarily 

planetary missions for development Phases C 

and D as well as the operations Phase E. When 

reviewing and evaluating the basis of estimate 

for future science mission phases, aView can be 

used to better understand how the labor basis 

of estimate compares with past NASA missions. 

The Aerospace Corporation produces and 

delivers the aView tool for the benefit of its

NASA customer, the NASA Planetary Missions

Program Office at Marshall Space Flight Center

34_ Shaping Workforce with 

Business Intelligence

Authors: Jeff Fajardo

Presenters: Jeff Fajardo

Abstract: NASA is a trailblazer of space 

technologies and has been for decades. 

Evidence of this claim is clear in the evolution of 

the Agency programs, such as Hubble to Webb,

Apollo to Artemis, Curiosity to Perseverance, 

and so forth. NASA’s Project and Business 

management must also stay on the cutting-edge 

by advancing Business Intelligence Technologies 

and developing its workforce. In Information 

Systems, a common business major in academia, 

Moore’s Law is a familiar reference. Moore’s 

Law was an observation by Gordon Moore, the 

co-founder of Intel. Moore observed that the 

number of transistors in a dense integrated 

circuit doubles about every two years. The law is 

often applied to general computational progress 
suggesting that tech will become exponentially 

faster, smaller, and more efficient over time. Are 

NASA’s business technologies growing at an 

acceptable rate? At a rate that stays on pace 

with Moore’s Law? It is within our power to 

identify the steps needed to position the Agency 

on the leading-edge of business practices for our 

missions. Agency leadership has made decisions 

that propel the “Future of Work” such as making 

investments in software and tools, empowering 

the workforce to prioritize data-driven decision 

making. On November 18, 2022, the Office of 

the Chief Information Officer (OCIO) sent an 

agency-wide email stating, “As of Friday, 

November 18, Power BI Pro is now available to 

all NASA end users at no cost!” This was a giant 

leap towards stimulating business innovation. 

Pockets of dashboard builders are scattered 

across the NASA centers. They are recognizing 

the power of the PowerBI software. Like artists 

with canvases, paint, and brushes, these 

originators are creating their own masterpieces. 

We are seeing automation in business processes 

and the creation of advanced analytics that are 

giving new insights to key decision makers. The 

Orion Program’s PP&C office is embracing the 

maturation of the PowerBI technology. 

Ambitious team members are rethinking the 

ways analysts and managers interact with data

in the Program. Monthly and quarterly cost 

performance reporting is shifting to dashboards. 

The level of insights is expanding, the time spent 

producing reports is shrinking, and the skills of 

the team are blossoming. The presentation for 

this abstract showcases recent Orion business 

innovations and lays out a path for the future of 

work at NASA. This presentation was perfected 

in JSC’s Toastmasters club. It has been given to 

audiences in Orion, SLS, and EGS at the Cross 

Program Integrated PP&C Integration Team 

Face-to-Face meeting in January 2023. The 

response has been extremely positive. It turns 

on light bulb ideas for audience members.

36_ Thresher and TIDBIT: 

Tools on Automating 

Schedule Risk Assessments

Authors: Jessica Clarke, Patrick Schneider, 

Kimberly Smith 

Presenters: Jessica Clarke

Abstract: Schedule Risk Assessments can be a 

tedious and time-consuming effort without the 

right tools and context. Developing methods to 

automate the tedious efforts of Analysis 

schedule building and backing into triangular 

distributions will save time and reduce 

frequency of errors. The following tools are 

addressing needs that have been identified for 

use in NASA Artemis-related Schedule Risk 

Assessments.

Thresher: A tool for simplifying a schedule to 

only critical or near-critical tasks

Project schedules are often quite complex, 

running to many thousands of tasks owned by 

dozens of organizations and embodying huge 

amounts of data. This volume of information, 

though valuable for project management, makes 

it difficult and time consuming to easily perform 

analysis of critical and near critical path tasks.

Thresher is a new tool designed to reduce the 

amount of data and thus the complexity and 

time needed to analyze the critical and near 

critical path tasks of a schedule. Thresher 

creates a new MS Project file containing only 

selected tasks in it and preserves all data such as 
unique IDs, start/finish and baseline dates, logic, 

etc. for those selected tasks. This new file, 

containing a small subset of tasks from the 

original file, is simpler to navigate, easier to 

analyze and dramatically cuts down time 

needed to analyze the critical/near critical path 

of a schedule.

TIDBIT: Triangular Inverse Distribution Back In 

Tool

Uncertainty or risk distributions provided by 

organizations are often delivered in one of two 

formats: either a duration uncertainty or an 

array of best case, most likely, and worst-case 

dates. Encountering the latter is nearly 

guaranteed while conducting any schedule risk 

assessment. One current tactic is to back into 

the duration uncertainty using a triangular 

distribution formula, which could take weeks 

compared to receiving a duration uncertainty 

that is ready to be modeled historically with 

Microsoft Excel VBA macro. Once the best case, 

most likely, and worst-case dates are entered in, 

the triangular distribution formula guides checks 

and inputs through a trial-and-error process by 

the user. Checks are desired to resolve to 0 or 

within a ± 5-day range to ensure that an 

accurate distribution has been calculated. 

Achieving a distribution within the acceptable 

range of checks can take weeks depending on 

the number of iterations required for the 

model.

TIDBIT is a tool that runs both the triangular 

distribution formula and error checks 

significantly faster as compared to historical 

methods. This reduces the time required to 

develop an uncertainty distribution from weeks 

to a day. The tool comes in two formats: python 

Jupyter notebook and HTML/JavaScript, both of 

which use the same formula to calculate 

resulting distributions. The HTML/Javascript

version was created so that TIDBIT can be run 

without requiring the user to configure a local 

environment or install software 

dependencies. 

This brief will cover uses for both Thresher and 

TIDBIT in NASA Artemis-related Schedule Risk 

Assessments. 

37_ Breaking Down Data Silos

Authors: Kailey Melton

Presenters: Kailey Melton

Abstract: Siloed schedule data creates barriers 

to information sharing and collaboration. In 

short, siloed data is not healthy data. If schedule 

data isn't easy to find and use in a timely fashion 

(or can't be trusted when it is found) it isn’t 

adding value. Many programs, projects, and 

initiatives are accustomed to working in their 

own worlds with their own lingo, processes, and 

challenges. This culture of separation carries 

over to schedules. This presentation will 

showcase SCaN’s efforts to break down the silo 

mentality.  

38_ Successful EVM 

Implementation Depends on 

Surveillance

Authors: Nick Frazier, Briannah Smith

Presenters: Nick Frazier, Briannah Smith

Abstract: Routinely testing any system

ensures it is working properly. That system can 

be manufacturing line producing parts for

engines, a fire alarm at the Vehicle Assembly 

Building or an EVM system (EVMS). Checking 
the EVMS is necessary to provide performance 

data that is valid, accurate and timely. The key

word in an earned value management system is 

system. It is not just the data produced but is

comprised of the tools, people and process that 

work together to produce EVM data.

NASA began performing EVMS surveillance on 

our Suppliers in October 2019. In August 2021, 

the Agency Surveillance Plan for In-House 

Projects was updated to include surveillance

starting in early formulation. We have been 

gathering findings from quarterly surveillance 

events, captured in 50+ reports, and watching 

trends emerge in the data.

This presentation will cover EVMS surveillance 

and how it’s conducted from early formulation 

through Phase D. It will describe project 

planning and control functions that NASA 

performs well and areas where we struggle to 

implement sound project planning and control 

practices. The impacts of poor project 

management practices on the performance data

will be conveyed. A list of the improvements 

that have been made based on surveillance 

findings will be briefed. The goal is for 

participants to come out with an appreciation of

what an EVMS is, and how the success of the 

EVMS is predicated upon sound cost estimating, 

budgeting, scheduling practices.

39_ The Smart Projects and

Reviews with Transformative

Analytics (SPARTA) Project

Authors: James Price, Sharon Straka, Matthew

Dosberg, Amanda Cutright, Dan Friedrich, Tara 

Dulaney

Presenters: James Price

Abstract: The Smart Projects and Reviews 

with Transformative Analytics (SPARTA) project 

will transform NASA’s work by developing an

automated, customizable system that makes

programmatic data available on-demand to 

multiple levels of stakeholders.  It is developed

to improve communication of project progress 

and status by providing rolled up data while 

enabling drill-down of key programmatic and 

engineering information in a configurable

dashboard format. This will allow project and 

program managers, review board members, 

directorate, center, and headquarters leaders, 

as well as other stakeholders across the Agency

to monitor programmatic metrics, track project 

progress and challenges, assess mission risk,

enable informed decision making, and review 

data throughout the project lifecycle in one 

location.  SPARTA uses Microsoft Power BI to

increase efficiency, transparency, and 

communication targeted at each level of 

management, with easier access to key 

information across the NASA portfolio to better

facilitate those informed decisions. SPARTA

captures data over time to enable trending and 

other time series analysis.

The timely transfer of project data through 

multiple management levels has traditionally 

been a challenge.  Projects prepare their status 

review packages monthly in PowerPoint with 

inputs from their supporting team.  The 

resulting package is then presented or reviewed 

at several levels of management at the center 

before reaching headquarters, where it can 

undergo additional levels of reviews.  By the 

time the data gets to all levels of NASA 

management, it can be more than one to two 

months out of date.  SPARTA will streamline the 

time needed to prepare for these status 

reviews, make the data readily accessible in a 

customizable format, and reduce the review 

cycle time allowing information to be more 

current.  Searchable functions, slicers, and filters

allow project analytics to be compared to data 

from similar projects at the same development 

phase. Meeting projects where they are to have 

minimal impact to the project’s progress is a key 

aspect to SPARTA. Therefore, the SPARTA team 

has developed methods to extract information 

from existing diverse project sources including

PowerPoint, Excel, Project, and other 

applications, tools, and database sources.
40_ Mission Operations Cost 

Estimation Tool (MOCET)

Authors: Marc Hayhurst, Brian Wood, Cindy 

Daniels, Lissa Jordin, Washito Sasamoto, Waldo

Rodriguez

Presenters: Marc Hayhurst

Abstract: The Mission Operations Cost 

Estimation Tool (MOCET) team will present an 

overview of recent model updates and other

research topics. Model updates include changes 

to cost estimating relationships for earth 

science, planetary, and International Space 

Station (ISS) hosted missions. Research topics 

focused on modeling of extended mission costs 

and Level 2 work breakdown structure cost. An 

overview of the state of the user community will 

be presented including statistics from the One 

NASA Cost Engineering (ONCE) model portal and

software.nasa.gov. MOCET is a model 

developed by the Aerospace Corporation in 

partnership with NASA’s Science Office for 

Mission Assessments (SOMA), which provides 

the capability to generate cost estimates for the 

operational, or Phase E, portion of NASA science 

missions. MOCET is comprised of CERs that have 

been derived from historical data for Planetary, 

Earth Science, and Explorer missions. The 

resulting CERs and accompanying 

documentation have been implemented as a 

standalone Excel based tool which is now 

available via the One NASA Cost Engineering 

(ONCE) model portal and software.nasa.gov.

41_ The Mystery of the 

Metrics: Contractor Schedule 

Management

Authors: Erin Wood

Presenters: Erin Wood

Abstract: Many of us can recall the classic 

family game in which someone excitedly 

screams out “Colonel Mustard did it in the 

Conservatory with a revolver!”. The suspense 

and intrigue, the required logic, and the thrill of 

figuring out who the perpetrator of a mysterious 

murder is. Supporting the NASA side of the 

contractor project controls for Space Flight 

projects is not unlike that. I’m not intimately 

involved with the SpaceDoc teams as I would be

when embedded into an in-house project, so I 

must use the schedule and information from the 

533, resource analysts, and project managers 

from both NASA and the contractor teams to 

fully understand not only what happened in the 

preceding month, but also what concerns or 

pitfalls may lay ahead.

Travel with me through the 9 rooms of metrics 

management, each room representing a crucial 

piece of information that must be gathered to 

evaluate progress and the go forward plan. As 

we go, we’ll discover six weapons (tools &

software) to get the answers you need from 

limited data provided and meet the characters 

that represent some patterns that you may 

discover hidden in the numbers in our ultimate 

effort to remove the barriers that may be in the 

way of mission success.

We will put all of the clues the metrics provide 

us together to be able to solve the mystery and 

support project management, program 

management, and our CORs

• where (what information is important)

• with what weapon (what tool did we use to 

get that information)

• and who (telling patterns)
42_ Agile Methodology 

Applied to Systems 

Engineering in the Domain of 

Cost Analysis of a 6U 

CUBESAT

Authors: Ana Carolina Di Iorio Jeronymo,

Lidia Hissae Shibuya Sato, 

Luís Eduardo Vergueiro Loures da Costa, Jonas 

Bianchini Fulindi, Victoria de Souza Rodrigues 

Presenters: Victoria de Souza Rodrigues 

Abstract: This paper presents the use of agile

approach applied to system engineering in a 

cost analysis during the development of a 6U 

CubeSat called Scintillation Prediction Research 

Observations Task - SPORT.

A CubeSat project has characteristics that could

facilitate its development, due to the existing 

standardization and available commercial of the 

shelf products, but this does not imply that less 

effort needs to be done on the system 

engineering process. Usually, it is the other way

around in scientific missions, for instance: the 

effort can be as hard as in a large satellite to find 

a suitable trade-off among what is available, 

what can be developed and what the 

stakeholder needs. Agile methodologies are well 

known and largely used in software projects and 

its efficiency can be seen in the literature. 

Because of that, there is an increasing interest in 

using this approach in projects other than 

software-related ones, especially in complex 

projects.

Agile methodologies in general implies in 

developing a project in small portions known as 

sprints, and in each sprint a value is added to

the product delivered at that time, and it is seen 

if it attends the expectations of the 

stakeholders. The same concept is used in this 

work, where the agile approach guides the cost 

analysis in the activities of planning, developing 

and supporting a CubeSat-based small satellite 

mission.

The system engineering processes are applied in 

sprints being a valued product presented to the 

stakeholders at the end of each sprint.

This work demonstrates how the agile approach

makes use of the sprints and how it is derived to 

the cost analysis of a space project. Focusing on

the cost management and procurement 

guidance, our approach is demonstrated in a 

practical application of the SPORT project, a binational CubeSat-based scientific nanosatellite, 

which platform development is being made by

ITA.

45_ Through the Looking 

Glass- Why EVM Is An 

Essential Risk Mitigation 

Measure for Decision Makers 

and Program Managers at 

NASA

Authors: Symantha Loflin

Presenters: Symantha Loflin

Abstract: The author is providing research that

defends and strengthens the contractual 

addition of Earned Value Management (EVM) as 

a risk mitigation measure. As a key NASA

acquisition priority, EVM system compliance and 

surveillance are functions of the EVM discipline

that provides program managers with the 

capabilities required to execute the 

program/project as effective stewards of the

taxpayers’ money.

It takes a whole-of-government approach to 

defend and protect the world’s dependance on 

the sea, air, and space through conformality 

with legal and regulatory processes of the 

federal procurement of Made in America 

products and the growth of small businesses to 

achieve economic opportunities and national 

security strategy.
Each year, the federal government increases the 

funding of developmental contracts as a 

measure to “Protect Sea, Air, and Space” 

(National Security Strategy, October 2022). 

These efforts aim to protect U.S. interests in 

developing technologies, creating economic 

opportunities, and enabling climate surveillance, 

and to responsibly oversee the space 

environment.

As a risk mitigation measure, the 

implementation and use of EVM through the 

roles of the contracting officer, management, 

and functional specialist is essential to the

fundamental assessment of the program/project

performance. It is imperative that program 

managers and decision makers rely on current, 

accurate, and defensible data obtained from 

EVM compliance through surveillance to make 

informed performance decisions. These 

measures enhance risk mitigation by controlling 

cost, schedule visibility, and technical readiness.

In addition, value is added to the taxpayer, 

federal government, and the national security 

strategy.

47_ A Significant Other: 

Hypothesis Testing on

Qualitative Predictor 

Coefficients in CER 

Regression

Authors: Cassandra Chang

Presenters: Cassandra Chang

Abstract: Cost estimating relationships (CERs) 

are used to predict costs for new space missions 

based on historical data from past missions. 

Many of these CERs are developed using 

regression techniques. Variable selection for 

CER models tends to come from hypothesis 

testing on individual parameter coefficients. 

However, there are many nuances to consider 

when determining if a predictor is significant or 

not, especially with qualitative variables, also 

known as binary, dummy, or logical variables, in 

nonlinear regression. It is important to ensure 

that the null and alternative hypotheses being 

tested reflect the realistic effect that a predictor 

may have on the response variable. This can 

range from testing the fitted coefficient against 

a specified nonzero value to choosing between 

one sided and two sided hypothesis tests. 

Selecting variables that are statistically 

significant in the right contexts can help create 

more realistic CERs and better cost estimates.

48_ Aleatoric and Epistemic

Uncertainty Quantification in

Bayesian Dirichlet Cost Rules

of Thumb

Authors: Melissa Hooke

Presenters: Melissa Hooke

Abstract: The total cost of any project is the 

sum of the costs of its components. At NASA, 

these components are called the Work 

Breakdown Structure (WBS). There are eleven 

numbered elements at the top level of the NASA 

WBS, including 1: Project Management, 2: 

Systems Engineering, 4: Science/Technology, 5: 

Payload(s), and 6: Spacecraft. During the earliest 

phases of the project lifecycle, the costs of only 

one or a few of these components are well 

constrained, and the cost of the other elements 

must be estimated in order to submit a proposal 

with a reasonable chance of success. In the 

absence of time, money, or the level of detail to 

produce grassroots estimates, cost estimators 

look to past missions to gain insight; they have 

traditionally averaged the percentage 
allocations for each WBS breakdown from 

previously flown missions and used them to 

predict allocations for the new project that they 

are trying to cost. At NCSS 2022, we presented a 

novel costing method to project costs for future 

MIDEX class missions; rather than relying on 

average percentages, we proposed a Dirichletdistributed model, which aptly describes the 

uncertainty of resource allocation by capturing 

the correlation between components. Here, we 

expand on our work from last year by (a) 

applying similar methodologies to more mission 

classes other than MIDEX (Flagship, New 

Frontiers, Discovery, SMEX, etc.), (b) imputing 

missing data records using expert knowledge of 

how costs are bookkept, (c) expanding our 

communication to align with up-to-date 

Uncertainty Quantification standards 

(categorizing uncertainty as aleatoric or 

epistemic), and (d) generating an interactive 

web tool easily accessible and usable by systems 

engineers. This presentation will review the 

Dirichlet ROT model and demonstrate the 

capabilities of the online tool.

49_ ASCoT 3: Nonlinear 

Principal Components 

Analysis and Uncertainty 

Quantification in Early 

Concept Spacecraft Flight 

Software Cost Estimation

Authors: Melissa Hooke

Presenters: Melissa Hooke

Abstract: For mission planners and evaluators 

alike, value in cost models comes from a mean 

or median prediction, an understanding of the 

uncertainty on that prediction, and an 

understanding of model performance. Here we 

apply advanced statistical and machine learning 

methods to spacecraft flight software cost, 

effort, and SLOC estimation, and present the 

results in the latest version of the Analogy 

Software Cost Tool (ASCoT). We present in- and 

out-of-sample performance metrics for our 

models, each of which incorporate some 

amount of epistemic uncertainty. ASCoT, hosted 

on the One NASA Cost Engineering (ONCE) 

database via the Online NASA Space Estimation 

Tool (ONSET), was first showcased in 2016 as a 

number of analogy-based models and methods 

(kNN and Clustering) to support early project 

formulation. This ASCoT update improves upon 

the previous analogic methods by incorporating 

uncertainty in the data transformations. In 

particular, we use a Nonlinear Principal 

Components Analysis (NLPCA) to deal with 

ordinal data.

50_ Analogy Cost Estimation

for CubeSats using COMPACT

Authors: Melissa Hooke

Presenters: Melissa Hooke

Abstract: The CubeSat Or Microsat Probabilistic 

and Analogies Cost Tool, or COMPACT, is a NASA 

Headquarters funded effort to fill the gap in cost 

estimating capabilities for CubeSats during early

formulation. The COMPACT team has collected 

technical, programmatic and cost data on 

dozens of flown CubeSats missions led by NASA, 

research labs, and universities. The purpose of

COMPACT’s online cost tool (available through 

ONCE) is to provide transparent access to 

historical data and a framework for analogybased cost estimation for future CubeSat 

missions using verified cost data from historical 

CubeSat missions.

One of the challenges of analogy cost estimation 

is the identification of analogue missions to use 

for the cost estimate. COMPACT’s k-Nearest 
Neighbors (KNN) algorithm provides a ranked 

order list of possible analogy missions. 

COMPACT now also provides mission data 

sheets which can be used to read up on 

historical missions and view additional data

directly within the tool in order to determine 

whether the past mission qualifies as an

appropriate mission analogue. This presentation 

will walk through how to navigate the new 

mission data sheets and show an example of 

how the data sheets may be used for analogy 

comparisons relevant to early formulation 

costing.

52_ Joint Confidence Level 

Analysis for Projects with 

Multiple Objectives

Authors: Michael Trumper, Lev Virine

Presenters: Michael Trumper, Lev Virine

Abstract: Traditional Joint Confidence Level 

(JCL) is an integrated project cost and schedule 

risk analysis. The result of a JCL shows the 

probability that both a project’s cost will be 

equal to or less than the targeted cost that the 

schedule will be equal to or less than the 

targeted finish time. To perform JCL analysis, the 

summary cost-loaded project schedule is 

developed, risk and uncertainties are assigned 

to schedule activities, Monte Carlo simulations 

are performed, and results are shown on a 

scatter plot. The traditional methodology has a 

major limitation: JCL is performed only for 

project cost and schedule. In reality most 

projects have multiple objectives, including 

achieving technical performance, as well as 

quality, safety, security, environmental 

protection, public relations, and many other 

objectives. The impacts of risks and 

uncertainties are not limited to just schedule 

and cost, but can affect other such objectives.

The paper proposes a process of integrating 

multiple objectives to JCL analysis and 

presenting results on different plots. The results 

of analysis will be the probability that cost, 

schedule, technical performance, and others will 

meet certain targets.

The same project risks can affect multiple 

objectives, but with different impacts. For 

example, poor performance of subcontractor 

may increase project duration and cost but can 

also affect technical performance and safety. 

Duration and cost are schedule-related 

objectives because schedule needs to be 

recalculated to determine impacts of a risk on 

the project, while others are non-schedule. 

Different risks will have different impacts on the 

same objective, risks may or may not be 

correlated with each other. Risks impacting nonschedule objectives are assigned to the tasks 

and total impact of such risks is calculated using 

Monte Carlo simulations together with cost and 

schedule uncertainties.

JCL analysis with multiple objectives includes the 

following steps:

1. All non-schedule related objectives 

should be defined and quantified by 

establishing reference points. For 

example, for technical performance 

objective, minor issues may have 0-20% 

impact, moderate issues – 20-40%, etc. 

Organizations often have scales for 

different risks associated with such 

objectives in their risk management 

systems.

2. All objectives including schedule related 

(cost and schedule) and non-schedule 

(e.g., technical performance) must be 

prioritized. This prioritization can be 
achieved using a methodology derived 

from the Analytical Hierarchy Process. 

The process includes pairwise 

comparison of multiple objectives based 

on their quantitative scales. For 

example, risk impact on technical 

performance will be 5 times more 

critical than impact on cost. Essentially 

this is a judgement elicitation process 

from the experts involved in the 

project.

3. Risks are assigned to different tasks and 

resources of project schedule. The risk 

impact on schedule-related and nonschedule objectives are defined. Monte 

Carlo simulation of the project schedule 

is performed. Total impact of risks for 

each non-schedule objective is 

calculated as a sum of impact of 

individual risks.

4. Total probabilities and impacts will be 

normalized for all iterations on Monte 

Carlo simulations. The result analysis is 

scores for each objective. They are the 

indicators for each objective that can 

used to compare with predefined 

targets. For example, 0% score for 

technical performance would mean that 

all risks affecting technical performance 

will be avoided. All risks can be ranked 

based on score of individual objective or 

for all objective together based on 

priority of each objective.

5. Results can be presented on the scatter 

plot. The scatter plots can be 2D, such as 

finish time vs. technical performance or 

3D, such as cost, duration, and technical 

performance. The plot also includes 

frontier lines or lines where the 

combination of objectives meets certain 

predefined targets. In case of 3D chart 

frontier lines would become 3D surface 

where all three objectives (cost, 

duration, and technical performance) 

meet predefined targets.

6. If probability of cost, duration (finish 

time), technical performance and other 

objectives don’t meet predefined 

targets, certain risk mitigation measures 

should be implemented. In order to 

measure their efficiency new JCL 

analysis should be performed.

The proposed methodology has several 

advantages:

• JCL analysis with multiple objectives 

allows to estimate probability that 

targets related to project duration, cost, 

technical performance and other 

objectives will be met.

• Project risk analysis with multiple 

objectives allows to rank project risk 

based on integrated score for all 

objectives.

• JCL analysis can be used to assess 

efficiency of risk mitigation measures 

and how they would affect multiple 

objectives.

• The results of JCL analysis with multiple 

objectives can be presented on 2D and 

3D with frontier lines or frontier 

surfaces, which can be used to depict 

probabilities that objectives will meet 

predefined targets.
53_ Quantified Benefits of 

Earned Value...and Their 

Benefits

Authors: Matt Jones

Presenters: Matt Jones

Abstract: Historically, NASA has lagged behind

the Department of Defense (DoD) in adopting 

Earned Value Management (EVM). It is widely 

agreed that organizational buy-in is a key driver 

of adoption and quality of any given Earned

Value Management System (EVMS). EVM buy-in 

is hindered by the lack of compelling studies

with quantified data confirming the consistent.

Some recent quantitative studies have even 

claimed that EV benefits have decreased since 

1996.

This presentation delves into data from eight

recent NASA projects compiled by the EVMS of 

the Johns Hopkins University Applied Physics 

Laboratory (JHU/APL) as well as data from a

previous DoD study that attempted to quantify 

the benefits of EVM. Quantified EVM benefits 

are confirmed across industries. Differences

between industries, that are likely driven by

varying levels of scope risk, are investigated.

Statistical trends, especially the relationship 

between scope risk and EVM’s predictive power, 

are presented.

Potential future applications of these findings 

are also explored. These include techniques for 

using the findings to gain EVM buy-in and a 

novel methodology for using the quantified EVM

benefits metrics as a low cost alternative to 

labor intensive EVM surveillance processes.

54_ Programmatic Cost Tool

Authors: Joe Mrozinski

Presenters: Joe Mrozinski

Abstract: At the 2018 Symposium, we 

introduced the Programmatic Cost Tool (PCT), 

including inputs and outputs as one would 

expect, but also gave a deep dive into the 

historical lineage of the tool and how it grew out 

of previous efforts at NASA, industry and DoD. A 

few general results from previous studies were 

also reviewed.

At the 2022 Symposium, we took the audience 

along on a test drive as we ran the tool live 

during our talk to show the latest capabilities 

that had been added to the tool suite while 

analyzing a fictious architecture.

At the 2023 Symposium, we will talk about how 

PCT was used to help develop the “Concept for 

2033 Crewed Mars Orbital Mission with Venus 

Flyby,” which was published in 2022 in the

Journal of Spacecraft and Rockets. This complex 

architecture would require 17 launches of 

various systems over the course of 5 years. The 

key question attendees at this Symposium 

would be asking themselves: could all of the 

needed systems be developed and produced 

under a reasonable budget in the relatively 

compact schedule needed in order to take 

advantage of the rare orbital alignment needed 

by this architecture? We will show PCT’s success 

answering this question for this fascinating and 

surprisingly affordable concept for the first 

human flyby of Venus and the first human orbit 

of Mars.

56_ Yes, NASA, a project can

be cancelled

Authors: Justin Hornback

Presenters: Justin Hornback

Abstract: NASA projects and programs in 

development have faced continued challenges 

from COVID, higher inflation, and supply chain

disruptions over the past three years. Outside

of these unique challenges, traditional 

challenges of developing a solid baseline and 
executing to the baseline for robotic space flight 

projects remain.

This presentation will share an anonymous case 

study of a project that was cancelled in late 

Phase C (implementation/integration) after 

breaching the KDP-C cost and schedule 

commitment and going through one rebaseline 

prior to cancellation. The cancelled project was 

selected through a competed Announcement of

Opportunity (Ao). Proposal analysis estimated a

likely inability to delivery to technical 

requirements within schedule or cost.

Assessment of the project as it matured through 

its life cycle continued to show the likelihood of 

the project not meeting cost/schedule estimates 

and the inconsistency of these estimates with 

Agency requirements. Assessments were 

considered at Key Decision Points (KDPs) but 

ultimately did not drive significant changes with

the project until cancellation.

The presentation will also examine how project

reporting was disconnected with program office 

and independent assessments. Many metrics,

including Earned Value Management (EVM),

reporting did not reveal issues without detailed 

examination of project management products 

like the Integrated Master Schedule (IMS) and 

533 reports.

This presentation will also consider impacts of 

external factors like loss of proposed project

partners, COVID, and the project being named in

appropriations with its own operating plan as

additional factors for why the project continued 

given the significant risks identified at every 

stage of a project’s development life.

57_ EVM Analysis Tool for 

Complex Projects with Varied 

Data Sources

Authors: Donald P. Rice, Jr.

Presenters: Donald P. Rice, Jr.

Abstract: The Planetary Missions Program 

Office at NASA’s Marshall Space Flight Center is 

responsible for oversight of Science Mission 

Directorate/Planetary Science Division robotic 

exploration missions. Some of these missions 

are large & complex with work spread over 

multiple contracts. The purpose of this briefing 

is to demonstrate how a simple spreadsheet

tool can pull together the available data to 

provide meaningful analysis of large & complex 

projects, using Europa Clipper as an example of 

the initial setup, the monthly process, and how

to produce a final monthly analysis briefing.

To analyze the whole Europa Clipper project, 

data is collected on eight different contracts.

The data availability varies depending on the 

contract. Complete IPMRs are only available for 

two contracts. IPMR Format 1 data is available 

for six contracts. However, Empower data are 

only available for three of the contracts.

Additionally, there are a few other pieces of 

information delivered monthly, including 

management reserves log for one of the 

contracts, a liens and threats list, and a Control 

Account Manager Notebook. Initially, analysis

was only performed on the largest contract, 

covering about 70% of the total project value.

This left 30% of the project not being analyzed,

and a lot of the significant issues on the Europa

Clipper Project were found in the smaller value 

contracts.

With that in mind, a tool was developed to tie all 

the available data together and provide as much 

analysis as possible for the whole project and for 

the individual contracts. The analysis tool uses a 

set of automated calculations to develop a 

range of traditional EVM metrics (e.g., SPI(t), SPI, 

CPI, and TCPI for a variety of different targets), 

Earned Schedule, EAC estimates, and data 

trends. It also generates a consistent set of 

analysis charts that allows tracking at all Project 

levels. Examples of these charts will be 

demonstrated during the briefing.

Additionally, the monthly updating process will 

be described. Once monthly updates are 

completed, the analyst is ready to start working 

on the monthly EVM Analysis slides using the 

summary level charts and any of the contract 

level charts as needed for detailed analysis. The 

generated charts point the analyst towards the 
specific areas that need to be investigated 

further to support completion of monthly 

reporting to Program Office Management.

It is anticipated that this analysis tool could be 

used by others in the NASA EVM community for 

their complex projects.

58_ Examining the Effects of

Implementing Data-Driven

Uncertainty in Cost

Estimating Models

Authors: Vicky Nilsen

Presenters: Vicky Nilsen

Abstract: When conducting probabilistic cost 

analysis, correlation assumptions are key 

assumptions and often a driver for the total 

output or point estimate of a cost model. 

Although the National Aeronautics and Space 

Administration (NASA) has an entire community 

dedicated to the development of statistical cost 

estimating tools and techniques to manage 

program and project performance, the 

application of accurate and data-driven 

correlation coefficients within these models is 

often overlooked. Due to the uncertain nature 

of correlation between random variables, NASA 

has had difficulty quantifying the relationships 

between spacecraft subsystems with specific, 

data-driven correlation matrices. Previously, the 

NASA cost analysis community has addressed 

this challenge by either selecting a blanket 

correlation value to address uncertainty within 

the model or opting out of using any correlation 

value altogether. One hypothesized method of 

improving NASA cost estimates involves deriving 

subsystem correlation coefficients from the 

residuals of the regression equations for the 

cost estimating relationships (CERs) of various 

spacecraft subsystems and support functions. 

This paper investigates the feasibility of this 

methodology using the CERs from NASA’s 

Project Cost Estimating Capability (PCEC) model. 

The correlation coefficients for each subsystem 

of the NASA Work Breakdown Structure were 

determined by correlating the residuals of 

PCEC’s subsystem CERs. These correlation 

coefficients were then compiled into a 20x20 

correlation matrix and were implemented into 

PCEC as an uncertainty factor influencing the 

model’s pre-existing cost distributions. Once this 

correlation matrix was implemented into the 

cost distributions of PCEC, the Latin Hypercube 

Sampling function of the Microsoft Excel add-in 

Argo was used to simulate PCEC results for 40 

missions within the PCEC database. These steps 

were repeated three additional times using the 

following correlation matrices: (1) a correlation 

matrix assuming the correlation between each 

subsystem is zero, (2) a correlation matrix 

assuming the correlation between each 

subsystem is 1, and (3) a correlation matrix 

using a blanket value of 0.3. The results of these 

simulations showed that the correlation matrix 

derived from the residuals of the subsystem 

CERs significantly reduced bias and error within 

PCEC’s estimating capability. The results also 

indicated that the probability density function 

and cumulative distribution function of each 

mission in the PCEC database were altered 

significantly by the correlation matrices that 

were implemented into the model.

This research produced (1) a standard 

subsystem correlation matrix that has been 

proven to improve estimating accuracy within 

PCEC and (2) a replicable methodology for 

creating this correlation matrix that can be used 

in future cost estimating models. This

information can help the NASA cost analysis 

community understand the effects of applying 

uncertainty within cost models and perform 

sensitivity analyses on project cost estimates. 

This is significant because NASA has been 

frequently critiqued for underestimating project 

costs and this methodology has shown promise 

in improving NASA’s future cost estimates and 

painting a more realistic picture of the total 

possible range of spacecraft development costs.
59_ Bayesian Quantitative 

Risk Analysis

Authors: Christian Smart, Murray Cantor

Presenters: Christian Smart, Murray Cantor

Abstract: This presentation proposes extending 

the traditional practice of using three-point

estimates and Monte Carlo simulations in 

quantitative risk analysis by incorporating 

Bayesian updates using actual observations. This 

approach enables tracking schedule and cost 

risk throughout the life cycle and can raise red 

flags if the uncertainty does not decrease. These 

reports are particularly useful for innovative

projects with a high degree of uncertainty.

The presentation includes a case study 

demonstrating how incorporating actuals can 

reduce uncertainty. It concludes by discussing 

this approach's potential benefits and 

limitations and its implications for project 

management practice.

60_ Distribution Free 

Uncertainty for CERs

Authors: William King, Shaun Irvin

Presenters: William King, Shaun Irvin

Abstract: For this presentation we intend to 

introduce and demonstrate the application of 

conformal prediction as a tool to specify 

prediction intervals (PI) for any machine learning 

algorithm. Unlike the more commonly used 

methods for developing prediction intervals 

(e.g., which assume error normality and utilize a 

first order Taylor series approximation to 

estimate the variance of the functional form), 

conformal prediction intervals offer rigorous 

statistical coverage guarantees without

stringent distributional assumptions (required 

by classical inference), and only assumes the 

exchangeability of data, a weaker assumption 

than independence (which is also required by

classical inference). This means that conformal 

prediction can be applied to a wider range of 

problems. Generating these prediction intervals 

is simple and can be done as part of the k-fold 

cross-validation process. Specifically, we intend 

to demonstrate a conformal prediction

technique known as CV+ (Cross-Validation Plus), 

and its locally weighted variant, introduced in 

the paper Predictive Inference with the 

Jackknife+ by Barber, Candes, Ramdas,

Tibshirani (2021).

61_ONCE Database 

Authors: James Johnson, Eric Plumer, Julie 

McAfee, and Mike Blandford

Presenters: James Johnson

Abstract: The One NASA Cost Engineering 

database (ONCE) provides vital data on NASA 

projects to a community of close to 700 users 

that includes NASA Civil Servants, Contractors, 

FFRDCs/UARCs, and others. The data contained 

in the ONCE database comes from the official 

NASA Cost Analysis Data Requirements (CADRe) 

which includes a variety of important technical 

and programmatic information on projects and 

programs. CADRes were initially manually 

entered into the ONCE database and then were 

moved to an automated process several years 

ago that imported the Excel files into the SQL 

database via SQL Server Integration Services

(SSIS). The ONCE team has matured the CADRe 

import process yet again with an online entry 

capability for the CADRe developers. This allows 
for identification and elimination of any data 

anomalies before the data is entered into the 

ONCE database. Another exciting development 

is the capability for ONCE to host containers on 

Amazon Web Services (AWS).

Containers provide a standard way to package 

application's code, configurations, and 

dependencies into a single object. The 

containers hosted within ONCE share an 

operating system installed on the NASA HQ 

Managed Cloud Environment (MCE) and run as 

resource-isolated processes, ensuring quick, 

reliable, and consistent deployments, regardless 

of environment. This new capability allows the 

NASA PP&C Community to develop their own 

web applications quickly and easily with opensource software. ONCE currently hosts 

Aerospace’s aView model in a container and is 

working with several other developers to host 

their models and tools. This presentation will 

provide an overview of the new capabilities and

information on the future updates to the ONCE 

database.

62_ Diving Deep in the 

Domains, Running with

AzTech

Authors: Crystal Bonds

Presenters: Crystal Bonds

Abstract: This presentation will brief the 

Symposium on the newly identified eight (8) 

project performance domains listed from the

recently published 7

th Edition of Project 

Management PMBOK and will exhibit the use of 

a NASA provided schedule management tool,

RunAzTech (RAZ) and its ability to address 

activities and identify coordination of project 

deliverables recommended in the Planning 

Performance Domain.

1. Guidance and update of 7

th edition PMBOK

a. Project Performance Domains

This presentation will also cover the impression 

of the RAZ tool and its capacity to maintain, 

monitor, and articulate schedule expectation for 

program management teams, enabling effective 

tracing of the critical path, identifying out-ofsequence status, and navigating through large 

schedules on both Microsoft Project Desktop 

and Project Online/Server.

2. NASA management Tool enabling support 

of the PMBOK Planning Performance 

Domain

a. The GASP (Generally Accepted 

Scheduling Principles) of RAZ!

• 8 Doctrines

o 5 Qualities describing valid 

schedules

o 3 Qualities describing effective 

schedules

• Presentation to include a RAZ demo on 

utilizing the GASP’s as a governance 

mechanism for Planning and Scheduling.

64_Risk Management

Authors: Damaris Gonzalez

Presenters: Damaris Gonzalez

Abstract: The main emphasis of this

presentation will be on methods to establish 

effective processes at the outset of a project 

that contribute to effective risk analysis. The 

danger in formalized processes is that they can 

become bureaucratic and burdened with 

procedures, so that the project manager and 

his/her team lose sight of the benefits that come 

from managing their risks. This presentation will 

explain how to do manage risks through the 

definition of risk management and demonstrate

how these processes can be mapped onto the 
stages of the project life cycle, as they relate 

specifically to Project Planning & Control (PP&C) 

competencies, such as cost and schedule. 

This presentation will provide the audience with 

a fundamental understanding of risk 

management and how a consistent and tailored, 

but not overly prescriptive, process can 

contribute to an informed project management 

approach and project success. Balancing risk and 

expectation is one of the most challenging 

aspects of any effort, but it can also offer great 

satisfaction, provided the team is able to 

operate in a climate of understanding and 

openness about their risk posture.

65_ Using Excel to Facilitate 

JSC Analyses

Authors: Barney Roberts

Presenters: Barney Roberts

Abstract: Have you ever been in an SRB meeting 

trying to field questions such as:

• What does our baseline case look like when 

compared with a case where the estimates 

for risk X are twice (or half) the estimate by 

the project?

• What does Case A look like when compared 

to Case B?

• What does our worst case look like when 

compared to our best case?

• What does it look like when all of the 

uncertainties are changed to “x” and 

compared to the baseline case? To case A? 

to Case B?

• What would be required to mitigate risk A, B 

or C, or all together to get the launch date 

back into a reasonable likelihood of 

achieving?

While the JCL tools in the NASA inventory have 

acceptable graphical output capabilities, they 

are quite ponderous for quick response to most 

questions requiring comparisons of cases. 

Responding to such questions over the years has 

led to the development of a Microsoft Excel tool 

and associated input process that allows quick

response to such questions. In addition, Excel is 

very capable and flexible in producing graphical

JCL outputs that are much easier to manipulate 

to show comparisons and relevant data.

Any of the Monte Carlo JCL analysis tools 

provide the data for each individual random 

draw. That data is loaded into the Excel tool 

which produces the cost and schedule 

scatterplot, the Cumulative Distribution 

Function (CDF or S-Curve) for both cost and 

schedule for that case. But, so what, the NASA 

JCL tools also produce these plots. The 

advantage comes when each case is recorded on 

the first tab for tracking, then each case is 

relegated to its own tab. A few of the first tabs 

are set aside for summaries and comparisons. 

This is where the real advantage comes into 

play. Having quick access to all cases allows for

quick response to questions. Then when the 

request for a new case arises, any of the JCL tool

can run a case very quickly, the data file can be 

copied and loaded into a new tab in Excel, then 

all cases are available for comparison. 

Furthermore, the Excel plots can be hyperlinked 

into a Microsoft PowerPoint file for presentation

capability. Having such an Excel-based tool

greatly facilitates the SRB’s assessments and 

reporting capabilities.