National Aeronautics and Space Administration

0

National Aeronautics and Space Administration

FY 2022

ANNUAL 

EVALUATION

PLAN

 NASA Artemis Program and Stennis Space Center 

NASA’s Space Launch System (SLS) rocket prior to its Artemis I flight. The Green Run testing was the first top-to-bottom integrated 

testing of the stage’s systems prior to its maiden flight. Testing conducted on the B-2 Test Stand at Stennis, located near Bay St. 

Louis, Mississippi, and the nation’s largest rocket propulsion test site. Image credit: NASA/SSC 22 January 2020
National Aeronautics and Space Administration

1

Table of Contents 

NASA’s FY 2022 Annual Evaluation Plan .......................................................................................................2

Evaluation Culture.....................................................................................................................................2

Evaluation Standards ............................................................................................................................2

Intended Use and Users........................................................................................................................2

Dissemination and Sharing ...................................................................................................................2

NASA’s Criteria for Defining “Significant” Evaluations .........................................................................3

FY 2022 Significant Evaluations.................................................................................................................4

Technology Investment.........................................................................................................................4

Industrial Base.......................................................................................................................................8

Internship Outcomes ..........................................................................................................................10

Cost and Schedule...............................................................................................................................13
National Aeronautics and Space Administration

2

NASA’s FY 2022 Annual Evaluation Plan

The Foundations for Evidence-Based Policymaking Act of 2018 (Evidence Act) reinforces and supports

Federal evidence-building activities, Open, Public, Electronic, and Necessary Government Data Act, and 

Confidential Information Protection and Statistical Efficiency Act. The Evidence Act requires CFO-Act 

Agencies to publish an Annual Evaluation Plan (AEP) that conveys significant evaluations across the 

Agency each fiscal year, developed in coordination with the Annual Performance Plan, and is published 

in NASA’s Volume of Integrated Performance. The AEP establishes and informs NASA’s key stakeholders 

about planned evaluations. Evaluations will uncover findings that will inform NASA program budgets,

the Strategic Plan and Learning Agenda, annual Strategic Review, ongoing program management and 

development, and connect the performance planning process.

Evaluation Culture

Evaluation Standards 

NASA relies on a culture of evidence-based, data-driven research designs and methodologies to evaluate 

its programs, policies, and organizations across the agency. Evaluations, as defined by the Evidence Act, 

are the use of systematic data collection and analysis to assess effectiveness and efficiency or examine 

interventions of one or more programs, policies, or organizations. The Annual Evaluation Plan details

only those NASA evaluations that meet the Agency’s definition of “significant” evaluations. Led by 

NASA’s Evaluation Officer, in conjunction with the Statistical Officer and Chief Data Officer, five 

standards guide NASA’s evaluation culture: rigor, relevance and utility, independence and objectivity, 

transparency, and ethics. These standards, in addition to the criteria established for “significant” 

evaluations are the foundation that NASA uses to support its array of evaluation activities. 

Intended Use and Users

The AEP identifies planned “significant” evaluations from across the Agency. It serves as a primary 

means to inform Agency senior officials where the most significant evaluations are conducted, cultivate 

data sharing and resources between NASA organizations, and provide information to help support the 

Agency’s evidence-driven culture. In future years, the AEP will build upon an inventory of significant 

evaluations to help project where evaluation resources should be allocated based on recommendations 

from Agency senior officials, the Executive Branch, and Congress. Similarly, the AEP will spur 

collaboration to better leverage data-sharing and evaluation strategies where possible.

Dissemination and Sharing

NASA has long been committed to disseminating and sharing results from its evidence-building activities 

with the greater scientific community and, when permissible, making this information broadly available 

to the public. As detailed in NASA Procedural Requirement (NPR) 2200.2D – Requirements for 

Documentation, Approval and Dissemination of Scientific and Technical Information, the Agency strives 

for the widest practicable and appropriate dissemination of information concerning its activities and 

scientific and technical information. NASA will leverage this framework in sharing findings from its 

“significant” evaluations.

The Agency’s dissemination framework includes an array of symposium presentations, peer reviewed 

journal publications, and NASA internal and external council discussions. Agency evaluations that 

provide promising and effective findings are systematically and broadly disseminated to potential 

beneficiaries and to federal agency partners. Criteria and requirements for the dissemination of 
National Aeronautics and Space Administration

3

symposia lectures and papers, in addition to journal materials beyond the Agency, are detailed in 

Chapter 5 of NPR 2200.2D to ensure proper review of substantive content, technical accuracy, overall 

quality, and value to the larger scientific community. The Evaluation Officer, as well as Mission

Directorate Associate Administrators and Center Directors, have responsibility for the technical, 

scientific, and programmatic accuracy of information released externally from the Agency by their 

respective programs. 

While NASA maintains a free exchange of scientific and technological information among scientists and 

engineers, between NASA staff and the scientific community, and between NASA employees and the 

public, the AEP is a formal dissemination of “significant” evaluations. Table 1, below, depicts broad 

evaluation dissemination methods by stakeholder groups and the formats used to share significant 

evaluations.

Table 1. Dissemination and Sharing Summary

Stakeholder Group Dissemination Channel

NASA Senior Officials Council meetings, Conferences, Reports

Centers and Mission Directorates Leadership Conferences, Webinars, Performance Reviews

Internal Councils and Symposia Reports, Briefings, Conferences 

External Councils, NASA Advisory Council (NAC) Conferences, Webinars

Congress Committee hearings, Briefings 

Office of Management and Budget (OMB) Budget Submission and Reviews, Council Meetings

Public, National Academies Press releases, Webinars, NAC meetings, 

Conferences 

NASA’s Criteria for Defining “Significant” Evaluations

NASA’s extensive evidence culture leverages the findings from evaluations across the Agency. Some of 

these evaluate success of a specific intervention or program and provide targeted findings, while others

involve a broader scope and inform policymaking. NASA determined a list of criteria to distinguish which 

evaluations rise to policymaking significance in accordance with the Evidence Act. These criteria identify 

NASA’s most “significant” evaluations and include those that fit one or more of the criteria illustrated in 

Figure 1. 

Figure 1. Criteria for NASA’s Significant Evaluations
National Aeronautics and Space Administration

4

FY 2022 Significant Evaluations

Technology Investment 

NASA invests in innovative, early-stage technology concepts that could lead to future breakthrough 

capabilities and enable new paradigms or mission types. NASA is evaluating its strategy for investments 

in early-stage innovation that enables potential breakthroughs of tomorrow while also ensuring that 

near-term needs are met through the development and demonstration of more mature technologies. 

Diversity is a key aspect of an early-stage investment and partnership strategy. Enabling technology 

breakthroughs of the future requires NASA to look to diverse sources for ideas and innovation. 

Companies—small and large, academia, research institutions, students, individual inventors and 

hobbyists, NASA researchers and others can all provide rich inputs to a thriving innovation ecosystem.

NASA’s Space Technology Mission Directorate (STMD) early-stage programs have encouraged

participation from underserved and underrepresented communities―including women and womenowned businesses; socially- and economically-disadvantaged individuals, businesses, and research 

institutions; and entrepreneurs living in or whose businesses are located and operate in states with a 

lower number of awards―in technology development activities through outreach activities. 

For example, the Small Business Innovative Research/Small Business Technology Transfer (SBIR/STTR), 

NASA Innovative Advanced Concepts (NIAC), and the Space Technology Research Grants (STRG) 

programs have increased their outreach and communications to underserved and underrepresented 

communities, including Historically Black College and Universities (HBCU) and Minority Serving 

Institutions (MSI). The SBIR/STTR programs have supported 15 Technology Infusion Road tours that have 

attracted over 1,200 participants and have resulted in increased participation in STTR proposals. From 

2010 through 2020, approximately 10% of SBIR/STTR Phase I awards have been to minority-owned 

firms. Additionally, NIAC has seen an increase in female grant awardees from 5.6% in 2013 to 12% in 

recent years. STRG’s largest awards – the four Space Technology Research Institutes (STRIs) – all feature 

MSI participation. One STRI is now led by an MSI university, which recently, post-award, achieved MSI 

status.

Despite these improvements, NASA believes that more MSIs and HBCUs could be engaged and 

contributing to space technology research and innovation. Outreach alone does not sufficiently address 

the underlying factors that create challenges for underserved and underrepresented communities to 

participate and successfully win awards in early-stage space technology innovation. STMD has begun 

piloting new approaches, including a directorate-wide pilot—NASA’s Minority University Research and 

Education Project (MUREP) Space Technology Artemis Research (M-STAR)—to seek to address other 

underlying factors for participation, such as lack of access to NASA experts and lack of understanding of 

how MSI capabilities align with the NASA Mission. In order to assist HBCUs and MSIs through an 

increased understanding of these linkages, M-STAR was established to strengthen and develop the 

research capacity of HBCUs and MSIs in areas of strategic importance to STMD’s technology focus areas. 

In August 2020, M-STAR awarded $604,000 to 15 universities. At the end of the period of performance, 

the HBCUs and MSIs will submit implementation plans to STMD detailing their institution’s plan to 

compete for STMD opportunities as well as identify any obstacles to success. This initiative will deliver 

valuable insight to STMD from our HBCU and MSI stakeholders. M-STAR also aims to connect the 

institutions with STMD experts and familiarize them with STMD’s work. 
National Aeronautics and Space Administration

5

Building on past and current experience, starting in FY 2022, NASA will evaluate other factors that could

help to increase HBCU and MSI contributions toward early-stage space technology innovation. This 

evaluation seeks to understand the factors that are most important in building successful partnerships 

among HBCUs, MSIs, small businesses, and NASA to develop space technology. NASA will leverage these 

findings in program interventions to increase the number of HBCUs and MSIs that are awarded STTRs to 

bring diverse ideas to NASA missions and expand NASA’s economic impact in underrepresented 

communities.

Theory of change 

If NASA examines the core capabilities that exist at HBCUs and MSIs and synergies between those 

capabilities and NASA core competencies, the Agency will develop a more effective platform of 

information-sharing and relationship-building with HBCUs/MSIs, including access to funding and a more 

diverse pipeline of innovative solutions. 

Evaluation question(s)

What are the factors and to what extent does each factor help increase the contribution of HBCUs and 

MSIs towards early stage innovation?

NASA will evaluate this question through a study conducted by the NASA STTR program. This evaluation 

will analyze data in following areas:

• HBCUs/MSIs:

o The capabilities of the nation’s HBCUs and MSIs.

o How does NASA align those capabilities with the core competencies of the Agency in 

order to understand potential synergies?

o What mechanisms facilitate engagement between HBCUs/MSIs, small businesses and 

NASA to encourage commercialization of technologies in partnership with 

underrepresented groups?

o What incentives does NASA utilize to increase participation in these mechanisms?

o What approach(es) result in successful partnerships between HBCUs/MSIs and small 

businesses that win NASA’s STTR awards, contribute to NASA’s missions, and 

commercialize early-stage innovations from diverse sources?

Since the NASA SBIR/STTR has a wealth of success from non-HBCU/MSI research institutes, it would be 

valuable to determine if any of the contributing factors that have led to their success could also be 

factors that could contribute to the success of HBCUs and MSIs. 

Data and information

This evaluation will collect data through an intermediary that works directly with the HBCUs and MSIs as 

well as from customer experience surveys. In addition, existing data from the program’s proposal 

process will be analyzed. The SBIR/STTR program plans to use an agreement with an intermediary in FY 

2021 and conduct the evaluation in FY 2022. 
National Aeronautics and Space Administration

6

Table 2. Data for Technology Investment Evaluation 

Source Purpose

Intermediary data Understand present trends from HBCUs and MSIs

Existing data Collected from past awardees through annual STTR proposal processes that will 

supplement active data gathering to analyze and determine trends

Section 280 

customer 

experience 

clearance

Learn more about what made past awardees successful and what part of their 

customer experience with the STTR program may have discouraged participation 

or success, if any

Methods to be used and evaluation design 

NASA will undertake an experimental design process evaluation to understand strategies to increase 

HBCU and MSI contributions. This will be a mixed-method evaluation. Qualitative and quantitative 

approaches will be used in the implementation study. This evaluation design will be refined through a 

partnership with GSA’s Office of Evaluation Science (OES) in FY 2021.

Table 3. Technology Investment Evaluation Design 

Inputs Capture quantitative and qualitative data for research institution capabilities through an 

intermediary

• Quantitative data example: number of HBCUs/MSIs that submit capabilities 

and/or number that have technology areas where there is a great deal of 

synergy, some synergy, or little to no synergy with NASA core competency areas

• Qualitative data example: Awareness of the existence of the capabilities 

submission database and understanding its benefits

Capture and analyze STTR past participation data

Consider insights from NASA’s MUREP

Processes Analyze what synergies can be made to inform which mechanism might best support 

engagement and ways to experiment with incentives that encourage engagement

Survey successful STTR awardees through our Section 280 customer experience 

clearance to see which parts of their experience with the program most contributed to 

successful partnering between research institutions and small businesses

Expected 

outputs

Identification of the factors within NASA’s span of control (outreach, networking, 

funding, technical and business assistance, etc.) that are most relevant when considering 

partnerships between HBCUs/MSIs and small businesses and any negative impacts

Inform and update the policy to increase the number of HBCUs and MSIs that are 

awarded STTRs to bring diverse ideas to NASA’s missions and expand NASA’s economic 

impact in underrepresented communities

Challenges

Since the HBCU/MSI community is small, getting enough participants will rely on active marketing 

through a well-connected community intermediary. The ability to get qualitative data about successful 

STTR awardee partnership characteristics will be limited by the size of the pool and their willingness to 

provide information. NASA seeks to leverage an intermediary with long standing relationships to the 

HBCU/MSI community to effectively communicate positive impacts and provide study data.
National Aeronautics and Space Administration

7

Dissemination strategies

The results will be shared broadly within NASA to all programs concerned with increasing the 

participation of underserved and underrepresented communities, with the caveat that this evaluation 

will focus on the intersection of successful small business and research institution partnerships—a core 

part of lab-to-market efforts. These results could inform experiments and programmatic strategies to be 

considered by other programs that engage the small business, entrepreneur, or research community 

within other early stage programs, the Office of STEM Engagement (OSTEM), and the Office of Small 

Business Programs. Evaluation results would benefit other agencies’ SBIR/STTR programs and the Small 

Business Administration on ways to increase participation. 

Timeframe

The evaluation design will be finalized in FY 2021, and active partnering and evaluation will occur in FY 

2022. 
National Aeronautics and Space Administration

8

Industrial Base

NASA and the Department of Commerce’s Bureau of Industry and Security (BIS), National Oceanic and 

Atmospheric Administration (NOAA), National Environmental Satellite, Data, and Information Service, 

will conduct an evaluation of the factors that drive efficiency in the NASA and United States civil space 

supply chain network. 

This multi-year collaboration will build upon a substantial record of space sector analysis conducted 

between BIS, NASA and the broader U.S. Government to identify traits that benefit or impede the 

current health and competitiveness of the civil segment of the U.S. Space Industrial Base. Early findings 

will also inform the planning and execution of the civil space provisions of the 2020 National Space 

Policy. NASA and NOAA seek visibility into the current and prospective performance of the civil space 

community to achieve enhanced situational awareness in times of uncertainty. By understanding more 

of the industrial base ecosystem that serves the civil space sector, NASA will be able to better forecast 

pressures, anticipate risks, and mitigate against forces that affect the aerospace supply chain to drive 

adaptability and enhance decision making. For NASA, the evidence generated by this survey will be used 

to inform Agency and program acquisition strategy. 

Theory of change 

If survey results generate valuable information regarding the current state of civil space sector (in 

particular, concerning sectoral trends and gaps/weaknesses or other deficiencies in the supply chain), 

this may inform and lead to the modification of Agency acquisition strategy, ultimately improving the 

efficiency, governance, and management of the Agency and its missions. 

Evaluation question(s)

What are the underlying factors and to what extent do these factors affect the aerospace industrial base 

to support federal civil space acquisition and development?

NASA will evaluate this question through a survey and analysis that looks at the supply chain and 

underlying factors related to the aerospace industrial base. This evaluation seeks to address impacts to 

the aerospace supply chain through the following factors:

• Impacts from the COVID-19 pandemic;

• Corporate financial distress;

• Mergers and acquisitions;

• Costs related to foreign sourcing, offshoring of critical supply chain components, and 

alternatively, reshoring;

• Disruptive new technologies, including advances in artificial intelligence, quantum computing, 

and additive manufacturing;

• Changing workforce dynamics including emphasis on STEM practices;

• Cybersecurity investments and cybercrime impacts; and 

• Constraints related to U.S. federal acquisition reforms.

Data and information

Data gathered as part of this evaluation will be sourced from survey responses comprising of prime 

contractors and their affiliated vendors, universities, laboratories, grant recipients, investment banks, 

non-profits, select federal facilities, and both direct and indirect suppliers to NASA, NOAA, and the larger 

U.S. civil space community.
National Aeronautics and Space Administration

9

BIS will work closely with NASA, NOAA and other space sector affiliates, including associations, to 

minimize the burden on industry while ensuring the robust participation of suppliers currently 

supporting civil space-related applications, such as human spaceflight vehicles, robotic spacecraft, 

spacecraft buses, small satellite components, and microelectronics. 

Methods to be used and evaluation design 

NASA will undertake a non-experimental formative evaluation design to analyze survey results, including 

a time series analysis of responses over the course of three observation years:

• FY 2021: Survey development, testing, distribution, and organization of mailing list and portal 

hosted by the Census Bureau; 

• FY 2022: Survey analysis of roughly 500 completed survey responses in Wave 1 compiled, 

analyzed, briefed, and shared; and

• FY 2023: Survey analysis and comparison of roughly 500 to 1,000 completed survey responses in 

Wave 2 compiled, analyzed, briefed, and shared.

Further evaluation design details are under discussion, with several options, techniques, and tools under 

consideration. Final determination is expected to be made by the end of the calendar year 2021. 

Challenges

Anticipated challenges include willingness of survey respondents to provide detailed, current, and 

historical information regarding performance, financials and supply chain issues. 

Dissemination strategies 

A final dissemination plan has not yet been approved. Initial considerations suggest a wide 

dissemination within NASA to all impacted programs, the broader U.S. civil space community, including 

primary space agencies and associated partners. In addition, a summary of findings is likely to be 

disseminated to relevant stakeholders including the Administration, Congress, industry, academia and 

the public. 

Timeframe 

The timeframe for this evaluation is FY 2021 through FY 2023.
National Aeronautics and Space Administration

10

Internship Outcomes

NASA internships and fellowships leverage NASA’s unique missions and programs to enhance and 

increase the capability, diversity and size of the nation’s future science, technology, engineering and 

mathematics (STEM) workforce. Research demonstrates that internships and work-based learning 

experiences are positively associated with student outcomes such as STEM concept knowledge and 

STEM persistence.

1 Thus, participation in such experiences is an important evidence-based practice to 

addressing current STEM workforce needs. Although there is extant literature documenting the 

outcomes of such experiences on students, there is much less research documenting the contributions 

of such experiences to the STEM field. This study will generate evidence that can be used to assess 

whether NASA internships are associated with positive student experiences and outcomes and 

ultimately inform programmatic improvement. Additionally, this study will assess the extent to which 

NASA internships are associated with benefits to NASA.

The purpose of the Internship Outcome Assessment evaluation is to measure students’ immediate 

outcomes of participating in a NASA internship and assess how and to what extent interns are 

contributing to NASA’s missions. Additionally, this evaluation will identify sources of group differences 

and address how NASA can continue to broaden participation of students from historically 

underrepresented groups in STEM fields. The findings of this evaluation are intended to be used for 

programmatic improvement and to assess the feasibility of instituting a yearly evaluation cycle. By 

comparing and analyzing survey responses and contributions by NASA interns, the Agency will be better 

able to determine differences based on underrepresented group status and isolate avenues for growth 

amongst this population. 

Theory of change 

If higher education students participate in NASA STEM experiences (e.g., internships, competitions, 

challenges, and activities), NASA will benefit from their contribution to missions and a diverse group of 

students will be better equipped to persist in STEM academic and career pursuits.

Evaluation question(s)

What common experiential factors exhibited across NASA interns can be diagnosed to broaden the 

participation of students from historically underrepresented groups in STEM fields in future years?

NASA will evaluate this question through a survey and skills assessment conducted before and after the 

internship program to measure immediate outcomes across participants. This evaluation will address 

the following questions:

• To what extent is participation in NASA internships associated with:

o Intern satisfaction with the program?

o Interns’ self-reported gains in science and research-related outcomes?

o Interns’ likelihood to pursue future STEM-related activities, education, and careers?

• Are there any differences in interns’ reports based upon underrepresented group status?

• What insight do mentors provide on the intern program?

• How do mentors characterize interns’ contributions to NASA’s missions?

 1 National Academies of Sciences, Engineering, and Medicine, 2017
National Aeronautics and Space Administration

11

• Do interns demonstrate growth toward mastery of 21st Century Skills across the duration of the 

internship as assessed by their mentor?

Data and information

This evaluation will utilize two modes of data collection: 1) participant surveys, or questionnaires (for 

interns and mentors) and 2) a 21st Century Skills Assessment (administered by the mentor). The intern 

and mentor surveys have been adapted from the Department of Defense Army Educational Outreach 

Program with permission from program leadership. NASA modified the survey to fit the contextual 

circumstances, vision, goals, and objectives of NASA’s internship program. The questionnaire addresses 

the following areas of interest:

• Satisfaction with the internship program resources and features;

• Experiences with effective teaching and mentoring practices;

• Gains in science and research-related outcomes; and

• Likelihood to pursue future STEM-related activities, education, and careers 

The 21st Century Skills Assessment is an objective assessment measure that is completed by each 

interns’ mentor regarding their progress toward mastery of important 21st Century skills.

2 Mentors 

assess each intern in a pre/post manner. The first assessment will be completed in the first days of the 

program (pre). The second assessment will be completed at the end of the program (post). The 

assessment is used to determine the growth toward mastery for each participant during their time in 

the internship. Mentors rate each participants’ skills in six domains of 21st Century skills:

• Creativity and Innovation;

• Critical Thinking and Problem Solving;

• Communication, Collaboration, Social, and Cross-Cultural Skills;

• Information, Media, and Technological Literacy;

• Flexibility, Adaptability, Initiative, and Self-Direction; and

• Productivity, Accountability, Leadership, and Responsibility

Methods to be used and evaluation design 

NASA will undertake a non-experimental outcome evaluation to measure students’ immediate 

outcomes of participating in a NASA internship and assess how sources of group differences can be 

interpreted to broaden participation of students from historically underrepresented groups in STEM 

fields. This evaluation will use both quantitative and qualitative methods to analyze survey data. 

Quantitative data will be summarized using descriptive statistics such as numbers of respondents, 

frequencies and proportions of responses, average response when responses categories are assigned to 

a Likert scale (e.g., 1 = “Never used” to 4 = “Used every day”), and standard deviations. Emergent coding 

will be used for the qualitative data to identify the most common themes in responses.

Inferential statistics will be used to identify sources of group differences and in support of NASA’s goal 

to broaden participation of historically underrepresented groups in STEM fields. Statistical significance 

will be determined with t-tests, chi-square tests, or various non-parametric tests as appropriate, with 

significance defined at p < 0.05. Because statistical significance is sensitive to the number of 

 2 Sondergeld, & Johnson, 2016
National Aeronautics and Space Administration

12

respondents, it is more difficult to detect significant changes with small numbers of respondents. 

Therefore, practical significance, also known as effect size, will be reported when differences are 

statistically significant. 

21st Century Skills Assessment data will be analyzed using 2-Between, 2-Within Repeated-Measures 

analysis of variance to examine potential differences from pre- to post-observation by underrepresented 

group status and setting. Descriptive and inferential statistics will be included.

Challenges

NASA strives to execute a utilization-focused evaluation that provides data that is useful for continual 

programmatic improvement and evidence-based decision-making. Thus, NASA anticipates challenges 

related to COVID-19 and the practical trade-offs that must be considered in less than ideal contexts. 

Accurate and timely survey responses will be closely monitored throughout the evaluation. 

To ensure a high-quality evaluation is conducted and mitigate against risk, NASA has used the program 

evaluation standards for evaluator credibility to assess the evaluator’s potential bias. To address 

evaluator credibility, Paragon TEC, Inc, the independent contractor conducting the evaluation, has 

selected highly qualified, experienced program evaluators to design and conduct this study. The 

individuals selected to design and conduct this evaluation are independent from the policies, decisionmaking, operations, and implementation of the activities which are the subject of this investigation.

Dissemination strategies 

The findings of this evaluation will be summarized in an evaluation report. The report will be shared 

broadly among the NASA STEM Engagement community. Primary users and stakeholders will attend at 

the conclusion of the evaluation. The findings will also be presented at the Office of STEM Engagement 

(OSTEM) Performance and Evaluation (P&E) Community of Practice, STEM Engagement Council meeting, 

and to the NASA Advisory Council STEM Engagement Sub-Committee. The OSTEM P&E Team will work 

with OSTEM leadership and internship project management to finalize a dissemination strategy and 

identify appropriate audiences. 

Timeframe 

This evaluation will take place across calendar year 2021 (FY 2021 and FY 2022). NASA will work with 

OSTEM leadership, internship project management and internship stakeholders to approve the 

timeframe and identify appropriate cohorts of interns to participate in this evaluation (e.g., spring, 

summer and/or fall).
National Aeronautics and Space Administration

13

Cost and Schedule 

NASA is on the cutting edge of scientific discovery and commands space exploration programs that 

require large funding commitments spanning years or decades. A significant management challenge for 

NASA is controlling cost and schedule given the size and technical complexity of these projects. Projects 

can be either directed or competed, and by evaluating the factors that lead to better performance in 

either of these two acquisition methods, NASA can inform policy to better manage programmatic 

performance.

In order to evaluate programmatic performance between directed and competed approaches across 

NASA’s space flight acquisition planning process, the Agency will employ an outcome evaluation to 

compare the two mission acquisition types. By uncovering favorable attributes associated with better 

performance of each mission type in different scenarios, evaluators will seek to uncover factors that can 

be used to improve cost and schedule models and help NASA better deliver on its commitments. 

Theory of change 

If NASA examines the outcomes and drivers for cost and schedule performance of directed and 

competed missions then the results of that examination can better inform the Agency on a) determining 

which missions should be directed or competed; and b) understanding the programmatic risk posture 

for each mission based on performance drivers. 

Evaluation question(s)

To what extent is there a programmatic performance difference between directed and competed space 

flight projects and what factors or criteria are most attributable to that performance difference?

As described in NASA policy NPR 7120.5E, NASA initiates space flight projects in one of two ways: 1) the 

project is assigned to a NASA Center, either directly or by the Mission Directorate or 2) the assignment is 

made through a competitive process such as an Announcement of Opportunity (AO).

• (1) A “directed” mission is generated in a top-down process from the Agency strategic goals and 

through the strategic acquisition planning process. It is defined and directed by the Agency, 

assigned to a Center or implementing organization.

• (2) A “competed” mission is open to a larger community for conceptualization and definition 

through a Request for Proposal (RFP) or competitive selection process, such as an 

Announcement of Opportunity (AO), before entering the conventional life-cycle process.

This evaluation will examine the performance of directed and competed missions to determine how 

programmatic performance compares between the two types of projects and what, if any, are the 

factors or criteria that are most attributable to that performance difference. Findings from this 

evaluation will help inform the Agency on when it’s appropriate to utilize a directed versus competed 

mission approach and what factors or criteria should be included to increase the likelihood of improved

programmatic performance. 

Data and information

Data will include cost and schedule projections and actuals from various points in project lifecycles. 

Most data will be collected via the Once NASA Cost Engineering (ONCE) database, including technical 

and mission classification data. ONCE is composed of historical NASA Cost Analysis Data Requirements 

(CADRe) that have programmatic data by milestone. Additionally, a project’s Key Decision Point Decision 
National Aeronautics and Space Administration

14

Memos and OMB Quarterly reporting data will be utilized. Other NASA systems will be utilized for data 

that may be missing from the sources listed above. Data prior to 2001 predated CADRe and is harder to 

locate.

Methods to be used and evaluation design 

NASA will employ a non-experimental outcome evaluation to compare the two mission types. The 

evaluation itself will have no direct impact on mission type designation (directed vs. competed) and thus 

will not assign randomization to mission type. Statistical analyses will be conducted to determine if cost 

and schedule performance is statistically different between the two types of missions. Additionally, the 

evaluation will attempt to identify and control for variables that could drive performance so the design 

can better isolate effects of mission type. 

NASA began initial data gathering and research in FY 2020. Initial analysis examined raw cost and 

schedule performance between the two types of missions and determined performance differences by 

utilizing a two-sample, unequal variance t-test to determine a p-value between the two mission types. 

NASA will next expand the data to a broader mission set, including a rerun of t-tests in the data set, and 

identify and isolate other variables to determine each impact. Currently, evaluators have identified the 

following variables as having potential impacts on performance: mission category, mission risk 

classification, mission complexity, technical descopes, and technical drivers.

Due to the limitation of CADRe data, mission datasets will include missions that have launched from 

2001 to present. Current analysis includes only Science Mission Directorate missions, but data from 

other Mission Directorates will be added to data collection and analysis once available. 

Challenges

Several challenges present themselves in this evaluation. The mission types explained above have

historically large variations in mission performance. This indicates that there are several factors that 

drive cost and schedule performance. Identifying and isolating those factors will be a challenge. 

Additionally, the dataset being analyzed is heterogeneous in nature presenting issues with degrees of 

freedom.

Missions span several Agency policy iterations. Over the last 15 years, NASA has conducted several 

programmatic policy updates that could affect results. Additionally, it is safe to assume that NASA’s 

process for competed missions has improved, or at least has varied on performance, within this 

timeframe. This temporal variable will be difficult to control for or could make for inconclusive results. 

Dissemination strategies 

Results from this evaluation will be communicated to NASA leadership and through one of the Agency’s 

management forums, the Agency Program Management Council.

Timeframe 

Initial data collection and raw cost and schedule performance evaluations occurred in FY 2020. The 

evaluation of the variables of performance will occur from FY 2021 through FY 2022.