stem.nasa.gov

NASA STEM ENGAGEMENT LEARNING AGENDA

REPORT BRIEF (DECEMBER 2022)

CALL TO ACTION: THE FOUNDATIONS 

FOR EVIDENCE-BASED POLICYMAKING ACT OF 2018 

The Foundations for Evidence-Based Policymaking Act of 2018 (Public Law No. 115-435) (Evidence 

Act) calls upon all Chief Financial Officer (CFO) Act agencies to build and use evidence for decision 

making. Specifically, the Evidence Act requires each CFO agency to concretize and operationalize “a 

systematic plan for identifying and addressing policy questions relevant to the programs, policies, and 

regulations of the agency.” In support of the Evidence Act and related guidance from the Office of 

Management and Budget (OMB Circular No. A-11, OMB M-19-23, OMB M-20-12, and OMB M-21-27), 

this report brief presents the NASA Office of STEM Engagement (OSTEM) Learning Agenda for FY 2019–

FY 2021, which aligns with and serves as amplification to the Agency’s strategic plan and learning 

agenda. 

ENACTING A CULTURE OF EVIDENCE 

In alignment with federal government evidence-based policy initiatives, Section 203 (a) (3) of the 

Space Act, and NASA’s goals and priorities for STEM engagement (NASA Strategic Plan), NASA’s 

OSTEM undertook a rigorous iterative development process to establish the first OSTEM Learning 

Agenda. The FY 2019-FY 2021 Learning Agenda serves as the foundational document for building a 

culture of learning and continual improvement within NASA’s OSTEM. The implementation of the 

Learning Agenda provides a systematic approach for building and using new knowledge about project 

and operational performance for evidence-based decision making and continual improvement. 

DEVELOPMENT OF THE OSTEM LEARNING AGENDA 

The FY 2019–FY 2021 Learning Agenda development process was guided by a theory of change, which 

posits: If strategic Learning Questions are developed, Learning Activities are conducted to answer the 

Learning Questions, and the evidence gathered from the Learning Activities is used to create Learning 

Products, then the Office of STEM Engagement will be better equipped with evidence to make 

budgetary, programmatic, and operational decisions. The intended long-term change (outcome) of 

implementing this Learning Agenda is to develop a culture of learning and continual improvement 

within NASA’s OSTEM. 

Figure 1. NASA’s Office of STEM Engagement Learning Agenda Theory of Change.
stem.nasa.gov

There was extensive engagement and collaboration with internal and external stakeholders in the 

execution of the FY 2019–FY 2021 Learning Agenda. These stakeholders were provided an overview of 

the Learning Agenda and invited to multiple meetings where there were opportunities to shape the 

direction of Learning Questions, provide input into the development of the evidence-based decisionmaking process, and give guidance on evaluation of progress toward meeting Learning Agenda goals 

and objectives. The primary source of engagement was Expert Review Panels (ERP), comprised of 

nationally recognized experts, where participants were engaged at strategic decision points to provide 

recommendations and feedback on Learning Questions, Learning Activities, Learning Products, and the 

use of Learning Products for evidence-based decisions. NASA’s OSTEM Learning Agenda Collaborative 

Development and Execution Process was followed in the execution of the FY 2019–FY 2021 Learning 

Agenda. 

The NASA OSTEM FY 2019–FY 2021 Learning Agenda Version 1 (FY 2019–FY 2020) was created between 

October 2017–August 2018 using the following three development steps (see Figure 2). 

Figure 2. NASA’s Office of STEM Engagement Learning Agenda Version 1 (FY 2019–FY 2020) Development Process.

Following the development of the NASA OSTEM FY 2019–FY 2020 Learning Agenda, it was implemented 

using the three execution steps illustrated in Figure 3. 

Figure 3. NASA OSTEM Learning Agenda Execution Process.
stem.nasa.gov

LEARNING AGENDA QUESTIONS & REFINEMENT 

The FY 2019-FY 2020 Learning Agenda development process resulted in the creation and prioritization 

of three primary Learning Questions. During the execution of the FY 2019-FY 2020 Learning Agenda, 

additional evidence and feedback from stakeholders informed the revision of one Learning Agenda 

Question and the addition of a fourth Learning Agenda Question. In FY 2021, Learning Agenda Question 

3 was revised to align with the Agency focus on broadening participation of students from groups 

historically underrepresented and underserved in NASA STEM Engagement activities, as outlined in the 

updated strategy for NASA STEM Engagement, and a new Learning Agenda Question was added. The 

fourth Learning Question was added in response to stakeholder feedback, which illuminated the need 

for a targeted focus on NASA’s STEM Engagements’ K-12 programming and activities – resulting in a 

more inclusive and holistic approach to examining the NASA STEM Engagement portfolio. 

NASA’s OSTEM’s commitment to evidence-building practices resulted in the following Learning 

Questions aligned to NASA’s priorities for STEM Engagement and focus areas where evidence is needed 

to inform strategy and programmatic decisions. 

ALIGNMENT OF LEARNING AGENDA QUESTIONS WITH NASA OSTEM FOCUS AREAS 

FOCUS AREAS LEARNING AGENDA QUESTIONS 

CONTRIBUTIONS TO NASA’S MISSIONS & 

WORK 

Learning Agenda Question 1: To what extent are 

NASA’s STEM engagement investments contributing 

to NASA’s missions and work?

DIVERSITY OF THE FUTURE STEM 

WORKFORCE 

Learning Agenda Question 2*: How have NASA 

STEM Engagement investments broadened 

participation of historically underrepresented and 

underserved groups in STEM fields in NASA STEM 

Engagement activities?

PERFORMANCE ASSESSMENT & EVALUATION Learning Agenda Question 3: To what extent have 

enhancements to STEM engagement performance 

assessment and evaluation been implemented?

UNDERSTANDING K-12 STEM ENGAGEMENT 

INVESTMENTS 

Learning Agenda Question 4: What are effective 

strategies to support and measure NASA STEM 

Engagement K-12 investments’ ability to spark 

student interest in STEM?

*Original (Version 1) Learning Question 2: To what extent are NASA’s STEM engagement investments contributing to the 

diversity of the future aerospace industry’s STEM workforce?

LEARNING ACTIVITIES 

Once the Learning Questions were established, appropriate Learning Activities, including strategies, 

tasks, methods, and mechanisms were developed to answer the Learning Questions. Learning 

Activities were scaled to the stage of project execution and the level of investment (see Figure 4). 
stem.nasa.gov

Figure 4. Tiered Approach to Performance Assessment and Evaluation.

EVIDENCE-BASED DECISION-MAKING PROCESS – ENHANCING EVIDENCE USE 

To facilitate the use of Learning Products, including each Learning Question’s portfolio of evidence to 

inform budgetary, policy, programmatic, and operational decisions, NASA’s OSTEM created an

evidence-based decision-making process, which was piloted in the first year of the Learning Agenda 

implementation. The process was created after reviewing government and agency policies and priorities 

and evidence-based decision-making documents of other federal agencies and private organizations.

The evidence-based decision-making process was further refined in FY 2020 to include additional steps 

to the continual learning cycle to better facilitate use of learning products. The refined and current 

version and corresponding clarifying descriptions are presented in Figure 5 and Table 1, respectively. 

Figure 5. Evidence-Based Decision-Making Process.

 
stem.nasa.gov

Table 1. Evidence-Based Decision-Making Process Steps 

Action Explanation 

1. Identify "Decisions" To Be Made Relevant stakeholders gather to identify the decisions to be 

made. Identify how decisions align with overall strategic 

goals.

2. Determine Learning Questions Determine a strategy for how to make the decisions; plan 

should include Learning Questions. 

3. Identify and Execute Learning Activities Determine appropriate internal and external Learning 

Activities. Execute Learning Activities. 

4a. Examine Internal Evidence (Note: Examination of internal and external evidence may 

occur concurrently, sequentially, or in reverse order.)

Conduct formative and summative studies to examine 

internal evidence using qualitative and quantitative data.

4b. Examine External Evidence Gather and examine relevant external evidence using data 

collected from activity and program evaluations. The value of 

the external evidence will be highly variable; therefore, it 

should be critically appraised. 

Critically appraise external evidence. Consider two 

underlying facets of the external evidence:

a. Strength of Evidence

‒ How rigorously has the program, practice, or 

policy been evaluated?

‒ How strong is the evidence in determining that the 

program or policy is producing the desired 

outcomes?

‒ How much evidence exists to determine that 

something other than this program or policy is 

responsible for producing the desired outcomes?

b. Effectiveness

‒ Is this program, practice, or policy producing 

desired outcomes?

‒ Is it producing non-desirable outcomes?

5. Answer Learning Questions Using Internal 

and External Evidence 

The two sources of information (internal and external) may 

be supportive, non-supportive, or conflicting. How the 

decision is made when non-supportive or conflicting will 

depend on multiple factors. 

Options include comparing the internal evidence to the 

external evidence. Alternatively, it may be determined that 

the external evidence is not sufficiently convincing, and the 

internal evidence may be given more weight. The conflict 

between the internal and external evidence may be 

discussed with stakeholders and/or an expert review panel in 

a manner that enables the stakeholders to weigh in on the 

decision-making process.

6. Use Evidence To Make Decisions Upon completion of Step 5, a decision will be made. Evaluate 

extent to which decision aligns with objectives and rationale 

defined in Step 1. 

7. Reflect On Process Once the decision has been made, the process should be 

considered and critically evaluated to identify opportunities 

for improvement.
stem.nasa.gov

EVIDENCE FROM LEARNING ACTIVITIES 

There were several evidence-building activities that were associated with each of the Learning Agenda 

Questions. These were characterized as Learning Activities and findings from each provided a rich 

evidence base for evaluating progress toward NASA’s annual performance goals and targets. A 

summary of the key evidence-building activities associated with the four Learning Agenda Questions 

and their respective sub-questions are presented below. 

LEARNING AGENDA QUESTION 1: TO WHAT EXTENT ARE NASA’S STEM ENGAGEMENT 

INVESTMENTS CONTRIBUTING TO NASA’S MISSIONS AND WORK?

a. What characteristics define students’ contributions to 

NASA’s aeronautics, space, and science missions and 

work in exploration and discovery? 

b. To what extent are NASA student experiences 

aligned with evidence-based, high-quality student 

engagement? 

c. How can NASA develop data collection, analysis, and 

reporting mechanisms to better assess NASA STEM 

engagement activities’ contributions to NASA 

missions and work?

DEVELOPMENTAL HYPOTHESIS: If student contributions to 

NASA’s missions and work are characterized, then current 

levels of contribution can be assessed, and performance assessment and evaluation strategies can be 

more clearly defined. As a result, the Office of STEM Engagement will be better equipped to prioritize 

STEM engagement investments that provide opportunities for students to contribute to NASA’s 

missions and work and share effective practices across the portfolio.

STEM ENGAGEMENT FOCUS: Create unique opportunities for students and the public to contribute to 

NASA’s work in exploration and discovery.

KEY LEARNING ACTIVITIES: 
stem.nasa.gov

KEY IMPACTS: Three key Learning Activities mapped to Learning Agenda Question 1 across FY19, FY20,

and FY21: 1) FY19 Higher Education Challenges, Competitions, and Internships Multiple Case Study; 2) 

FY20 Diversity Deep Dive; and 3) FY21 Intern Outcome Assessment. Each Learning Activity generated 

outcomes, which served as new knowledge and foundational information for additional evidencebuilding activities that enhanced the growing evidence base the NASA OSTEM continues to leverage for 

decision making. Significant key impacts resulting from each Learning Activity are discussed below: 

‒ The FY19 Higher Education Challenges, Competitions, and Internships Multiple Case Study 

enabled OSTEM to concretize the definition of “Contribution” to NASA’s mission. As a result of the 

study, OSTEM was able to define and operationalize a clear, shared definition of contributions to 

NASA’s mission. NASA’s OSTEM was also able to develop metrics to support the assessment of 

the Office’s performance goal related to R&D products. 

‒ In FY20, the Diversity Deep Dive study identified evidence-based effective practices for 

broadening participation of students from groups historically underrepresented in STEM. Notably, 

the findings from the Diversity Deep Dive study enabled the development of the NASA OSTEM 

Broadening Student Participation Action Plan. 

‒ Findings from the FY20 Diversity Deep Dive study informed the development of the FY21 Intern 

Outcome Assessment, which served a critical role in the development of the Agency Equity Action 

Plan and the Agency Priority Learning Agenda Question focused on internships. 

LEARNING AGENDA QUESTION 2: HOW HAVE NASA STEM ENGAGEMENT INVESTMENTS 

BROADENED PARTICIPATION OF HISTORICALLY UNDERREPRESENTED AND UNDERSERVED GROUPS IN 

STEM FIELDS IN NASA STEM ENGAGEMENT ACTIVITIES? 

a. Does NASA provide higher education challenges, 

competitions, internships, and fellowships to 

students across all institutional categories and 

levels at percentages that meet or exceed the 

national percentages of science and engineering 

college graduates? 

b. In comparison to the prior fiscal year, to what 

extent do gaps decrease between percentages of 

(1) racially or ethnically underrepresented and nonunderrepresented students; (2) female and male 

students; and (3) persons with disabilities and 

persons without disabilities who applied and/or 

received internships and fellowships? 

c. How and to what extent has MUREP achieved its intended goals and objectives? 

d. To what extent are NASA interns entering the NASA or other STEM workforce or entering a STEMfocused post-graduate academic area? 

e. How can NASA OSTEM provide STEM engagement opportunities that are adaptable to the unique 

needs and contextual factors of various beneficiaries? 

DEVELOPMENT HYPOTHESIS: If data is collected to better understand the demographic characteristics of 

higher education awardees and how NASA STEM Engagement investments in higher education awards 

contribute to the diversity of the future aerospace industry’s STEM workforce, then the Office of STEM 
stem.nasa.gov

Engagement will be better equipped to make budgetary, programmatic, and operational decisions 

regarding the design, execution, and focus of future investments. 

STEM ENGAGEMENT FOCUS: Build a diverse future STEM workforce by engaging students in authentic 

learning experiences with NASA’s people, content, and facilities.

KEY LEARNING ACTIVITIES: 

KEY IMPACTS: Five key Learning Activities were developed to align with Learning Agenda Question 2

across FY19, FY20, and FY21: 1) FY19 STEM Engagement Activity Participant Longitudinal Tracking 

Feasibility Assessment; 2) FY20 Pilot Internship Longitudinal Study; 3) FY20 Diversity Deep Dive; 4) 

FY20 MUREP Portfolio Assessment; and 5) FY21 Intern Outcome Assessment. The following section will 

focus on the key impacts derived from the studies not yet discussed above for Learning Agenda 

Question 1: 

‒ Results from the FY19 STEM Engagement Activity Participant Longitudinal Tracking Feasibility 

Assessment informed the design of an enhanced data management system as well as improved 

upon the quality of performance assessment and evaluation. Another key outcome of this 

Learning Activity was the Pilot Internship Longitudinal Study that was designed to assess longterm outcomes aligned to NASA STEM Engagement goals and objectives. 

‒ In FY20, the Pilot Internship Longitudinal Study compared multiple methodologies to determine 

which would be most successful in understanding the extent to which the NASA Internship 

Program attracted students to careers in STEM and contributed to the mission of helping to 

develop a skilled and diverse future STEM workforce. This study resulted in the methodological

approach with which OSTEM ultimately employed to understand impacts of NASA student supports 

on the present and future STEM workforce. 

‒ The FY20 MUREP Portfolio Assessment was the first program-level evaluation OSTEM conducted

on one of its four Congressionally funded projects. This study established an evidence base for a 

future outcome assessment focused on the role of partnerships and Minority-Serving Institutions 

(MSIs) and the constructs related to student STEM identity, sense of belonging, self-efficacy, and 

how MSIs recruit and retain students from groups historically underrepresented in STEM. 
stem.nasa.gov

LEARNING AGENDA QUESTION 3: TO WHAT EXTENT HAVE ENHANCEMENTS TO STEM ENGAGEMENT 

PERFORMANCE ASSESSMENT AND EVALUATION BEEN IMPLEMENTED? (NOTE: LEARNING AGENDA QUESTION 3 WAS 

RETIRED AFTER FY20)

a. To what extent has NASA designed a data 

management system aligned to performance 

assessment and evaluation strategy to 

collect, analyze, and report data for STEM 

engagement investments been? 

b. To what extent has NASA developed an 

evaluation plan for outcome evaluation of 

higher education and challenges and 

competitions? 

c. To what extent has NASA increased 

transparency of STEM engagement 

performance data and reporting? 

DEVELOPMENT HYPOTHESIS: If enhancements to STEM engagement performance assessment and 

evaluation have been implemented, then the Office of STEM Engagement will be better equipped to 

collect and use evidence for budgetary, programmatic, and operational decisions.

STEM ENGAGEMENT FOCUS: Align to the federal government evidence-based policy initiatives and 

increased need for rigorous performance measures. Move from basic quantitative output performance 

measures to more robust outcome measures and develop a portfolio of evidence for use in decision 

making.

KEY LEARNING ACTIVITIES: 

KEY IMPACTS: Learning Agenda Question 3 included two key Learning Activities, framed in the format of 

a “call to action.” In FY19, NASA’s OSTEM set out to design a comprehensive data management 

system aligned to its performance assessment and evaluation strategy to collect, analyze, and report 

data for STEM engagement investments. Then, in FY20, NASA’s OSTEM went forth to award a 
stem.nasa.gov

competitive agreement to conduct a multi-year, third-party, project-level evaluation of the National 

Space Grant College and Fellowship Project. In addition to enabling NASA’s OSTEM to establish rigorous 

performance measures and collect and use evidence for budgetary, programmatic, and operational 

decisions, these Learning Activities also resulted in the following key outcomes: 

‒ The development of performance assessment and evaluation requirements and recommendations 

to support the development of an enterprise-wide performance management system—the NASA 

STEM Gateway.

‒ Inclusion of language and practices aligned to the federal government evidence-based policy 

initiatives in all of the NASA OSTEM solicitations. 

LEARNING AGENDA Question 4: WHAT ARE EFFECTIVE STRATEGIES TO SUPPORT AND MEASURE 

NASA STEM ENGAGEMENT K-12 INVESTMENTS’ ABILITY TO SPARK STUDENT INTEREST IN STEM? 

a. In what ways are NASA STEM engagement K12 activities aligned to NASA’s aeronautics, 

space, and/or science missions? 

b. How do NASA’s STEM engagement K-12 

activities serve national and regional STEM 

ecosystems? 

c. How do NASA’s STEM engagement K-12 

activities attract diverse groups of students to 

STEM? 

d. How can NASA develop data collection, 

analysis, and reporting mechanisms to better 

assess the extent to which K-12 STEM 

engagement activities’ spark students’ interest in STEM? 

DEVELOPMENT HYPOTHESIS: If NASA STEM Engagement K-12 programs utilize evidence-based practices 

for engaging students in authentic STEM experiences, student interest in STEM will be “sparked” (ignite 

and increase). 

STEM ENGAGEMENT FOCUS: Build a diverse future STEM workforce by engaging students in authentic 

learning experiences with NASA’s people, content, and facilities.

KEY LEARNING ACTIVITIES: 

KEY IMPACTS: In FY21, the fourth Learning Agenda Question was added in response to stakeholder 

feedback, which illuminated the need for a targeted focus on NASA STEM Engagements’ K-12 
stem.nasa.gov

programming and activities. By intentional design, previous evidence-building activities focused on 

higher education. Accordingly, the addition of Learning Agenda Question 4 resulted in a more inclusive 

and holistic approach to examining the NASA STEM Engagement portfolio. Two key Learning Activities 

mapped to Learning Agenda Question 4: 1) FY21 K-12 Internal Strategic Assessment, and 2) FY21 

NextGen STEM Pilot Outcome Study. Below is a discussion of their key impacts. 

‒ In FY21, the K-12 Internal Strategic Assessment resulted in the development of a NASA K-12 

STEM Engagement theory of change and logic model. 

‒ The FY21 NextGen STEM Pilot Outcome Study was conducted to develop data collection, analysis, 

and reporting mechanisms to better assess the extent to which K-12 STEM engagement activities 

spark students’ interest in STEM. This Learning Activity provided a foundational knowledge base 

for NASA’s OSTEM to identify common methods for assessing K-12 student outcomes and to 

better understand the extent to which NASA provides STEM Engagement opportunities (e.g., 

challenges, competitions, etc.) for K-12 students.

AD ASTRA – WHERE DO WE GO FROM HERE 

The FY 2019-FY 2021 Learning Agenda has set the foundation for institutionalizing a culture of evidence

across the NASA OSTEM enterprise. It has served as a key document for building a culture of learning 

and continual improvement within NASA’s OSTEM. Implementation of the Learning Agenda has 

provided a systematic approach for building and using new knowledge about project and operational 

performance for evidence-based decision making and continual improvement. 

In reflection across the years, there has been tremendous shift in the perceptions and use of evidence

across the NASA OSTEM enterprise. Throughout the NASA OSTEM enterprise, project key personnel 

and leaders are actively building and using evidence for budgetary, programmatic, and operational 

decisions regarding the design, execution, and focus of existing and future STEM Engagement

investments. 

From the strength of this foundation, NASA OSTEM will continue to enhance its role in the Agency by 

applying evidence generated from the FY 2019-FY 2021 Learning Agenda and leveraging OSTEM’s 

Learning Agenda expertise to support the development of the next multi-year OSTEM Learning Agenda, 

Agency Strategic Plan, and Agency-Level Learning Agenda. Without a doubt, the NASA OSTEM 

Learning Agenda is a key tool to the continued work of identifying priorities which will enhance decision 

and policy making through strategic systematic evidence-building, across the OSTEM enterprise and 

the Agency, effectively continuing to amplify NASA’s long held commitment to strong evidence-building 

practices.