National Aeronautics and 

Space Administration

Assessing the Impact of NASA’s STEM Engagement Investments: 

Development of External & Internal Performance Measures

March 20, 2018

NASA Glenn Research Center

Rob LaSalvia, Center Education Director

Rick Gilmore, Evaluation Manager

Dr. Catherine Graves, Lead Evaluation Specialist

Dr. Tara Strang, Evaluation Specialist
Performance Assessment Strategy, Framework,

& Measures Development Process
Assessing the Impact of NASA’s STEM Engagement Investments: 

Performance Assessment & Evaluation Overview

Strategy 

•Broad strategic goals designed to advance NASA’s mission and 

address relevant national problems, needs, challenges and 

Performance opportunities.

Framework

•Timeframe: 2018 - 2022

•Ongoing monitoring and reporting of program 

accomplishments, particularly progress toward prePerformance established goals.

•Includes outputs such as: demographics, course 

Assessment development, paper/presentation; and outcomes such as 

building research capacity of higher education institutions.

•Timeframe: Short term (Annual)

•Systematic study using research methods to collect and 

analyze data to assess how well a program is working and 

why.

Evaluation •Includes outcomes such as: science identity, scientific 

literacy, STEM graduate program matriculation/career 

outcomes.

•Timeframe: Long term 

Assessment

Performance 

Evaluation
External and Internal Performance Measures 

Development Process

STEP ONE

Review Past Performance

A brief history of NASA’s Office of 

Education performance measures, 

data collection processes and tools, and 

performance results

• FY 2014 – FY 2017 Performance Goals, 

Annual Performance Indicators, Results

• Strategic Object Annual Reviews (SOAR)

• OMB and BSA guidance and 

recommendations: Concerns, need for 

change/reformed measures, processes, 

and tools

STEP TWO

Benchmark Other Federal 

Agencies, Literature Review, 

& Stakeholder Discussions

Summary of guidance from 

literature and practice 

(BSA priorities, literature review, 

benchmarking, internal and external 

stakeholder discussions)

• BSA priorities

• Literature review findings

• Benchmarking findings

• Internal stakeholder discussions

• Mission Directorate discussions

• Expert Review Panel - External 

stakeholder discussions

STEP THREE

Finalized Candidate 

Performance Measures

Recommended External and Internal 

Performance Measures, 

Data Collection Sources, Processes, and Tools 

to Capture and Report Performance Data

• Recommended external and internal 

performance measures

• Data collection sources

• Data collection processes

• Data collection tools

• Assessing and analyzing performance data 

(including ERP procedure and scorecards)

• Reporting performance data
External and Internal Performance Measures 

Development Schedule

Performance Measures Development Milestones

1. Internal Assess of the Office of Education’s Evaluation & 

Performance Measurement including pervious APR’s, SOAR 

assessments, & OMB guidance (October 2017).

2. Review of relevant literature (evidence-based practices & current 

trends in STEM education and evaluation) & Benchmarking of 

Federal Agencies’ STEM education performance measurement 

strategies and tools (November 2017 – March 2018).

3. Internal NASA Stakeholder Discussions (March 2018).

4. Expert Panel Review (April 2018).

5. Draft External and Internal Performance Measures Document, 

including data collection tools and processes (May 2018).

6. Final External and Internal Performance Measures Document, 

including data collection tools and processes (June 2018).

7. (anticipated) Candidate FY19 & FY20 Performance Goals and Annual 

Performance Indicators due to OCFO-SID (July 2018).

Oct. Nov. Dec. Jan. Feb. Mar. Apr. May Jun. Jul. Aug. Sep.

1

2

3

4

5

6

7

Initial Input 

Requested

Follow-Up Input 

Requested

Alignment with OSE Strategy Development
Gathering Input from Internal Stakeholders and 

External Experts
Internal Stakeholder Discussions

Feb. 27 – Mar. 5

• Kris Brown

• Joeletta Patrick (MUREP)

• Chris Fitzsimonds (OCFO-SID)

• Carolyn Knowles (NIFS)

• Jeppie Compton/Crystal Bassett (EPSCoR)

• Joeletta Patrick/Erica Alston (Space Grant)

• Diane Detroye (SEAP/NGS)

• Roosevelt Johnson (Education Senior 

Advisor)

• Bev Girten (Informal)

• Holly Degn (Budget)

• Tammy Rowan (IT Infrastructure)

• HQ Strategy Meeting Attendees 

• National Space Grant Directors’ Meeting - Flip Session Discussions 

Mar. 12 – 23

• Center Education Directors & Deputy Directors

• Mission Directorates Education Leads

• MIRO PIs

• EPSCoR PIs

• Selected Activity Managers and Third-Party Evaluators 

with demonstrated success in evaluation and 

performance assessment

• SME’s (Mission - Technical People with a History of 

working with Education)
External Expert Review Panel Discussions

1. Expert Review Panel to be held in Washington, D.C. late April or early May

2. Process for Identifying Candidates for Expert Review Panel (ERP) 

• Representatives from the NAC and NSF along with evaluation experts were provided

• Nationally Recognized Experts with publication records and experience in our needed areas of 

expertise

Program Structure and Evaluation

Higher Education Diversity

Building Technical Research Capacity

IT Systems / Social Media & Emerging Technologies

Science Literacy / Large-scale Public Engagement Campaigns 

• Experience in developing national policy, strategies, work with federal-funded programs
Office of STEM Engagement

Performance Assessment Framework

DRAFT
Proposed Office of STEM Engagement Performance Assessment Strategy Framework

Strategic Goals

Strategic Objectives

2018 NASA Strategic Plan

Multi-year 

Performance Goals Annual Performance 

Indicators

Agency 

Priority Goals

Cross-Agency 

Priority Goals

Mission Directorates/Mission Support Offices

Traditional Performance Assessment Strategy Enhanced Performance Assessment Strategy

2018 NASA Strategic Plan

Strategic Goal 3: Address National Challenges & Catalyze Economic Growth

Strategic Objective 3.3: Inspire & Engage the Public in 

Aeronautics, Space, & Science

Strategic 

Performance 

Assessment 

Questions

Success Criteria

Office of STEM Engagement

External Multi-year 

Performance Goals

External Annual 

Performance 

Indicators

Internal Multi-year 

Performance Goals

Internal Annual 

Performance 

Indicators

Strategic 

Performance 

Assessment 

Questions

Success Criteria

Enhancement 

Based on NSF 

Performance 

Assessment 

Model

Draft 3-20-2018
Proposed Office of STEM Engagement Performance Assessment Strategy Alignment to the NASA Public & STEM Engagement Focus Areas

NASA Public & STEM 

Engagement Focus Area 1

Create unique opportunities 

for students and the public 

to contribute to NASA’s 

work in exploration and 

discovery

1. Enhance student 

experiences through 

NASA unique projects 

and challenges.

2. Enable public 

participation in and 

contributions to NASA’s 

work.

3. Provide opportunities 

for students and the 

public to share ideas and 

engage with NASA in 

addressing real-world 

problems.

NASA Public & STEM 

Engagement Focus Area 2

Build a diverse future STEM 

workforce by engaging 

students in authentic 

learning experiences with 

NASA’s people, content, and 

facilities. 

1. Provide learning 

opportunities to attract 

and engage all students 

in STEM, including those 

historically 

underrepresented and 

underserved. 

2. Enhance capabilities of 

institutions and support 

educators to position 

them to better serve 

students, including those 

historically 

underrepresented and 

underserved. 

3. Cultivate strategic 

partnerships with statebased regional and 

federal public, academic 

and private institutions 

to magnify NASA’s 

efforts and investments. 

NASA Public & STEM 

Engagement Focus Area 3

Strengthen public 

understanding by enabling 

powerful connections to 

NASA’s mission and work. 

1. Provide a broad 

spectrum of 

opportunities to attract 

and engage the public. 

2. Target a diverse set of 

audiences to increase 

public understanding of 

NASA’s work and to 

enhance appreciation of 

STEM.

3. Provide direct public 

access to NASA missions 

and discoveries in an 

array of settings, 

capitalizing on 

technology to extend our 

reach and impact. 

STEM Literate 

Citizenry

STEM 

Workforce 

Development

Institutional 

Research 

Capacity

Government 

Efficiency

Evaluation &

Accountability

Digital 

Assets

Programmatic Strategic Investment Areas

Cross-Cutting Operational Support Areas

Aligned to OMB 

and Administration 

Cross-Agency 

Priorities

Draft 3-20-2018
Programmatic Strategic Investment Areas

STEM Literate 

Citizenry

STEM Workforce 

Development

Institutional 

Research 

Capacity

Enhance the vitality and diversity of 

the nation's STEM and aerospace 

workforce through investments in 

higher education. 

Bolster the STEM 

research capacity 

of universities and 

colleges including 

Minority-Serving 

Institutions.

Support the capacity 

of education 

institutions, partners, 

and collaborators to 

provide NASA mission 

themed authentic 

STEM experiences

(ASE). 

Provide opportunities 

for strategic partners 

to collaborate with 

NASA to engage 

learners in STEM 

activities that 

capitalize on 

NASA’s unique 

assets and 

content.

Provide significant, direct student awards in higher 

education aligned to NASA's STEM and technical capacity.

Space Grant, 

EPSCoR, & MUREP 

investments will 

contribute to 

American technical 

capability through TBD 

peer-review 

publications and 

technical 

presentations. IRC 1: Are NASA’s 

STEM engagement 

investments 

contributing to the 

research capacity of 

higher education 

institutions overall and 

specifically the 

research capacity of 

minority-serving 

institutions? 

SWD 1: Are significant direct higher education awards meeting or 

exceeding the national science and engineering workforce percentages for 

racially or ethnically underrepresented students, women, and persons with 

disabilities as determined by the most recent, publicly available data from the 

National Survey of College Graduates? Have gaps between the percentages of: (1) 

racially or ethnically underrepresented and non-underrepresented; (2) women and 

men; and (3) persons with disabilities and persons without disabilities who 

received significant direct student awards decreased from the previous fiscal year?

SWD 2: Are higher education investments 

contributing to the Agency's ability to 

successfully compete for the nation's STEM 

talent? Are higher education investments 

competitive with other Federal and commercial 

STEM & aerospace talent development investments 

in their ability to attract and retain diverse STEM 

talent?

SLC 1: Are STEM 

engagement 

investments 

providing ASE to 

learners who are 

representative of 

the diversity of the 

nation and 

geographically 

distributed across 

the nation?

Strategic 

Assessment 

Questions

Strategic 

Assessment 

Questions

Annual 

Performance 

Indicators

Annual 

Performance 

Indicators

Performance Goals Strategic Investment Themes Performance Goals

SLC 2: Are STEM 

engagement investments 

supporting the capacity 

of strategic partners

to provide NASA 

mission-themed ASE

to learners?

Draft 3-20-2018
Strategic 

Assessment 

Questions

Strategic 

Assessment 

Questions

Annual 

Performance 

Indicators

Annual 

Performance 

Indicators

Performance Goals Strategic Investment Themes Performance Goals

Cross-Cutting Operational Support Areas

Enhance the effectiveness of education 

investments using evaluation-driven 

processes.

Enhance the 

efficiency & reach 

of NASA's STEM 

engagement 

investments 

through the use of 

digital assets.

Ensure NASA's STEM 

engagement 

investments are 

managed and 

operated efficiently. 

Demonstrate planned 

progress to 

efficiently execute 

the budget, scope, 

and schedule of 

NASA's STEM 

engagement 

Investments.

Demonstrate planned progress in developing outcomebased evaluation capacity for internal and third-party 

evaluations as determined by an expert review panel.

Demonstrate 

planned progress 

to utilize digital 

assets to increase 

the efficiency and 

reach of NASA's 

STEM engagement 

investments. 

Government 

Efficiency

Evaluation &

Accountability

Digital 

Assets

DA 1: Are digital assets 

being used effectively to 

increase the efficiency and 

reach of NASA's Office of 

STEM Engagement's 

investments through the 

use of digital assets?

EA 1: Are NASA’s Office of STEM 

engagement investments demonstrating 

planned progress in developing outcomebased evaluation capacity for internal and 

third-party evaluations? 

GE 1: Are NASA Office of 

STEM Engagement’s 

investments run 

efficiently with respect 

to cost, scope, and 

schedule?

EA 2: Assess the feasibility of evaluation 

of major grant programs and develop an 

implementation schedule.

Draft 3-20-2018
Proposed Office of STEM Engagement Performance Assessment Strategy:

Institutional Research Capacity (IRC)

Programmatic Strategic Investment Area: Institutional Research Capacity (IRC)

Strategic Investment 

Area:

Institutional STEM Research Capacity (IRC)

Performance Goal: Bolster the STEM research capacity of universities and colleges 

including Minority-Serving Institutions.

Annual Performance 

Indicator:

Space Grant, EPSCoR, & MUREP investments will contribute to 

American technical capability through TBD peer-review publications 

and technical presentations. 

Strategic Assessment 

Question:

Are NASA’s STEM engagement investments contributing to the 

research capacity of higher education institutions overall and 

specifically the research capacity of minority-serving institutions? 

Success Criteria: • Dissemination of TBD research (e.g., peer-reviewed research 

publications, technical paper presentations...)

• Development of institutional research capacity (e.g., research facility 

development, new course development, providing research 

experiences) at TBD higher education institutions.

Contributing Investments: EPSCoR, MUREP, & Space Grant

Draft 3-20-2018
Proposed Office of STEM Engagement Performance Assessment Strategy Framework:

Government Efficiency (GE)

Cross-Cutting Operational Support Area: Government Efficiency (GE)

Cross-Cutting Operational 

Focus Area

Government Efficiency (GE)

Performance Goal: Ensure NASA's STEM engagement investments are managed and 

operated efficiently. 

Annual Performance 

Indicator:

Demonstrate planned progress to efficiently execute the budget,

scope, and schedule of NASA's STEM engagement Investments.

Strategic Assessment 

Question:

GE 1: Are NASA Office of STEM Engagement’s investments run 

efficiently with respect to cost, scope, and schedule?

Success Criteria: • Meet or exceed the budget execute schedule. 

• Investments meet or exceed their scope. 

• Investments meet or exceed their schedule. 

Contributing Investments: EPSCoR, MUREP, Next Generation STEM, & Space Grant

Cross-Cutting Operational Support Areas

Draft 3-20-2018
Questions