# Presentation: 302506

## Preparations for NASA’s 2009 Earth Science Senior ReviewOctober 28, 2008

- Cheryl Yuhas
- Earth Science Division
- Science Mission Directorate

## Outline

- Objectives of the 2009 Senior Review
- 2007 Senior Review Results
- Implementation of the 2009 Senior Review

## Objectives of the Senior Review Process

- Congressional Direction:
- NASA shall “assess the cost and benefits of extending the date of determination of data collection from those missions”, and
- “for those mission with an operational component, NOAA or any other affected agency shall be consulted and potential benefits from missions ... taken into account.” [from Congressional authorizations]
- The ultimate goal is to *evaluate the allocation of mission operation and data analysis funds with the aim of maximizing the missions’ contributions to NASA’s and the nation’s goals within the available resources.*
- Finally, our internal goal is to develop a repeatable process for subsequent cycles, and to improve our overall management of our aging satellite assets

## Earth Science Missions

## Earth Science Missions

- Jun-11

## History and Context of Senior Review

- Senior Review for Earth Science initiated in 2004
- Replace an ad hoc process for termination decisions with an open process
- Used approach employed by Space Science with minor changes
- 1st SR convened April 2005
- Included: TRMM, Terra, ICESat, TOMS, Jason-1, ERBE, GPS, UARS, SAGE III, QuikSCAT, GRACE, Acrimsat
- Resulted in termination recommendation for UARS and ERBS; GPS moved to R&A program for review
- SAGE III, ERBS, TOMS, and UARS failed or were terminated before end of 2006
- 2nd SR convened April 2007
- Included: CloudSat, Aqua, SORCE, EO-1, TRMM, Terra, ICESat, Jason-1, QuikSCAT, GRACE, Acrimsat
- All missions continued through FY2008 & FY2009, all except Acrimsat recommended for continuation for FY2010 & FY2011
- 3rd SR to be convened April 2009
- Will include: Aura, CALIPSO, CloudSat, Aqua, SORCE, EO-1, TRMM, Terra, ICESat, Jason-1, QuikSCAT, GRACE, Acrimsat
- To cover the budget years of FY2010 & FY2011 for commitments, FY2012 & FY2013 for projection.

## Process and Resultsof the2007 Senior Review

## Senior Review Panels

- Three Panels were established by the ESD to conduct the Senior Review:
- Senior Review Science Panel
- Education & Public Outreach Panel (E/PO)
- Core mission Review Panel (CoMRP)
- E/PO panel reviewed the E/PO section of the proposal only, and reported findings to the Senior Review panel and to the ESD directly.
- Core Mission Review Panel (CoMRP) reviewed basic continuation issues only.  The CoMRP reported its findings to the Senior Review Science panel and to the ESD directly.
- Reports from all three panels are available on the web at

- http://nasascience.nasa.gov/earth-science/mission_list

## 2007 Senior Review Science Findings

- Science Panel provided general findings applicable to all missions under review, these were:
- NASA could be more proactive in facilitating the integrated use of the measurement constellation via competed research proposals (e.g. NASA R&A).
- Basic continuation is critically important for all missions through FY2008-2009 to best enable NASA to continue to meet its stated science objectives.
- A detailed analysis of the basic mission science budgets could be led by NASA HQ to verify the cost-effectiveness of current core mission activities, and to seek ways that mission budgets could be decreased while maintaining the production of routine data streams.
- Nearly all proposed enhanced mission products were viewed as being more appropriate for core mission or for open competition (NASA R&A) rather than as products that could be uniquely derived by mission teams.
- In addition more detailed findings for each mission were presented in their final report.

## Conducting the CoMRP Review

- Core Mission Review Panel (CoMRP) conducted assessments in three areas related to the Basic Mission proposal:
- Core science data products
- Instrument and spacecraft health and performance
- Mission operations planning and execution
- The CoMRP included different technical expertise to conduct these different evaluations
- Partner agency - data users - to evaluate the utility of the core science data products
- Technical experts to evaluate instrument & spacecraft health
- Mission ops expertise (possibly related to #2) to evaluate operations execution plans

## CoMRP Charter (1) - Data Products

- Evaluated the value of the science data products to the National strategic requirements.
- Are the measurements (still) critical to the National interests and are the data products of sufficient quality for National needs? *As determined by the Partner representatives.*
- Are the products listed as “core” truly useable without additional science input (beyond routine calibration and validation activities necessary to maintain the integrity of the products)?
- Looking across the entire portfolio of mission data products, is there any unnecessary duplication of science data products between missions?
- The output from this portion of the evaluation was a quality ranking (High/Medium/Low Utility) of the core data products from all missions, focusing on the importance of the products to National objectives.
- CoMRP panel members also served an outreach & coordination function within their home organizations relaying information about NASA’s mission extension process.

## Summary Core Data Products Ranking

## CoMRP Charter (2) - Technical Status

- Evaluated the instrument and spacecraft heath and performance, addressing the following questions:
- Can the instruments still produce science quality observations?
- Are the instruments healthy and functioning within defined limits?
- Does the spacecraft performance meet mission requirements?
- Evaluated the completeness of the mission operations approach.
- Are the proposed operations plans adequate to ensure the continued healthy operations of the satellite?
- Is there the possibility of **significant** cost savings through  operational improvements which have not considered by the mission team?
- And for both, what is the projected performance for these parameters throughout the proposed duration of the mission extension (up to four years through September 2011)?

## Technical Assessment Results

- Comments
- “Age” means there was no overriding failure mechanism identified, but the advancing age placed the satellite at general risk
- SORCE SIM and TIM are low risk to continue for 4 years, but the overall mission is at medium risk if XPS & SOLSTICE are critical to mission success
- Terra is at medium risk if no operational workarounds are possible for the degrading Solid State Recorder (SSR)

## 2007 Technical Review Observations

- The proposals did not have sufficient information for a definitive technical assessment
- The Panel identified follow-up questions for each mission to improve the quality of the assessments.
- Primarily related to instrument performance and performance projections.  Where detail was lacking we took the proposer’s presentation on face value, but identified the lack of detail as an issue.
- This is the start of a continuing assessment task, to develop a better understanding of the reliability of aging instruments.

## Preparations for the 2009 Senior Review

## Preparations for 2009 Senior Review

- We will generally follow the approach taken for the 2007 Senior Review.
- Preparation for the scope and execution has been following parallel paths
- Define scope of Senior Review, including available budget, missions included and schedule
- Collect the three review panels: Science, CoMRP, and E/PO
- And is then followed by one primary path
- Finalize Senior Review process (includes formal announcement letter)
- Issue Request for Proposals to missions
- Missions generate proposals
- Collect and review proposals
- Formal presentation to the SR panel and obtain panel report
- Complete ES internal review and decision process

**Notes:**
Lessons learned we are incorporating:

Drop enhanced products proposals
Do not include competed science 
Better define budget line items, core DA and Mission Ops scope
Clearly state connection of SR and PPBE budgets in call letter
Adjust Proposal guidelines to accommodate flagship missions and facility instruments
Modify approach to E/PO to be less single-mission oriented.
Add OMB efficiency metric: a multiple-choice option of previously-OMB-approved metrics.  Proposal must baseline 2009 and propose goals for 2010 & 2011. Example metrics:  publications, #data products, data latency, decreased operating costs.  Other ideas welcome.  Team can propose one of their own, but can’t be assured that OMB will accept.

Organize panel & review schedule to ensure panel has thoroughly reviewed proposals.   Improve focus of what we ask for to minimize length. Limiting to status updates not practical since panel will be new, and we can’t ask them to read 2 sets of proposals.

## 2009 Senior Review Schedule of Activities

## Senior Review Roles & Responsibilities

- ESD Director - Mike Freilich
- Decision authority for Programmatic decisions
- ESD Science Lead - Jack Kaye
- Leads science focus area managers
- Senior Review Flight Program Official - Steve Volz
- Collects review panels and oversees execution of process and objectives
- Senior Review Program Officer - Cheryl Yuhas
- Executes the overall senior review process
- ESD Focus Area Scientists - all
- Involved as the deliberative group evaluating panel findings and formulating programmatic recommendations to the ESD Director

## Senior Review Science Evaluation Panel

- Drawn from outside of NASA entirely (preferable), from outside of the immediate NASA Earth Science organizations (definitely)
- 2009 Chair to be chosen from previous Senior Review panel
- In general, the other panel members will be new to the process
- The goal for the panel is balance across earth science disciplines (oceans, atmospheric chemistry, weather, climate)
- The Panel is providing “findings” only to the Science Directorate, not formal recommendations

- Gregory P. Asner (chair), Steve Ackerman, Anthony Busalacchi, Janet Campbell, Tsing-Chang Chen, Dennis Hartmann, Thomas Herring, Ian Joughin, Michael Prather, William Rossow, Azadeh Tabazadeh, Xubin Zeng

## What will be the 2009 Senior Review Evaluation Criteria?

- The primary criteria will be scientific relevance of the mission/measurement to NASA Science Strategic Plan, and an evaluation of the scientific merit of the individual missions
- This will establish the criterion for continuation
- Secondary but still important criteria will be used for fine tuning budget allocations between missions and within individual missions (between MO and DA):
- Efficiency and cost effectiveness of the mission operations
- Multiple instrument and satellite utility of the data products
- Quality and timeliness of the baseline data products
- Operational users’ considerations and needs
- Also needed for the proposal will be an assessment of the projected mission spacecraft and instrument performance

## What are we looking for in the proposals?

- Mission Operations
- Is the implementation efficient and cost effective?
- Is the risk management approach appropriate?
- Will the instrument(s) and spacecraft continue to meet measurement requirements?
- Core Science
- Are the data products critical to addressing the SMD strategic science objectives (tied to the strategic plan)?
- Are the mission specific data products produced efficiently and effectively?
- Are the data products of use and being used by the science community, and by the broader national and international user community?

## What are the possible Senior Review outcomes?

- Direction will come from the Earth Science Division (ESD) Director.
- Possible outcomes include:
- Mission continuation per proposal
- Concurrence on overall mission operations and data analysis approach
- Mission continuation with modifications
- Possibly to the data analysis portion, including revision of the implementation (including funding) approach
- Possibly to the mission operations portion, including possible adjustments to funding, staffing or risk approach
- Mission termination
- Direction for preparing and executing the termination of the overall mission
- Mission teams will be given time to prepare a detailed response to the programmatic direction from the ESD

## Changes based on 2007 Lessons Learned

- Drop enhanced products proposals
- Do not include competed science
- Better define budget line items, core DA and Mission Ops scope
- Adjust Proposal limits to accommodate flagship missions and facility instruments
- Clearly state connection of SR and PPBE budgets in call letter
- Improve focus of what we ask for to minimize length.
- Modify approach to E/PO to be less single-mission oriented.
- Organize panel & review schedule to ensure panel has thoroughly reviewed proposals.
- Add OMB efficiency metric: a multiple-choice option of previously-OMB-approved metrics.
- Proposal must establish a 2009 baseline and propose goals for 2010 & 2011.
- Example metrics:  publications, #data products, data latency, decreased operating costs.  Other ideas welcome.