# Presentation: 835262

**Metrics Planning and Reporting (MPAR) WG**
**Overview**
**H. K. (Rama) Ramapriyan**
**NASA/GSFC**
**Paul Davis**
**University of Maryland**
**Co-Chairs, MPARWG**
**2****nd**** Earth Science Data Systems Working Group**
**Joint Working Group Meeting**
**Greenbelt, MD **
**October 18 – 19, 2004**

## MPAR Working Group

**Mission Statement for the WG****   **
- Review and recommend program-level performance metrics and collection tools that measure how well each data activity supports the NASA Science Mission Directorate’s Earth science, application and education programs
**Membership in WG**
- WG membership open to NASA data and service provider community  (REASoN projects, DAACs, SIPSs, etc.)
- We are open to suggestions for participation by others
**Scope of Work**
- WG provides on-going MPAR review, evaluation, recommendations and metrics evolution for the NASA ES data and service provider community
-  WG recommends additions, deletions or modifications to set of metrics. Recommendations may be approved or rejected by NASA. If approved, NASA Science Mission Directorate funded Earth science data and service providers will have to make recommended changes in their reporting

## Status of FY 2004 Work Plan – Completed Actions

- Elected Co-Chair (Paul Davis)
- Adopted WG charter and rules of operation
- Organized WG Web page with native and pdf document formats
- Reviewed draft Program Metrics (10 REASoN approved metrics)
- Developed and Reviewed web-based metrics collection tool
- Developed time-phased (FY2004/Phase 1 and FY2005/Phase 2) implementation plan in response to review comments
- Formed 2 subgroups:  Education (Glen Schuster) and Unique Methods of Measuring Metrics (Chris Kummerow)
- Forwarded WG metrics and collection tool (UMd Metrics Tool) recommendations to HQ for approval (Approved July 13)
- Completed baseline implementation of UMd Metrics Tool
- Opened web-based tool for REASoN Projects’ monthly inputs
- Four projects have been providing inputs regularly

## MPARWG – Education Subgroup

** ****ACTIVE MEMBERS:** Glen Schuster, John Pickle, Carol Meyer, Rita Freuder, Jeffrey Beaudry
-  **PURPOSE**:  To create ‘*new generation*’ Education Metrics to serve NASA as well as REASoN Projects
-  **STATUS**: **ACTIVE**
- Telecons
- Met at ESIP meeting to develop education community survey framework
**PLANS**:  Surveys to serve earth science education community
- QUALITATIVE as well as hard data
- Serving all stakeholders (teachers, students, administrators, the public, faculty, etc.)
- Bank of questions is being developed
- Research
-  **GOAL** for meeting
- Generation of survey questions for OMB
- Input from full MPARWG

## MPARWG – “Unique Methods of Measuring Metrics” Subgroup

- Active Members: Chris Kummerow, S. Adamson, W. Berg, N. Saleous, W. Teng, L. Voorhees
- Goal: Develop more flexible metrics collection that better reflects progress of the data system towards meeting user needs.
- Status: Three subgroups formed to -
- 	Study feasibility of automated collection of raw data that can more readily be interpreted according to NASA/OMB requests. Goal is to minimize effort on individual data centers while optimizing centralized data interpretation software.
- 	Monitor progress of data systems towards the “perfect” system in which users get exactly what they want when they want it.  Focus on monitoring positive attributes of data system (e.g. user specified spatial subsets, parameter subsets, or data merging).
- 	Study feasibility of using formal surveys to monitor user satisfaction in a number of areas related to ease of use, data system responsiveness and outcomes of efforts involving data.
- Each group has made an initial draft proposal for this meeting.
- Plans:
- Go over each proposal at this meeting to receive comments and recommendations from entire MPAR WG membership.
- Proceed based upon input from this meeting.  Welcome new members with diverse viewpoints.

## University of Maryland Metrics Tool

**U of MD’s web-based tool will be demonstrated and available for hands-on demos**

## MPARWG Breakout Session

**Topics to be Covered**
- Subgroup Recommendations and Their Implementation
- Leftover Items from FY2004 Work Plan
- Monitor and Assess initial metrics collection program
- Adopt an annual cycle for review of the metrics baseline
- 1st year progress report
- Work Plan for FY2005
- Review Phase 2 items
- Agree on 2005 Work Plan
- Metrics Collection Status and Issues
- Disposition of FY2004 (Phase 1) UMd Metrics Tool Recommendations
- Determine causes for low reporting numbers
- Discuss new ideas for publishing metrics information and success stories/nuggets
- Other Working Group Business
- Membership – adequate representation?
- Other items as presented by the WG

## MPARWG Meeting, Greenbelt, Md, October 18-19, 2004

- BACKGROUND SLIDES

## MPARWG Program Metrics

**Draft set of core (baseline) Program-Level Metrics**
- Number of Distinct Users
- Characterization of Distinct Users Requesting Products and Information (by Internet domain)
- Number of Products Delivered to Users
- Number of Distinct Product Types Produced and Maintained by Project
- Volume of Data Distributed
- Total Volume of Data Available for Research and Other Users
- Delivery Time of Products to Users
- Support for ESE Science Focus Areas *
- Support for ESE Applications of National Importance *
- Support for ESE Education Initiatives *
- * *When applicable*

## ESE MPAR Working Group – Rules of Operation

**MPAR WG Recommendations to NASA HQ / ESE:**
**Recommendation can (per charter) be:**
- To add, revise or drop one or more metrics;
- To adopt a particular collection / reporting tool.
**Recommendation must be accompanied by:**
- Definition and rationale (e.g. what does this metric mean, why does it matter?);
- Collection method (how would this metric be collected, based on what input?);
- Intended Use (what analysis would this metric allow, how would the program office or DSPs use it?)
- Justification (e.g. how does this metric measure how a DSP supports specific ESE objectives);
- Impact analysis (e.g. cost and effort required to implement).
**MPAR WG should consider ‘beta testing’ draft recommendations to prove feasibility of collection or feasibility of use of a proposed tool prior to final recommendation.**

## ESE MPAR WG – Rules of Operation, Continued

**MPAR WG Internal Processes:**
_**Proposed Process**_** to adopt recommendation (Depending on recommendation, WG Chair can determine degree of review and number of necessary steps):**
- Majority vote of MPAR WG members to adopt proposed recommendation as a WG draft;
- One MPAR WG member appointed shepherd
- 30 day period of ESE activity review (to include other Earth Science WGs) for WG draft (not all ESE activities will be MPAR WG members) coordinated by shepherd;
- Shepherd assembles comments, drafts revisions to recommendation per activity feedback, presents summary of feedback and draft revisions to full WG;
- WG considers revisions and need for ‘beta test’;
- Majority vote of MPAR WG members to adopt revised WG draft;
- Shepherd coordinates Impact Analysis, Rationale, Justification
- Two thirds vote of MPAR WG members to adopt final recommendation package and send to HQ / ESE.

## ESE MPAR WG – Rules of Operation, Continued

**MPAR WG Internal Processes:**
**Officers.**
- Co-Chair, elected by majority of MPAR WG members, one year term.
- Executive secretary, appointed by NASA/GSFC
- SGT contract support
- Facilitate WG coordination, documentation, and action items
- Core WG membership includes DSP and User representation.
- All classes of ESE DSPs to be included.
- Form Subgroups, elect chairs, per charter as needed.
**Frequency of Meetings.**
- Telecons, as required
- Semi-Annual, or as needed, meetings.
- Make the most of e-mail, posts to MPAR WG website, and groupware.

## ESE MPAR WG – First Year Work Plan

**January 2004 – September 30, 2004 (synch up on fiscal years)**
- Adopt charter, elect Co-Chair, adopt rules of operation.
- Review draft Program Metrics, prepare recommendation(s) for NASA HQ on these, by March, 2004.
- Review collection tools (e.g. U MD and EDGRS) and concepts of operation, make recommendation on these, by March 2004.
- Secure HQ approval of metrics/tools baseline by April 2004.
- Complete implementation of collection tool(s), by June, 2004.
- Monitor initial metrics collection, assess effectiveness of collection and reporting process and assess quality of the collected metrics.
- Adopt an annual cycle for review of the metrics baseline that meets HQ / ESE requirements.
- Provide first year progress report; FY05 work plan, September 30, 2004.

## Background – Study Team Recommendations on Metrics

_**Recommendation #1: **_**It is recommended that ESE not seek exceptions to the current set of NASA regulations and guidelines for solicitation opportunities and funding instruments. **
_**Recommendation #2: **_**It is recommended that the appropriate level of accountability for a DSP be defined by a combination of adherence to NASA’s “Principal Purpose Test,” as found in NASA Procedures and Guidelines (NPG) 58001, Part 1260.12, and implementation of the SEEDS accountability classification for DSPs [see the Formulation Team Report]. The levels of accountability required depend on the levels of service, and the metrics given in the following tables are examples of how the accountability and the levels of service could be ensured. Both NASA funding instrument reporting requirements and a SEEDS level of accountability can be used to define appropriate metrics collection and reporting as a function of roles and responsibilities for potential DSPs.**_** **_
_**Recommendation #3: **_**Because of the need to improve sponsor-required user satisfaction metrics or outcome metrics, it is recommended that this class of metrics be studied further. An extension of this study should be to identify metrics that are directly traceable to the objectives of the ESE science and applications program, so that the effectiveness of the support that ESE data management activities provide to the science and applications program can be documented, and thus the contribution of ESE data management to successful outcomes of the science and applications program can be shown. **

## Background – Study Team Recommendations on Metrics, Continued

_**Recommendation #4: **_**It is recommended that the SEEDS Program Office ... take on the responsibility of managing and collecting program level metrics and accomplishments as an enterprise function. It is recommended that metrics activity by the SEEDS Program Office be limited to those metrics that are required for program level assessment and monitoring, and the SEEDS Program Office not become involved with metrics that are used internally by data management activities for their own management and monitoring. Thus the SEEDS Program Office would be involved with one set of defined metrics for ESE data and information management and services, and would obtain from each data management activity that subset of the metrics appropriate for it (e.g. metrics required from operating activities would not be the same as those appropriate for research activities). The SEEDS Program Office would maintain and update the program level metrics over time. **
_**Recommendation #5: **_**It is recommended that a MPAR working group (WG) be established for ongoing evaluation and evolution of appropriate metrics. The MPAR WG would also look into means of minimizing the impact of program metrics collection on DSPs. This may include exploring commonality among metrics to be reported by various DSPs and recommending/providing tools to assist in gathering, maintaining and reporting on metrics. **
_**Recommendation #6: **_**It is recommended that future solicitations for DSPs include a requirement for the bidders to suggest a set of metrics that demonstrate how their proposed activities will address the goals of ESE’s science and applications programs and require participation by the selected DSPs in the MPAR WG. The solicitations also must require the DSPs to gather and report on an agreed upon set of metrics. **

## MPAR Working Group Membership

**SEEDS MPAR Study Team members (Feb 2002 to Sept 2003)**
- Bud Booth - SGT
- Howard Burrows – AUSI (ESIP with IBM/JHU)
- Bob Chen - SEDAC
- Don Collins – JPL PO.DAAC (now retired)
- Kathy Fontaine – GSFC (GCDC)
- Greg Hunolt - SGT
- Steve Kempler – GSFC (GES DAAC)
- Frank Lindsay – UMD (now at NASA HQ)
- H. K. Ramapriyan – GSFC (ESDIS Project)
- Hank Wolf - GMU

## MPARWG Membership

**MPARWG members (Oct 2003 to Present)**