# Presentation: 472718

## Metrics Planning & Reporting Study               H. K. Ramapriyan

**Purpose of Study**
**Identify various types of institutions to be funded and appropriate funding mechanisms for participants**
**Define appropriate metrics collection and monitoring mechanisms for reporting (publicizing) performance (accomplishments)**
**Recommend, to Earth Science Enterprise, appropriate language for inclusion in various types of solicitations **
**Schedule**
**Draft questions to send to sponsors and implementing organizations – January 4, 2002 ****(*****completed)***
**Community Workshop - February 5-7, 2002 *****(completed)***
**Refine questions and “visit list” - February 15, 2002 *****(completed)***
**Distribute questionnaires to “visit list” - March 8, 2002**
**Draft report on metrics planning and reporting as a function of “class” – April 1, 2002**
**Obtain responses and conduct follow-up interviews – March – May 2002**
**Preliminary report – June 30, 2002**
**Further contacts with sponsors and implementing organizations as needed - July - October 2002**
**Recommendations to  ESE about metrics planning and reporting mechanisms - December 2002**

**Approach**
**Engage community through workshops and survey interviews**
**Survey sponsoring and implementing organizations**
**Identify/Define “classes” of participants (e.g., types of ESIPs; Program and Project offices) and define reporting requirements**
**Survey existing mechanisms for metrics planning and reporting, and their pros and cons**
**Contact projects (e.g., HST, ESSP), ESIP federation members and other entities to learn about mechanisms being used**
**Obtain opinions of sponsoring organizations about metrics information they are getting (and missing)**
**Identify metrics planning and reporting requirements for solicitations and funding instruments**
**Identify requirements mandated by the government (NPGs etc.) as appropriate to different classes of participants and dollar levels**
**Identify documentation requirements for different classes of participants (Grants, Working Agreements, Contracts, etc.)**
**Identify requirements/funding flow options for the different classes of participants**
**Status**
**Started task with SGT - 12/1/01**
**Drafted survey questionnaire, letters, “visit list”**
**Added 3 “community” participants to study team (“deep involvement”) after February 2002 workshop**
**Questionnaire ready to be sent to initial list of 26 respondents**
**Reviewing NASA solicitations and funding instruments to prepare report on metrics planning and reporting as function of “class” of participants**
**March 5, 2002**

## Metrics Planning & Reporting Study               H. K. Ramapriyan

**~15 individuals attended breakout session on Metrics Planning and Reporting**
**Representatives from HQ, DAACs, ESIPs and SEEDS team**
**What we heard (highlights)**
**General consensus: Current metrics only partially reflect a provider’s performance, e.g., measures of utilization of data and products by science community and their value to society are currently not reflected in metrics collection**
**Measuring “value” is important, but no good ways exist yet to do this**
**Multiple viewpoints need to be considered for metrics planning and reporting depending on classes of participants (e.g., NASA HQ, Project sponsors, Data/Service providers, Provider internal organizations,End users)**
**Questionnaire should include more direct questions that deal with accountability**
**“****Visit list” should include other organizations (e.g., NSF, DOE, LASP, Vexcel, financial community)**
**Metrics collection offers an opportunity for corrective action, advertising and publicizing successes**
**The study team should review the Federation’s metrics processes and protocols as a body of lessons learned**
**There is a potential for lack of data stewardship across entities in the SEEDS environment**
**Governance, management, and authority in the “SEEDS era” have not been addressed**

**Community Workshop - February 5-7, 2002**

## Metrics Planning & Reporting Study               H. K. Ramapriyan

**Added 3 new participants to study team and have been “deeply involved” as a Focus Review team**
- Don Collins (DAAC Alliance), Manager, JPL PODAAC
- Frank Lindsay (Federation) - Manager, Global Land Cover Facility ESIP-2, University of Maryland
- Hank Wolf (Federation), Assistant Director of CEOSR, George Mason University,  Member, Seasonal to Inter-annual ESIP-2 Project
**Sent draft question list to above three (Focus Review Team) for review and comment, resulting in excellent feedback:**
- Current draft largely ignores direct questions on accountability
- Need to differentiate between a project’s organization and its institutional organization – two very different views (e.g., task-reporting and administrative-reporting)
- To improve clarity and simplicity,  like questions should be grouped under their own headings, several questions should be merged and simplified, and several new questions are needed.
**Revised question list based on feedback - expect to send to “initial visit list” of 26 organizations by March 8, 2002**
**Developed a matrix of drivers for Success, Accountability and Value to reflect multiple viewpoints for the different classes of participants**

**Post-Workshop Actions**