# Presentation: 083281

## Metrics Planning & Reporting Study               H. K. Ramapriyan

- Metrics Planning and Reporting Study
- Status Overview
**SEEDS Community Workshop - 6/19/02**

**Study Team:  **
**H. K. Ramapriyan (Rama), Kathy Fontaine, NASA GSFC**
**Bud Booth, Greg Hunolt, SGT, Inc.**
**“****Community” Participants: **
**Don Collins, Manager, JPL PO.DAAC**
**Frank Lindsay, Manager, GLCF (ESIP-2), U of MD**
**Hank Wolf, Assistant Director, CEOSR **
**	****	****   ****and** **Member, SIESIP (ESIP-2), GMU**

## Metrics Planning & Reporting                        H. K. Ramapriyan

**Purpose of Study**
**Identify various types of institutions to be funded and appropriate funding mechanisms for participants**
**Define appropriate metrics collection and monitoring mechanisms for reporting (publicizing) performance (accomplishments)**
**Identify various governance options, their impact on metrics planning and reporting, and how they relate to ESE mission roles/responsibilities**
**Recommend, to Earth Science Enterprise, appropriate language for inclusion in various types of solicitations **

## Metrics Planning & Reporting                       H. K. Ramapriyan

**Approach**
**Engage community through workshops and survey interviews**
**Survey sponsoring and implementing organizations**
**Identify/Define “classes” of participants (data service provider classes similar to types of ESIPs; Program and Project offices) and define reporting requirements**
**Survey existing mechanisms for metrics planning and reporting, and their pros and cons**
**Identify options for governance structures**
**Impact on metrics planning and reporting**
**Relationship to ESE mission roles and responsibilities**
**Identify metrics planning and reporting requirements for announcement opportunities and funding instruments**
**Identify requirements mandated by the government (NPGs etc.) as appropriate to different classes of participants and dollar levels**
**Identify documentation requirements for different classes of participants (Grants, Cooperative Agreements, Working Agreements, Contracts, IRDs, ICDs, Operations Agreements, etc.)**

## Metrics Planning & Reporting                         H. K. Ramapriyan

**Community Workshop, Feb 5 – 7, 2002**
- ~15 individuals attended breakout session
- Representatives from HQ, DAACs, ESIPs and SEEDS team
- 3 new participants added to team, all 3 participate in weekly telecons:
- Don Collins, Manager, JPL PODAAC
- Frank Lindsay, Manager, Global Land Cover Facility ESIP-2, University of Maryland
- Hank Wolf, Assistant Director of CEOSR and Member, Seasonal to Inter-annual ESIP-2, George Mason University
- Reinforced multiple viewpoints for metrics planning and reporting.  This will provide a basic framework for the study since it defines the relationships among the various “classes” of participants.
- Currently looking at 5 classes for SEEDS:
- NASA HQ, End Users, NASA (and Non-NASA) project sponsors, Data Providers, and Provider internal organizations.
- Accountability and metrics management, including specification of “value” and “success” measures all depend on what class you are considering.
- General consensus was that current metrics only partially reflect a provider’s performance, e.g., measures of utilization of data and products by the science community are currently not reflected in metrics collection.  The solution to this is not easy.

**Status**

## Metrics Planning & Reporting                         H. K. Ramapriyan

**Preliminary Results from Metrics Survey**
**As of June 12, 2002, eighteen Activities (of thirty solicited) have responded:**
- 7 Data Centers (LP DAAC, PO.DAAC, ORNL DAAC, GES DAAC, NSSDC, GHRC, SEDAC)
- 1 Science Data Processing Center (AMSR-E SIPS)
- 5 Science Data Centers (Type 2 ESIPS: GLCF, SIESIP, EOS-WEBSTER, OceanESIP, PM-ESIP)
- 4 Applications Activities (Type 3 ESIPS: EDDC, TerraSIP, BASIC, TERC)
- 1 ‘Infrastructure’ Activity (DODS, also an ESIP)
**Responding Activities operate under several funding mechanisms:**
- Contracts, Cooperative Agreements, Grants, NASA Internal Processes, Inter-Agency Agreements
**Responses from the eighteen Activities were mostly complete, in some cases considerable detail was provided.**
- Discussion of metrics - most useful metrics, problems with metrics, suggestions for changes to metrics – provided in detail.
**The mix of activity types and depth of information provided allow some tentative conclusions to be drawn (next charts) these will be updated as more responses are received.**
**Preliminary study report (includes survey results) – June 30, 2002**

## Metrics Planning & Reporting                         H. K. Ramapriyan

**Preliminary Conclusions:**
**1. The current use of administrative and funding mechanisms is mostly appropriate and mostly successful.**
- Most Activities reported satisfaction, most felt they had the needed authority to meet their responsibilities, all reported no difficulties in resolving conflicts with multiple sponsors.
- No systemic problems seen, but some site specific problems:
**Two activities seemed to be operating under an inappropriate mechanism – operational science processing center and data center under cooperative agreements instead of contracts.**
**Activities cited difficulties with their funding mechanism (e.g., conflict with their host institution’s NASA funding mechanism, promptness of NASA payments, prohibition from subcontracting to a private company).**
**Activities cited what they considered to be restrictions on their authority over their work (e.g. prohibition from distributing near real-time data to users, long lead times for approval of foreign travel and restrictions on equipment purchase authority).**
- Some considered effort in collecting and reporting metrics to be significant and an “unfunded mandate” – including responding to new requirements beyond initial sets.

## Metrics Planning & Reporting                         H. K. Ramapriyan

**Preliminary Conclusions, Continued:**
**2. Sponsor required metrics are useful, but miss user satisfaction and value to users.**
- Thirteen of the fourteen responding activities are ESE-funded DAACs or ESIPs who respond to NASA HQ and/or ESDIS Project requirements for metrics.
- Consensus that the statistics do not measure success as users see it – easy access to readily usable, well-supported data, products, and services.
- Consensus that statistics do not measure value of data and services to users.
- One exception – ‘nuggets’ collected and provided by ESIPs – seen by ESIPs as best indication of user satisfaction.
- Some remedies were suggested, e.g. citations in peer reviewed literature (now regarded as a key measure by one ESE activity and the one non-ESE responder – NSSDC), growth of user base to include new types of users.
**3. Possible role for ‘SEEDS Office’ to improve measure of user satisfaction**
- Develop cross-ESE (DAACs, ESIPs, etc.) systematic search for citations, data use in scientific, policy, popular literature – central effort more cost effective and objective.
- Search results would document use, in advancing ESE science and applications program, scientific contributions, aid to policy decisions.
- Fund ESE activities to assemble special collections of scientific papers that utilize their data and products.

## Metrics Planning & Reporting                         H. K. Ramapriyan

**Preliminary Conclusions, Continued:**
**4. The topic of Accountability needs study and policy review.**
- Responses to accountability questions (covering IT security, user privacy, etc.) revealed a wide disparity between accountability requirements and reporting between the data centers and other activities.
- Data centers – strict requirements from sponsor, required reporting.
- Others – Seem to have virtually no requirements or reporting – performance on IT security, user privacy dependent on host institution practice and activities’ own judgment.
- What should SEEDS-era policies be? Governance policies need to be established - “one size does not fit all”.
**5. Accountability for data stewardship – a special case needing study:**
- Responses indicate that Activities, especially data centers, are aware of responsibility for data stewardship, and that User Working Groups are concerned with their performance.
- Responses report no sponsor guidelines or requirements or reporting on data stewardship beyond noting that some routine metrics are relevant.
- Review of data management planning, data stewardship practices, and metrics that would measure success or detect problems seems needed.

## Metrics Planning & Reporting                         H. K. Ramapriyan

**Governance**
**Goal: Identify options for governance structures**
**Relationship to ESE mission role and responsibilities**
**Impact on metrics planning and reporting**
**Given a set of three possible coexisting, overlapping governance structures (see next slide)...**
**What other structures are possible/desirable? **
**What other structures have been tried elsewhere (i.e., other than NASA ESE environment)?**
**What are the criteria to determine appropriateness of governance structure for a given activity? Criticality – examples of criteria:**
**Budget Thresholds, i.e. resource commitment or resource at risk**
**Consequences of Failure (Ability/Cost/Time to recover, Embarrassment factor)**
**What are the levels of control appropriate to different activities? **
**How do we ensure that the responsibility and authority are delegated to the proper level commensurate with the types of activities?**
**Who chooses the levels of control and when should it be determined?****How should control be applied?**
**What, besides metrics planning and reporting, is needed to ensure accountability?**
**How do we ensure delegation to lowest appropriate level? **

## Metrics Planning & Reporting                         H. K. Ramapriyan

**Three Possible ESE Coexisting Governance Structures**
**ESE Program Components – Data and Information Services**
- One Program Office – must see all parts of the program, ensure program integrity and that over all program goals are formulated and met.
- Coordinating Activity – Needed in cases where operational coordination across operating field activities is required for success of a defined portion of the ESE program (e.g. Terra/Aqua data flow, production: ground stations – EDOS – SIPS - DAACs).
- Operating Field Activities – Various sub-types, e.g. produce and distribute products on an operational basis, sometimes with critical dependencies (e.g. SIPS, DAACs)
- Research / Experimental Activities – Various sub-types, no critical dependencies, inherently risky by choice, successes may propagate to operational domain (e.g. Type 2 ESIPs).
**Three possible structures that would co-exist:**
- Program Office – Coordinating Activity – Operating Field Activity
- Program Office – Operating Field Activity
- Program Office – Research / Experimental Activity
- Note: An institution can host / serve as an Operating Field Activity(s) and Research/Experimental Activity(s) – so Governance structures coexist and can overlap.

## Metrics Planning & Reporting                         H. K. Ramapriyan

**“****Metrics” Breakout Session**
**Metrics planning and reporting - process questions**
- Who establishes the "rules of the game", and how?
- What are the processes to set up agreements among partners: peer-to-peer and performer-to-sponsor?
- How do you assure that each of the participants is meeting the commitments (schedule, budget, technical, etc.)?
- What is the reporting chain?
- What are the performance metrics?
- How do you publicize your accomplishments?
**Governance - process questions**
- As in previous charts

## Metrics Planning & Reporting                         H. K. Ramapriyan

**Schedule**
**Task Start – December 2001**
**Draft questions to send to sponsors and implementing organizations – January 4, 2002 ****(*****completed)***
**Community Workshop - February 5-7, 2002 *****(completed)***
**Refine questions and “visit list” - February 15, 2002 *****(completed)***
**Distribute questionnaires to “visit list” - March 8, 2002 *****(completed)***
**Interim report on aggregated survey results – April 15, 2002 *****(completed)***
**Obtain responses and conduct follow-up interviews – March – May 2002**
**Preliminary study report (includes survey results) – June 30, 2002**
**Further contacts with sponsors and implementing organizations as needed - July - October 2002**
**Recommendations to  ESE about SEEDS governance, metrics planning and reporting mechanisms - December 2002**

## Metrics Planning & Reporting                         H. K. Ramapriyan

**Initial Visit List**