#### Metrics Planning and Reporting (MPAR) Breakout Session -- 6/19/02

**H. K. Ramapriyan**

Attendees: Bud Booth (GST -- SEEDSFT MPAR team), Howard Burrows (AUSI --
ESIP), Don Collins (JPL -- DAAC -- SEEDSFT MPAR Team, breakout session
chair), Helen Conover (UAH -- SIPS, ESIP), Ruth Duerr (NSIDC -- DAAC),
Yonsook Enloe (SGT - SEEDSFT), Michael Goodman (MSFC/GHRC -- SIPS,
ESIP), Steve Kempler (GSFC -- DAAC), Ken McDonald (GSFC -- SEEDSFT),
Rich McKinney (EDC -- DAAC), H. Ramapriyan (GSFC -- SEEDSFT MPAR team
lead), Michael Singer (Harris), Laura Slaughter (UMD -- ESIP), C. Sung
(Consultant, U of Phoenix), Vince Troisi (NSIDC -- DAAC), Larry Voorhees
(ORNL -- DAAC), Hank Wolf (GMU -- ESIP, SEEDSFT MPAR team), Jingli Yang
(ERT -- SEEDSFT)

The breakout session was temporally divided into two parts -- 1. MPAR
and 2. Governance

**1. MPAR**

**The following questions were put forward initially to guide the
discussions even though the group was not constrained by them as seen
below:**

-   Who establishes the rules of the game and how

-   What are the processes to set up agreements among partners: peer to
    peer and performer to sponsor?

-   How do you assure that each of the participants is meeting the
    commitments (schedule, budget, technical, etc.)?

-   What is the reporting chain?

-   What are the performance metrics?

-   How do you publicize your accomplishments?

## **Discussion**

-   We need a clear statement of goals of SEEDS, and goals for the Earth
    Science Enterprise's Data Systems in order to define appropriate
    metrics. Metrics should indicate how we are performing in achieving
    those goals

-   We have science goals of ESE expressed in terms of the science
    questions, we have ESE's applications strategy that gives goals in
    applications. Data system goals should be expressed as enabling and
    supporting the science and applications' goals

-   What defines success?

    -   There are different levels at which success is defined. E.g.,
        meeting overall requirements of ESE (expressed as GPRA metrics)
        vs. detailed metrics for a given implementing organization to
        improve efficiency

    -   What we are trying to do is report success up the chain. Trying
        to make sure there is future funding for the organization's
        activities.

    -   Used in a management sense: judging performance against other
        organizations.

    -   GPRA must be reported. In one case you are in sell mode, GPRA
        metrics may not reflect your successes. Current GPRA does not
        reflect success with users

-   We need to think about what we have to measure: Are we measuring
    whether there is regular use? best practices? What should the
    metrics be?

-   NASA should indicate "up front", i.e., in the procurement documents,
    that metrics' reporting is a mandatory requirement. It is difficult
    to be specific about metrics when we have a heterogeneous set of
    bidders proposing to perform very different sets of functions. But,
    each participant should show how their metrics/performance will
    contribute to demonstrated achievement of higher level goals, where
    possible.

-   Measuring data volumes, number of users, number of distinct data
    products, etc. is easy, but does not provide a complete picture of
    the value of the program to the community

-   **Output** metrics can be measured easily; **outcome** metrics are
    difficult to measure, except as anecdotal evidence, but give a
    better picture of value to the community

-   We need a baseline with respect to ESE's program (e.g., baseline for
    disaster management, global warming, etc.). A performing entity
    should get "points" for recognizing utility of data for a new
    science problem

-   We (providers of data and information systems and services) are
    supporting science. We can contribute and we get extra points when
    we make a contribution such as defining/closing an information (what
    missions we need)

-   Metrics should measure public use, public benefit and potential
    public benefit.

-   John Townshend, in his presentation at the Plenary session,
    mentioned the MODIS (Rapid Fire System's) success. That was an
    example of a quick resolution of a practical problem. But success is
    reported as an anecdote. How do we come up with metrics that can
    identify the aspects of this kind of success?

-   We should define the specific areas of performance and the metrics
    that apply

-   We need to account for the fact that we are providing the data
    value-adding providers and have measures of the benefits that in
    many cases can only be derived through them

-   A measure that some data centers use is the number of journal
    articles that cite data from the data center that were used in the
    published research. This is difficult to count accurately

-   Auditing metrics is an issue. Metrics are claims of success, and
    they should be independently verifiable, usually through end users
    of the information.

-   Since it is difficult to measure outcome/value, it is useful to get
    individual opinions on 2 or 3 key measures of value that apply to
    the breakout session attendees' activities. Following is a
    compilation of such measures:

***Value measures:***

Rama

-   Of the data we sent to the science/applications users how much was
    used for addressing science questions/applications?

-   How much data and how many different datasets were used to produce
    science publications?

-   What is the time lag between data collection and its utilization for
    science/application?

-   What is the number of different applications communities supported?

-   What is the number of value-added providers supported and what is
    the "user fan-out factor"?

Rich

-   Emergency response program -- ASTER Emergency Expedited data in
    support of FEMA -- addressing national disasters -- Annual dollar
    value placed on time saved, lives saved

-   Large customer in Australia adding value -- DEMs, decorrelation
    stretch, etc. -- we could determine what his sales are, etc.

-   Earthsat data buy -- Landsat mosaic data -- ordering landsat-7 and
    mosaicking entire globe -- how does it benefit science, regional
    planning, etc.

Michael

-   How many graduate degrees came out of the datasets we sent out?

-   Measure efficiency of reducing data volume

Ruth

-   Market share served (percentage of target user community)

-   Requests to become an archive (or other core areas) -- unsolicited
    or solicited new business

Sung

-   Availability -- response -time and cost

-   Integrity of data

-   Technical support

-   City planning example -- how many cities are using the data; how
    extensively are they using data; can get this information from
    cities and quantify. It is possible to generalize this into other
    areas (Collins -- a similar example if what George Seielstad is
    doing in ND; difficult to get the numbers)

Howard

Public good

-   Lives saved, GNP increase, etc. -- it is difficult to measure our
    contribution; experts in field agree that data provided by us

-   Energy, health, public security -- number of possible answers and
    amount of trust in the answers -- new increases that improve trust
    of an NAS panel

Larry

-   Number of new users in a year per FTE staff

-   ORNL Data are oriented to the research scientist -- number of
    peer-reviewed publications resulting from data

-   Use of data in classroom and teaching environments

-   Unsolicited new business

-   \[It is important to understand how a given metric is used -- e.g.,
    number of products sent out may have different definitions for
    different data centers\]

Hank

-   Saved time in getting access to subset of data in the right format,
    correctly binned

-   Reduction of search time to find where data are stored covering a
    specific geographic area; how quickly you can get access to data

-   Time saved in data subsetting and going to the analysis step

-   \[All of us have to reach out to user community to figure out how
    well we are doing to get these measures\]

Laura

-   \[you don't know how important things are until they are gone; we
    heard a lot from users when our system was down\]

-   Nuggets categories are showing the same kinds of things we are
    talking about

*It is also to be noted that Vanessa Griffin is working the DAACs and
the Federation partners to discuss GPRA metrics NASA will provide
starting in FY 2004. The efforts from those discussions need to be
folded into this study.*

# 2. Governance 

**The following questions were put forward initially to guide the
discussions even though the group was not constrained by them as seen
below:**

-   Goal: Identify options for governance structures

```{=html}
<!-- -->
```
-   Relationship to ESE mission role and responsibilities

-   Impact on metrics planning and reporting

```{=html}
<!-- -->
```
-   Given a set of three possible coexisting, overlapping governance
    structures below

> Program Office -- Coordinating Activity -- Operating Field Activity
>
> Program Office -- Operating Field Activity
>
> Program Office -- Research / Experimental Activity

-   What other structures are possible/desirable?

-   What other structures have been tried elsewhere (i.e., other than
    NASA ESE environment)?

```{=html}
<!-- -->
```
-   What are the criteria to determine appropriateness of governance
    structure for a given activity? Criticality -- examples of criteria:

```{=html}
<!-- -->
```
-   Budget Thresholds, i.e. resource commitment or resource at risk

-   Consequences of Failure (Ability/Cost/Time to recover, Embarrassment
    factor)

```{=html}
<!-- -->
```
-   What are the levels of control appropriate to different activities?

```{=html}
<!-- -->
```
-   How do we ensure that the responsibility and authority are delegated
    to the proper level commensurate with the types of activities?

-   Who chooses the levels of control and when should it be determined?\
    How should control be applied?

-   What, besides metrics planning and reporting, is needed to ensure
    accountability?

```{=html}
<!-- -->
```
-   How do we ensure delegation to lowest appropriate level?

### Discussion

-   What are the activities governance has to look after? What is it
    that we want to govern? Where is coordination being done? What
    activities need coordination and what do not?

```{=html}
<!-- -->
```
-   What are the degrees of freedom? What kinds of functions/activities
    need what kind of governance?

```{=html}
<!-- -->
```
-   One aspect of governance is how funding is decided, priorities are
    worked, who allocates/shares the budget, etc.

```{=html}
<!-- -->
```
-   In the case of Space Station, the program office set philosophy,
    policy, schedule and strategy, and the prime contractor was to be
    responsible for implementation

```{=html}
<!-- -->
```
-   For science and applications functionality -- the Program Office
    should provide priorities and budget and let the activity run on its
    own depending on the proposing organization's initiative, etc.
    However, for EDOS-like function, where data loss is of serious
    consequence, the activity needs to be strictly governed

```{=html}
<!-- -->
```
-   Types of Functions -- data capture and ingest, higher level standard
    product generation and archiving, research product generation,
    applications development, technology development (e.g., standards)

```{=html}
<!-- -->
```
-   We need to look at categories of governance:

```{=html}
<!-- -->
```
-   Data Stewardship

-   Tracking Data

-   Data Capture

-   Data Ingest

-   Data Archive

-   Infrastructure

-   Interoperability

-   Standards

-   Security

-   Accountability

-   Metrics

-   Agreements (ICDs, etc.)

-   Rules of Engagement

-   Partners

-   New Development

-   OTHERS: Need to acquire governance functionalities that need to be
    included, surfaced from other Study Teams

```{=html}
<!-- -->
```
-   We should include Program Office in the Federation's "Petri dish"
    diagram.

-   Governance could be looked at as the glue that holds the next data
    system structure together

-   Governance: Light touch -- Works for low risk areas; encourages
    creativity. Heavy touch -- Works for higher risk areas. \[We need to
    understand the risks for which SEEDS is responsible or will be
    viewed as responsible for\]

```{=html}
<!-- -->
```
-   One of the criteria to be used in determining governance is how well
    we understand what is to be done. If a function/activity is well
    understood, central control works; in cases with greater ambiguity,
    we need lighter touch. If creativity wanted, go light touch; if
    on-time delivery is mandatory, "crack the whip".

```{=html}
<!-- -->
```
-   We need to be sure that there is autonomy for groups -- authority
    commensurate with responsibility

-   Production chains serving individual clumps of users need to be
    considered in assigning governance structures \[note discussion of
    need for interface control documents in data life cycle breakout
    session\]

-   As federation grows we will need to have appropriate
    representational structures (since we will not be able to have each
    organization or individual participate in the governance)

-   SEEDS should play a role in getting all the partners to work
    together

```{=html}
<!-- -->
```
-   Where does SEEDS invest dollars for technology growth? Should this
    be done top down or bottoms up?