# Presentation: 083318

##  Metrics BaselineGreg Hunolt, SGTES-DSWG / MPAR-WGOctober 21-23, 2008

## Metrics Mission Statement

- To measure the success of each project in meeting its stated goals and objectives, to show the role and contribution of each project to the NASA science, application, and education programs, and to enable an overall assessment of the success of programs such as REASoN / ACCESS / MEaSUREs and their contribution to NASA’s goals.

## Metrics Overview

**Programmatic Role Metrics: **Characterize the role of the project within the NASA science, applications, and/or education programs by indicating program areas the project supports.
**Products and Services Metrics: **Measure the number of types of products and/or services provided by a project, with data volumes as applicable.
**Common Metrics:** Reported by most if not all projects, will be overall measures with sufficient cross-project commonality to allow assessment of the REASoN / ACCESS (etc.) program(s) as a whole, and will not be used as comparative measures of project performance.
**Project-Specific Metrics: **Projects may add one or more project-specific metrics, defined by each Project as best measures of its performance against its objectives.
**Reporting: **Both common metrics and project-specific metrics will be reported monthly unless otherwise agreed between a project and its study manager.
**Impact  Metrics: ** Specific success stories that provide an example (s) of how the project’s products / services  have directly benefited a user , organization, or activity it supports.

## Common Metric #1, Distinct Users

**Purpose:** To measure the size of the activity’s user community, to be assessed in the context of its NASA program role.
**Website Question:** Please enter the count of individuals who, by any means, request and receive or in some other way use products, services and/or other information during the reporting period.

| Metric | Definition |
| --- | --- |
| Number of Distinct Users | The number of distinct individual users who, by any means, request and receive or in some other way use products, services and/or other information during the reporting period. |

## Common Metric #2, Distinct Users by Class 

**Purpose:** To measure the types of users served by the activity, to be assessed in the context of its NASA program role.
**Website Question:** Please enter the number of users who obtain products and services from your project by the following classes:

| Metric | Definition |
| --- | --- |
| Characterization of Distinct Users Requesting Products and Information | Classes of users who obtain products and services from the project.  The metric will show the numbers of distinct users accessing data and services from classes identified by internet domains or other identification that can be related to domain classes: from a) first-tier domains:  [...], and b) second-tier domains, [...].
 |

## Common Metric #4, Product Types

**Purpose:** The count of product types produced is a useful measure because of the effort by the activity required to develop and support each of its product types. A REASoN’s values for this metric are to be assessed in the context of its NASA program role.
**Website Question:** Please enter the number of product types made available to users during the reporting period.

| Metric | Definition |
| --- | --- |
| Number of Distinct Product Types Produced and Maintained by Project | A product type refers to a collection of ‘products’ of the same type such as “sea surface temperature datasets”, tools or other capabilities, and/or types of separately available information products that a project such as a REASoN or ACCESS provides to its users. The intent is to capture the user view of the product types provided by the project, i.e. how the project’s product types are presented to the user. |

## Common Metric #3, Products Delivered

**Purpose:** To measure, in conjunction with items 4, 5, and 6, the data and information produced and distributed by the activity, to be assessed in the context of its NASA program role. A particular set of values for these metrics might be much smaller for one activity than another activity, but in each case could represent excellent performance, given the particular role of each activity. The count of products delivered is a useful measure given the user oriented definition of a ‘product’ that is independent of how the product is constituted or how large it is.
**Website Question:** Please enter the number of products provided to users during the reporting period.

| Metric | Definition |
| --- | --- |
| Number of Products Delivered to Users  | The number of separately cataloged and ordered, dynamically produced, or otherwise made available data and/or information products delivered, and/or tools provided to users during the reporting period by each project. 

A ‘product’ (an instance of a metric 4 product type) is an item (pre-prepared or produced on demand) that may be listed in a product catalog, inventory or menu, a publication, or a customized product produced by a project for a particular user. The intent is to capture the presentation to the user of the data products, information products and/or tools provided by the project.  |

## Common Metric #12, Service Types

**Purpose:** The count of service types produced is a useful measure because of the effort by the activity required to develop and support each of its service types. A REASoN’s values for this metric are to be assessed in the context of its NASA program role.
**Website Question:** Please enter the number of service types made available to users during the reporting period. This metric is experimental, and may be altered by request from NASA HQ.

| Metric | Definition |
| --- | --- |
| Number of Distinct Service Types Provided by Project | A service type refers to a separately accessible web service or other capability, and/or any other separately available service of any kind, that a REASoN / ACCESS Project provides to its users. The intent is to capture the user view of the services and capabilities provided by the project, i.e. how the project’s services are presented to the user. |

## Common Metric #11, Services Provided

**Purpose:** To measure, in conjunction with item 11the services performed by the activity, to be assessed in the context of its ESE role. A particular set of values for these metrics might be much smaller for one activity than another activity, but in each case could represent excellent performance, given the particular ESE role of each activity.
**Website Question:** Please enter the number of services provided to users during the reporting period. This metric is experimental, and may be altered by request from NASA HQ.

| Metric | Definition |
| --- | --- |
| Number of Services Provided to Users  | The number of services provided to users during the reporting period by a project such as a REASoN or ACCESS. A service provided is a session with, or invocation of, an on-line service or capability selectable from a general menu of available services or capabilities, or an instance of a service otherwise provided (e.g. an exhibit, workshop or presentation), or a customized service such as development of novel data, custom model runs, etc.).

The intent is to capture how the information services and capabilities provided by the project are presented to the user.  |

## Common Metric #5, Volume of Data Distributed 

**Purpose:** The volume distributed is a useful output measure but one which depends heavily on the particular types of data an activity produces and distributes and must be assessed in the context of the activity’s role and data it works with. See note in metric 3.
**Website Question:** Please enter the volume of data and/or data and/or information provided as web downloads or hard media or otherwise distributed to users during the reporting period (in MB, GB or TB as appropriate, to three significant digits precision, e.g. "10.2 GB" as opposed to "10,186 MB"). If data is available but none was distributed then enter "0" in the entry box and explain in the comment box. If this Project has none available, then enter "na" in the entry box and 'not applicable' in the comment box.

| Metric | Definition |
| --- | --- |
| Volume of Data Distributed  | The volume of data and/or data products and/or information provided as web downloads or otherwise distributed to users during the reporting period (in MB, GB or TB as appropriate, to three significant digits precision, e.g. “10.2 GB” as opposed to “10,186 MB” ). 

For some projects, not distributing digital data, the proper answer is ‘not applicable’.  |

## Common Metric #6, Data Volume Available 

**Purpose:** The cumulative volume available for users provides a measure of the total resource for users that the activity creates. See note in metric 4.
**Website Question:** Please enter the total cumulative volume, as of the end of the reporting period, of data and held by the project and available to researchers and other users (MB, GB or TB to three significant digits). This number can include data that are not on-line but are available through other means. If paper products, enter as pages (pp) or publications (pb). If this Project has no data available, then enter "na" in the entry box and 'not applicable' in the comment box.

| Metric | Definition |
| --- | --- |
| Total Volume of Data Available for Research and Other Uses  | The total cumulative volume, as of the end of the reporting period, of data and products held by the project and available to researchers and other users (MB, GB or TB to three significant digits).  This number can include data that are not on-line but are available through other means. 

For some projects, not holding digital data, the proper answer is ‘not applicable’.  |

## Programmatic Metric #8, Support for Science Focus Areas

**Baseline Definition:** The REASoN projects will include a quantitative summary of the data products supporting one or more of NASA’s science focus areas, and report any changes at the next monthly metrics submission.  The focus areas are: weather, climate change and variability, atmospheric composition, water and energy cycle, Earth surface and interior, and carbon cycle and ecosystems.
**Purpose:** To enable the ESE program office to determine which NASA Science Mission Directorate's Science Focus Areas are supported by the activity, and to assess how the data products provided by the activity relate to that support.
**Website Question:** Please list any products or services that support NASA Science Mission Directorate's Science Focus Areas and how many users were so provided within the last reporting period, and include which of the NASA categories they most correctly fit (may be multiple). This category is subject to definition between the REASoN management and their NASA program manager.
- Categories from NASA Science Mission Directorate's Science Focus Areas:
- 8.1 Weather, 8.2 Climate change and variability, 8.3 Atmospheric composition, 8.4 Water and energy cycle, 8.5 Earth surface and interior, 8.6 Carbon cycle and ecosystems, 8.7 Other- Please explain.

## Programmatic  Metric #9, Support for Applications Areas

**Baseline Definition:** The REASoN projects will include a quantitative summary of the data products supporting one or more of NASA’s Applications, and report any changes at the next monthly metrics submission.  The 12 applications areas are:  agricultural efficiency, air quality, aviation safety, carbon management, coastal management, ecosystems, disaster preparedness, energy forecasting, homeland security, invasive species, public health, and water management.
**Purpose:** To enable the ESE program office to determine which NASA Science Mission Directorate's Applications of National Importance are supported by the activity, and to assess how the data products provided by the activity relate to that support.
**Website Question:** Please list any products or services that support NASA Science Mission Directorate's Applications of National Importance and how many users were so provided within the last reporting period, and include which of the NASA categories they most correctly fit (may be multiple). This category is subject to definition between the REASoN management and their NASA program manager.
- Categories from NASA Science Mission Directorate's Application of National Importance:
- 9.1 Agricultural efficiency, 9.2 Air quality, 9.3 Aviation safety, 9.4 Carbon management, 9.5 Coastal management, 9.6 Ecological forecasting, 9.7 Disaster management, 9.8 Energy management, 9.9 Homeland security, 9.10 Invasive species, 9.11 Public health, 9.12 Water management, 9.13 Other - Please explain.

## Programmatic  Metric #10, Support for Education Initiatives

**Baseline Definition:** In partnership with the Study Manager the REASoN project will submit data pertaining to the adoption and use of educational products by noted audience categories (to be determined by project and study manager).  These groups can include higher education, K-12, museums, informal education, and others as appropriate.
**Purpose:** To enable the ESE program office to assess support provided by the activity to NASA Science Mission Directorate's education initiatives, by indicating use by education user groups of the activity’s products and services.
**Website Question:** Please list any products or services provided to educators within the last reporting period, and include which of the NASA education categories they most correctly fit (may be multiple). This category is subject to definition between the REASoN management and their NASA program manager.
- NASA Earth Science Education Plan:
- 10.1 Elementary and Secondary Education, 10.2 Higher Education, 10.3 Underrepresented and Underserved, 10.4 e-Education, 10.5 Informal Education, 10.6 Other- Please explain.

## Impact Metrics – Success Stories

**Impact Metrics Include:**
**A one page narrative describing the success story, i.e. how the project’s products / services  have directly benefited a user, organization, or activity the project supports.**
**The narrative might  include:**
**The name of the product and/or service, in the title if possible. **
**The direct benefit of using the product or service, the impact, in a prominent manner; in the title if possible. **
**Any collaboration with local groups, NGO's, businesses, or federal agencies. **
**How working with NASA helped make this happen. **
**A JPEG image or graphic that provides a visual complement to the narrative description.**

## Impact Metric:

**DISCOVER REASoN,**
**PI: Frank Wentz**
**May, 2008**
**Distributed Information Services: Climate/Ocean Products and Visualizations for Earth Research**
**Provided daily near real-time data plots to Cirene international  oceanographic cruise.**

## Impact Metric:

**AMAPS ACCESS**
**PI: Amy Braverman**
**September, 2007**
**(w/ Large Picture)**
**Aerosol Management **
**and Processing System**
**AMAPS provided MISR data to multi-agency AIRNow air quality project.**

## Slide 18

## Impact Metric:

**DIAL ACCESS**
**PI: Bruce Carron**
**January, 2007**
**w/ Large Picture**
**Data and Information Layer: Enabling NASA Data-Rich User Software Application Development**
**Provided graphics support to Science Education Resource Center  (SERC) at Carleton College to demonstrate benefits of sharing graphic assets.**

## Slide 20