# Presentation: 536467

## FY06 REASoN Metrics Review Greg Hunolt, SGTES-DSWG / MPAR-WGNovember 14-16, 2006

## Current Overall FY06 REASoN Metrics Status

**As of November 7:**
- Active REASoNs in progress – 39 (last year 41); one active REASoN was exempted from metrics reporting, so 38 is the base.
- REASoNs reporting at least some metrics – 34 (last year 30);
- REASoNs whose metrics are complete with no or minor problems – 29 (2 others will get complete for FY06);
- 17 REASoNs answered question about Study Manager feedback / discussion:
- 7 REASoNs report discussing metrics with study manager (last year 3);
- 10 REASoNs report no feedback / discussion (last year 5).

## Metric by Metric Status

| Metric | Total | Good | OK | N/A | Not Rep |
| --- | --- | --- | --- | --- | --- |
| 1 – Distinct Users | 34 | 32 (20) | 2 | 0 | 0 |
| 2 – Users by Class | 34 | 25 (10) | 6 | 0 | 3 |
| 3 – Prods Delivered | 34 | 31 (12) | 1 | 0 | 2 |
| 4 – Prod Types | 34 | 31 (14) | 1 | 0 | 2 |
| 5 – Vol Delivered | 34 | 29 (21) | 1 | 2 | 2 |
| 6 – Vol Available | 34 | 28 (16) | 1 | 2 | 3 |
| 7 - Response | 34 | 5 (10) | 22 | 3 | 4 |
| 8, and/or 9, a/or 10 | 34 | 30 (27) | 1 | 0 | 3 |
| 8 – Science Areas |  | 25  (23) |  |  |  |
| 9 – Applic Areas |  | 19  (19) |  |  |  |
| 10 – Educat Areas |  | 16  (18) |  |  |  |

- Number of REASoN Projects (out of 34) reporting each metric in FY06...
- (FY05 number in parens) (as of November 6)

## Impact Metrics - Project Comments

_**Only four projects have entered Impact Metrics, for a total of nine (five by one project).**_
- “What are these, nuggets?” (From four REASoNs)
- “The reason we have not yet entered any impact metrics is the phase that our project is in...” (From two REASoNs)
- “Primary reason is lack of time -- too busy to take the time to do it right.” (From three REASoNs)
- “It may help to have a few more examples for people to look at (maybe one or two for each ESIP type).”

## Impact Metrics - Project Comments, Cont.

- “Our project provides quality climate data and innovative web tools.  We find it hard to come up with impact metrics...that is, we provide the staples for other folks to use in applications ... This is not our work, so is the impact metric appropriate for acknowledging the roll [our] data played?”
- “What is the best way to communicate other scientists’ work when they have used our data or services?”
- “The example impact metric presents the expectation (whether real or not) that the metric needs to be highly polished and journalistic.  We often feel we do not have the time or material (images or plots) to create such a product.  ...”

## Project Comments – What Happens to Our Metrics?

- “I think that I would like to get some more feedback on the impact of this metrics at NASA HQ - i.e. is it received positively, negatively, can it be linked into the decision-making process? Is there any quantifiable measure of impact? As I stated above, I believe that they are relevant to us, but it would also be good to see how it fits into the bigger picture at NASA HQ.”
- “I would be curious to get some feedback as to what has actually been done with the metrics that we've reported.”

## Project Comments – Feedback / Discussion with Study Managers

**“**No. Not regarding the formal metrics. Although we regularly share our successes informally and get feedback from that.”
- “No. In fact I have had no feedback from [...], and this is becoming a problem.”
- “Yes, I present to [...] on our REASoN work at his site reviews. ... I also send him the reports, together with Lucia and Martha. We also discuss informally.”
- “I find that my reporting to my study manager is done outside of the metrics in the form of additional reports that I submit.”
- “... we've gotten into the habit of sharing those directly with program managers because we've never had anyone respond to what we put in the formal metrics. We get great feedback from the program managers as a result of directly informing them via email.”

## Project Comments – Feedback / Discussion with Study Managers, Cont.

- “[We] been in contact with our study manager, but they have not discussed metrics.”
- “Metric reporting and benchmarking have been frequently discussed with [...] PI.”
- “We presented an annual review to [study manager] in November 2005.  We discussed our thorough completion of the metrics...”
- “Yes, we have spoken with our program manager regarding metrics during our annual review.   He encouraged us to continue as we have to meet the cooperative agreement requirements.  No other comment from him regarding metrics.”
- “We have had regular feedback from our study managers concerning our project over the past few years and they have had no concerns about the metrics reporting.”

## Project Comments – on Metrics / Process

- “The one comment that I would have on the performance metrics is related to the questions 8, 9, and 10 (dealing with the SMD support). Rather than us having to download the PDF documents or in the latest version being redirected to the NASA SMD web site ... summarize it one paragraph and have only this paragraph show up. ... I think the SMD metrics are very important as they link our work with NASA's strategic plan and show the relevance.”
- “These are fine metrics, and the input process is very easy.”

## FY06 REASoN Metrics Review, Conclusions

**Most projects (31 out of 38) are making a good effort to report the metrics, in some cases despite the constrictions of the old baseline.**
**REASoN Project metrics for FY06 are in much better shape than in FY05.**
**There is no indication that study managers access the website, look at project metrics, and provide feedback on metrics to the project - **
**Based on project comments;**
**Based on the presence of a bug which caused the metrics to be garbled in a number of months... there would likely have been complaints.**

## Backup

## Metric 1 – Distinct Users

**FY06 Baseline Definition:** The number of distinct individual users (based on non-duplicated IP addresses) who request and/or receive products, services and/or other information during the reporting period.
**Purpose:** To measure the size of the activity’s user community, to be assessed in the context of its ESE role.
**Website Question:** Please enter the number of individual users (based on non-duplicated IP addresses) who request and/or receive data and/or other information during the reporting period.

## Metric 2 – Distinct Users by Class

**Baseline Definition:** Classes of users who obtain products and services from the project.  The metric will show the relative proportion of users accessing data and services from:
- a) first-tier domains: .com, .edu, .gov, .net, .mil, .org, summary of foreign countries, and  unresolved ,
- b) second-tier domains, such as “nasa.gov”, “unm.edu”, etc.
**Purpose:** To measure the types of users served by the activity, to be assessed in the context of its ESE role.
**Website Question:** Please enter the number of users, by relative proportion of total users, who obtain products and services from the following classes:
- 2.1 - Foreign Domains, 2.2 – Unresolved
-      Please enter the number of users who obtain products and services from your project by the following second tier domains:
- 2.3 - nasa.gov, 2.4 - usgs.gov, 2.5 - noaa.gov, 2.6 - epa.gov, 2.7 - nga.mil, 2.8 - usda.gov, 2.9 - dot.gov, 2.10 - other federal domains, 2.11 - state government domains, 2.12 - municipal and county govt domains, 2.13 - specific other domain 1 (e.g. umd.edu) specify, 2.14 - specific other domain 2 (e.g. umd.edu) specify, 2.15 – all .gov, 2.16 – all .mil, 2.17 – all.us, 2.18 – all.edu, 2.19 – all.org, 2.20, all.com, 2.21 – all .net

## Metric 4 – Product Types Produced & Maintained

**Baseline Definition:** A product type refers to a collection of ‘products’ of the same type such as “sea surface temperature” products.  The project may add many or few product types through time but these should be tracked independent of the number of ‘products’ delivered. (This metric is not expected to change frequently and may not require updates on a monthly basis).
**Purpose:** The count of product types produced is a useful measure because of the effort by the activity required to develop and support each of its product types. A REASoN’s values for this metric are to be assessed in the context of its ESE role. A particular set of values for these metrics might be much smaller for one activity than another activity, but in each case could represent excellent performance, given the particular ESE role of each activity.
**Website Question:** Please enter the number of distinct product types produced and maintained by your project. The project may add many or few product types through time but these should be tracked independent of the number of products delivered. (This metric is not expected to change frequently and may not require updates on a monthly basis).

## Metric 3 – Products Delivered

**Baseline Definition:** The number of separately cataloged and ordered data or information products delivered to users during the reporting period (by project-defined product ID).  A ‘product’ may consist of a number of items or files that comprise a single item in a product catalog or inventory; our intent is to capture the user view of the products provided by the project.
**Purpose:** To measure, in conjunction with items 4, 5, 6, and 7, the data and information produced and distributed by the activity, to be assessed in the context of its ESE role. A particular set of values for these metrics might be much smaller for one activity than another activity, but in each case could represent excellent performance, given the particular ESE role of each activity. The count of products delivered is a useful measure given the user oriented definition of a ‘product’ that is independent of how the product is constituted or how large it is.
**Website Question:** Please enter the number of separately cataloged and ordered data or information products delivered to users during the reporting period.

## Metric 5 – Volume of Data Distributed

**Baseline Definition:** The volume of data and/or data products distributed to users during the reporting period (in GB or TB as appropriate).
**Purpose:** The volume distributed is a useful output measure but one which depends heavily on the particular types of data an activity produces and distributes and must be assessed in the context of the activity’s ESE role and data it works with. See note in metric 3.
**Website Question:** Please enter the volume of data and/or data products distributed to users during the reporting period (specify in MB, GB or TB as appropriate).

## Metric 6 – Volume of Data Available

**Baseline Definition:** The total cumulative volume, as of the end of the reporting period, of data and products held by the project and available to researchers and other users (GB or TB).  This number can include data that is not on-line but is available through other means.
**Purpose:** The cumulative volume available for users provides a measure of the total resource for users that the activity creates. See note in metric 4.
**Website Question:** Please enter the total cumulative volume, as of the end of the reporting period, of data and products held by your project and available to researchers and other users (specify in MB, GB or TB). This number can include data that is not on-line but is available through other means.

## Metric 7 – Delivery Time of Products to Users

**Baseline Definition:** Response time for filling user requests during the reporting period. Averaged and standard deviation summary times are to be collected for both electronic (including subscription services) and physical hard media transfers.
**Purpose:** The delivery times for electronic and/or media transfers to users is a measure of the effectiveness of the activity’s service. See note in metric 3.
**Website Question:** Please enter the averaged response time for filling user requests during the reporting period. These are to be collected for both:
- 7.1.1 electronic transfers (including subscription services):  (column 7.1 on data worksheet)
- 7.1.2 physical hard media transfers  (column 7.2 on data worksheet)
- 7.2 Please enter the standard deviation of summary times for filling user requests during the reporting period. These are to be collected for both:
- 7.2.1 electronic transfers (including subscription services):  (column 7.3 on data worksheet)
- 7.2.2 physical hard media transfers:   (column 7.4 on data worksheet)

## Metric 8 – Support for Science Focus Areas

**Baseline Definition:** The REASoN projects will include a quantitative summary of the data products supporting one or more of NASA’s science focus areas, and report any changes at the next monthly metrics submission.  The focus areas are: weather, climate change and variability, atmospheric composition, water and energy cycle, Earth surface and interior, and carbon cycle and ecosystems.
**Purpose:** To enable the ESE program office to determine which NASA Science Mission Directorate's Science Focus Areas are supported by the activity, and to assess how the data products provided by the activity relate to that support.
**Website Question:** Please list any products or services that support NASA Science Mission Directorate's Science Focus Areas and how many users were so provided within the last reporting period, and include which of the NASA categories they most correctly fit (may be multiple). This category is subject to definition between the REASoN management and their NASA program manager.
- Categories from NASA Science Mission Directorate's Science Focus Areas:
- 8.1 Weather, 8.2 Climate change and variability, 8.3 Atmospheric composition, 8.4 Water and energy cycle, 8.5 Earth surface and interior, 8.6 Carbon cycle and ecosystems, 8.7 Other- Please explain.

## Metric 9 – Support for Applications Areas

**Baseline Definition:** The REASoN projects will include a quantitative summary of the data products supporting one or more of NASA’s Applications, and report any changes at the next monthly metrics submission.  The 12 applications areas are:  agricultural efficiency, air quality, aviation safety, carbon management, coastal management, ecosystems, disaster preparedness, energy forecasting, homeland security, invasive species, public health, and water management.
**Purpose:** To enable the ESE program office to determine which NASA Science Mission Directorate's Applications of National Importance are supported by the activity, and to assess how the data products provided by the activity relate to that support.
**Website Question:** Please list any products or services that support NASA Science Mission Directorate's Applications of National Importance and how many users were so provided within the last reporting period, and include which of the NASA categories they most correctly fit (may be multiple). This category is subject to definition between the REASoN management and their NASA program manager.
- Categories from NASA Science Mission Directorate's Application of National Importance:
- 9.1 Agricultural efficiency, 9.2 Air quality, 9.3 Aviation safety, 9.4 Carbon management, 9.5 Coastal management, 9.6 Ecological forecasting, 9.7 Disaster management, 9.8 Energy management, 9.9 Homeland security, 9.10 Invasive species, 9.11 Public health, 9.12 Water management, 9.13 Other - Please explain.

## Metric 10 – Support for Education Initiatives

**Baseline Definition:** In partnership with the Study Manager the REASoN project will submit data pertaining to the adoption and use of educational products by noted audience categories (to be determined by project and study manager).  These groups can include higher education, K-12, museums, informal education, and others as appropriate.
**Purpose:** To enable the ESE program office to assess support provided by the activity to NASA Science Mission Directorate's education initiatives, by indicating use by education user groups of the activity’s products and services.
**Website Question:** Please list any products or services provided to educators within the last reporting period, and include which of the NASA education categories they most correctly fit (may be multiple). This category is subject to definition between the REASoN management and their NASA program manager.
- NASA Earth Science Education Plan:
- 10.1 Elementary and Secondary Education, 10.2 Higher Education, 10.3 Underrepresented and Underserved, 10.4 e-Education, 10.5 Informal Education, 10.6 Other- Please explain.