# Presentation: 484586

## Packaging Climate Products for Users

***Packaging Climate Products for Users***

***Holly C. Hartmann***
***Department of Hydrology and Water Resources University of Arizona hollyoregon@juno.com***

**Notes:**
Keep this title? 

List support? 

## Stakeholder Use of Climate Info & Forecasts

- Uninformed, mistaken about forecast interpretation
- Understand implications of “normal” vs. “EC” vs. “unknown” forecasts
- Use of forecasts limited by lack of demonstrated forecast skill

***Unique among stakeholders***
- Relevant forecast variables, regions (location & scale), seasons, lead times, performance characteristics
- Technical sophistication: base probabilities, distributions, statistics
- Role of of forecasts in decision making

***Common across many, but not all, stakeholders***
- Have difficulty distinguishing between “good” & “bad” products
- Have difficulty placing forecasts in historical context

**Stakeholder Use of Climate Info & Forecasts**

## Re-Interpreted Forecast Products Often Wrong

## Goals of Forecast Communication

**Elements to Consider**
- - Variable depicted: temperature, temperature anomaly, probability, probability anomaly
- - Two-category or three-category forecast
- - Forecast reference period
- - Probability ranges: colors, numeric scales
- - ‘No forecast’/’No skill’ situation
- - Appropriate spatial scale
- - Translation of information
- - Extension of information

**Interpretation should be: Correct, Reliable, Easy**

## Field Survey Methodology

**Surveys at Professional Society Meetings**
**- Common population**
**- Independent testing of different forecast formats**
**- Preclude ‘learning’ by respondents**
**- Sufficient sample size (across surveys)**

**Experimental Protocol**
**    ****- Multiple forecast products**
**    ****- One forecast per survey**
**    ****- Well mixed distribution of surveys**
**    ****- Attempt 100% distribution**
**    ****- Supplement with 1-on-1 interviews**

**Discussions at professional developmental training classes**

## Australian Bureau of Meteorology

- Minimum temperature 	          Two-category:median   1961-1990 reference period           Forecast everywhere

## Climate Prediction Center – National Weather Service

**Modifications**
- Bar Charts, Pie Charts, %
- No Skill, No Forecast

- Terciles
- 1971-2000 reference period
- Forecasts of opportunity

## IRI for Climate and Society

- Terciles
- 1969-1998 reference period
- Forecasts of opportunity

## Canada: Temperature

- Three maps = one forecast
- Terciles
- 1961-1990 reference period
- Forecast everywhere

## CPC Probability of Exceedance Outlook

## CPC Probability of Exceedance: Modified

## ‘User’ Perceptions

**Survey quotes:**
**It’s obviously intended for use by meteorologists and climatologists who use these every day.**
**Get rid of techie language that only makes sense to other forecasters.**
**Make it more understandable for the layperson.**
**I just couldn’t figure out what they basically tried to tell me. **
**Interview messages:**
**This product must not be applicable to my work, because otherwise I would understand it. **

## Findings: People, Product

**Respondents had high potential for considering climate variability.**

- water quality, stormwater management,
- watershed restoration, water supply planning,
- floodplain studies, groundwater,
- saltwater intrusion, watershed management,
- water economics, utility asset management,
- permitting and regulatory review,
- geomorphology, water law and policy,
- forest hydrology, infrastructure design, planning, operations

## Findings: People, Product

**Respondents had high potential for considering climate variability.**
**Forecast users did worse at answering correctly than non-users!**

## Findings: People, Product

**Respondents had high potential for considering climate variability.**
**Forecast users did worse at answering correctly than non-users!**
**No format was more effective than any other.       POE is notably ineffective.**

## Findings: People, Product

**Respondents had high potential for considering climate variability.**
**Forecast users did worse at answering correctly than non-users!**
**No format was more effective than any other.       POE is notably ineffective.**
**Our ‘fixes’ were no better – except simplifying the POE graph. Issue: introducing more complexity, without structure, persistent language problems.**

## Findings: Identification, Discernment

**Reference Period: people could *****identify *****the reference period, mostly... but we didn’t test *****interpretation*****.**
**Spatial scale: local to regional to global. IRI (grid) and CPC with regional outline were best.**

## Findings: Identification, Discernment

**Reference Period: people could *****identify *****the reference period, mostly... but we didn’t test *****interpretation*****.**
**Spatial scale: local to regional to global. IRI (grid) and CPC with regional outline were best.**
**People *****can***** discern probabilistic nature, with assistance (%, pie chart, bar chart, big title)**

## Findings: Identification, Discernment

**Reference Period: people could *****identify *****the reference period, mostly... but we didn’t test *****interpretation*****.**
**Spatial scale: local to regional to global. IRI (grid) and CPC with regional outline were best.**
**People *****can***** discern probabilistic nature, with assistance (%, pie chart, bar chart, big title)**
**Expressing probability: difficulty putting bounds on probability ranges. Concept of a range of values:  people indicated precise probability rather than a range of probabilities. **

## Findings: Translation, Extension

**‘****No Forecast’ language: No Skill seems to work better than Not Available, Equal Chances, or Climatological Probabilities.**

**Reluctance to translate information**

**Notes:**
When people had to translate rather than mimic language, answers were similar to chance. 

## Decision = F ( 1.00[Imean] ) 

***Forecast for “Normal Conditions”***
- Decision = F ( 1.00[Imean] )
***Forecast for “Equal Chances”***
- Decision = F ( .33[Iquantile<33 ]+ .33[Iquantile33-66]+ .33[Iquantile66>])
***Forecast for “Unknown Chances”***
- Decision = F (?[Iquantile<33 ]+ ?[Iquantile33-66]+ ?[Iquantile66>])

**Where all quantities are based on 1971-2000**

** ****Strategies: hedging, local studies (need for climatologists)**

## Findings: Translation, Extension

**‘****No Forecast’ language: No Skill seems to work better than Not Available, Equal Chances, or Climatological Probabilities.**
**People have difficulty comprehending terciles, even with graphical assistance. Source of confusion = terminology: ‘above’, ‘below’, ‘normal’ and ‘near normal’.**

**Reluctance to translate information**

**Notes:**
When people had to translate rather than mimic language, answers were similar to chance. 

## Findings: Translation, Extension

**‘****No Forecast’ language: No Skill seems to work better than Not Available, Equal Chances, or Climatological Probabilities.**
**People have difficulty comprehending terciles, even with graphical assistance. Source of confusion = terminology: ‘above’, ‘below’, ‘normal’ and ‘near normal’.**
**Reluctance to link even median and upper/lower half of reference distribution.**

**Reluctance to translate information**

**Notes:**
When people had to translate rather than mimic language, answers were similar to chance. 

## Findings: Translation, Extension

**‘****No Forecast’ language: No Skill seems to work better than Not Available, Equal Chances, or Climatological Probabilities.**
**People have difficulty comprehending terciles, even with graphical assistance. Source of confusion = terminology: ‘above’, ‘below’, ‘normal’ and ‘near normal’.**
**Reluctance to link even median and upper/lower half of reference distribution.**
**Reluctance to specify probabilities for non-favored categories. **

**Reluctance to translate information**

**Notes:**
When people had to translate rather than mimic language, answers were similar to chance. 

## Probability of Exceedance Outlooks

**Return rate was extremely low and most questions had no correct answers.**
**Simplified version had highest response rate and much higher rate of correct answers.**
***No one***** correctly answered that the probability of a single value is nearly zero... **

**Unique product... unique results**

## Findings/Recommendations

**Information itself insufficient. Disconnected elements create confusion. People have trouble coordinating and connecting the dots... ...**

## Findings/Recommendations

**Information itself insufficient. Disconnected elements create confusion. People have trouble coordinating and connecting the dots... ...**
**Confusion, tentativeness over basic principles: e.g., probability range=0-100%; median divides upper and lower half of distribution; forecasts address all three terciles.**

## Findings/Recommendations

**Information itself insufficient. Disconnected elements create confusion. People have trouble coordinating and connecting the dots... ...**
**Confusion, tentativeness over basic principles: e.g., probability range=0-100%, median divides upper and lower half of distribution; forecasts address all three terciles.**

**Structure information within product format.**
**Explicit reinforcement of basic principles.**

## New Formats: Local Seasonal Temp Outlooks

**New Formats: Local Seasonal Temp Outlooks**

**Climate Services Division and Western Region NWS**
*Marina Timofeyeva, Andrea Bair*

## New Formats: Local Seasonal Temp Outlooks

**New Formats: Local Seasonal Temp Outlooks**

## New Formats: Local Seasonal Temp Outlooks

**New Formats: Local Seasonal Temp Outlooks**

## New Formats: Local Seasonal Temp Outlooks

## New Formats: Local Seasonal Temp Outlooks

**New Formats: Local Seasonal Temp Outlooks**

## http://fet.hwr.arizona.edu/ForecastEvaluationTool/

**http://fet.hwr.arizona.edu/ForecastEvaluationTool/**

**Initially for NWS CPC climate forecasts**
**Now includes local seasonal outlooks, Alaska**
**Frequent request: add water supply forecasts**
**Six elements in our webtool:**
** *****Forecast Interpretation – Tutorials***
** *****Exploring Forecast Progression***
*** ******Historical Context***
*** ******Forecast Performance***
*** ******Use in Decision Making ***
*** ******Details: Forecast Techniques, Research***

## Forecast Performance Evaluation

**Forecast Performance Evaluation**

## Forecast Performance Evaluation

**Forecast Performance Evaluation**

**Neutral**

**Notes:**
This slide illustrates the Forecast Performance Evaluation track. Users specify which seasons, leadtimes, and evaluation criteria they are interested in (our new redesign also allows them to select which years they are interested in – at the request of the NWS Climate Services Division). Results are shown for all climate forecast divisions, but users can see the quantitative results for each region (the left map shows that, for precipitation forecasts issued Aug-Oct and covering Dec-April, forecasts have been issued 64% of the time, with EC/CL, i.e., ‘no forecast’ issued 36% of the time). Then, users can see the data that went into computing the evaluation scores, as a bubble plot, with the vertical position of a bubble representing it’s probability statement, the color of the bubble representing the observed tercile that occurred, and the bubble size representing the forecast lead time. The user then can select a bubble (i.e., forecast-observation pair) and get the archived forecast map and an observation map, with the observations indicating the tercile category that actually occurred. 

## Probability of Exceedance Outlook: Median Anomaly Map 

**Terms avoided**
-  Most likely
-  Normal (in text)
**Terms reinforced**
-  Median, with interpretation
-  Reference period 1971-2000
-  Other products, too

## Overall Recommendations

- Recognize broad user needs for knowledge development and information management
- Routine empirical assessment of communication of new forecast products
- Exploit evolving technologies: ‘live’ file formats and products, customized products, efficient information management tools